Apache Arrow in WebAssembly

Related tags

Utilities arrow-wasm
Overview

WASM Arrow .github/workflows/test.yml npm version

This package compiles the Rust library of Apache Arrow to WebAssembly. This might be a viable alternative to the pure JavaScript library. Right now, this library is incomplete and the API will change so we recommend using JavaScript library.

You can try this library in Observable at https://observablehq.com/@domoritz/apache-arrow-in-webassembly.

Documentation

Coming later. The API is somewhat similar to the Rust version (https://docs.rs/arrow/3.0.0/arrow/) although there are some differences to make the API more familiar to JavaScript users.

Building and testing

Node

Run with wasm-pack build --target nodejs && node examples/flights.js.

To use a debug build, run wasm-pack build --target nodejs --dev && node examples/flights.js.

Browser

Build with wasm-pack build --target web. Then run python3 -m http.server and open http://localhost:8000/examples/.

Publishing

Run npm publish to build a bundle and release it to NPM.

Linting

Run cargo fmt && cargo clippy before committing.

Check file size

We can check how large the WASM file is after compression (which every web server probably does).

gzip -9 <pkg//arrow_wasm_bg.wasm | wc -c

Comments
  • Improve passing buffer to wasm

    Improve passing buffer to wasm

    Loading arrow from ipc will be a super common use case so it should be fast. We want to avoid as many memory copies as possible. We should look into what is happening and which copies we can avoid.

    Try https://github.com/rustwasm/wasm-bindgen/issues/1643 and https://github.com/rustwasm/wasm-bindgen/issues/1993 and https://github.com/rustwasm/wasm-bindgen/issues/2456

    enhancement 
    opened by domoritz 3
  • chore(deps): bump arrow from 10.0.0 to 26.0.0

    chore(deps): bump arrow from 10.0.0 to 26.0.0

    Bumps arrow from 10.0.0 to 26.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    25.0.0 (2022-10-14)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    Fixed bugs:

    • Don't try to infer nulls in CSV schema inference #2859 [arrow]
    • parquet::arrow::arrow_writer::ArrowWriter ignores page size properties #2853 [parquet]
    • Introducing ArrowNativeTypeOp made it impossible to call kernels from generics #2839 [arrow]
    • Unsound ArrayData to Array Conversions #2834 [parquet] [arrow]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 25.0.0

    chore(deps): bump arrow from 10.0.0 to 25.0.0

    Bumps arrow from 10.0.0 to 25.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    24.0.0 (2022-09-30)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • Include field name in Parquet PrimitiveTypeBuilder error messages #2804 [parquet]
    • Add PrimitiveArray::reinterpret_cast #2785
    • BinaryBuilder and StringBuilder initialization parameters in struct_builder may be wrong #2783 [arrow]
    • Add divide scalar dyn kernel which produces null for division by zero #2767 [arrow]
    • Add divide dyn kernel which produces null for division by zero #2763 [arrow]
    • Improve performance of checked kernels on non-null data #2747 [arrow]
    • Add overflow-checking variants of arithmetic dyn kernels #2739 [arrow]
    • The binary function should not panic on unequaled array length. #2721 [arrow]

    Fixed bugs:

    • min compute kernel is incorrect with sliced buffers in arrow 23 #2779 [arrow]
    • try_unary_dict should check value type of dictionary array #2754 [arrow]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 24.0.0

    chore(deps): bump arrow from 10.0.0 to 24.0.0

    Bumps arrow from 10.0.0 to 24.0.0.

    Changelog

    Sourced from arrow's changelog.

    24.0.0 (2022-09-16)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • Cleanup like and nlike utf8 kernels #2744 [arrow]
    • Speedup eq and neq kernels for utf8 arrays #2742 [arrow]
    • API for more ergonomic construction of RecordBatchOptions #2728 [arrow]
    • Automate updates to CHANGELOG-old.md #2726
    • Don't check the DivideByZero error for float modulus #2720 [arrow]
    • try_binary should not panic on unequaled array length. #2715 [arrow]
    • Add benchmark for bitwise operation #2714 [arrow]
    • Add overflow-checking variants of arithmetic scalar dyn kernels #2712 [arrow]
    • Add divide_opt kernel which produce null values on division by zero error #2709 [arrow]
    • Add DataType function to detect nested types #2704 [arrow]
    • Add support of sorting dictionary of other primitive types #2700 [arrow]
    • Sort indices of dictionary string values #2697 [arrow]
    • Support empty projection in RecordBatch::project #2690 [arrow]
    • Support sorting dictionary encoded primitive integer arrays #2679 [arrow]
    • Use BitIndexIterator in min_max_helper #2674 [arrow]
    • Support building comparator for dictionaries of primitive integer values #2672 [arrow]
    • Change max/min string macro to generic helper function min_max_helper #2657 [arrow]
    • Add overflow-checking variant of arithmetic scalar kernels #2651 [arrow]
    • Compare dictionary with binary array #2644 [arrow]
    • Add overflow-checking variant for primitive arithmetic kernels #2642 [arrow]
    • Use downcast_primitive_array in arithmetic kernels #2639 [arrow]
    • Support DictionaryArray in temporal kernels #2622 [arrow]
    • Inline Generated Thift Code Into Parquet Crate #2502 [parquet]

    Fixed bugs:

    • Escape contains patterns for utf8 like kernels #2745 [arrow]
    • Float Array should not panic on DivideByZero in the Divide kernel #2719 [arrow]
    • DictionaryBuilders can Create Invalid DictionaryArrays #2684 [parquet] [arrow]
    • arrow crate does not build with features = ["ffi"] and default_features = false. #2670 [arrow]
    • Invalid results with RowSelector having row_count of 0 #2669 [parquet]
    • clippy error: unresolved import crate::array::layout #2659 [arrow]
    • Cast the numeric without the CastOptions #2648 [arrow]
    • Explicitly define overflow behavior for primitive arithmetic kernels #2641 [arrow]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 23.0.0

    chore(deps): bump arrow from 10.0.0 to 23.0.0

    Bumps arrow from 10.0.0 to 23.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    22.0.0 (2022-09-02)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • Add Macros to assist with static dispatch #2635 [arrow]
    • Support comparison between DictionaryArray and BooleanArray #2617 [arrow]
    • Use total_cmp for floating value ordering and remove nan_ordering feature flag #2613 [arrow]
    • Support empty projection in CSV, JSON readers #2603 [arrow]
    • Support SQL-compliant NaN ordering between for DictionaryArray and non-DictionaryArray #2599 [arrow]
    • Add dyn_cmp_dict feature flag to gate dyn comparison of dictionary arrays #2596 [arrow]
    • Add max_dyn and min_dyn for max/min for dictionary array #2584 [arrow]
    • Allow FlightSQL implementers to extend do_get() #2581 [arrow-flight]
    • Support SQL-compliant behavior on eq_dyn, neq_dyn, lt_dyn, lt_eq_dyn, gt_dyn, gt_eq_dyn #2569 [arrow]
    • Add sql-compliant feature for enabling sql-compliant kernel behavior #2568
    • Calculate sum for dictionary array #2565 [arrow]

    ... (truncated)

    Commits
    • 5a55406 update new arrow-buffer crate to 23.0.0 (#2748)
    • 1da2bfb Update version to 23.0.0 and update CHANGELOG, add label_issue.py scrip...
    • 968a767 Speedup string equal/not equal to empty string, cleanup like/ilike kernels, f...
    • f572ec1 Update try_binary and checked_ops, and remove math_checked_op (#2717)
    • 43d912c Better construction of RecordBatchOptions (#2729)
    • 0ebd71e Partially flatten arrow-buffer (#2737)
    • 5238789 Automate updates to CHANGELOG-old.md (#2732)
    • eb9b456 benchmark: bitwise operation (#2718)
    • a7a9329 Update read parquet example in parquet/arrow home (#2730)
    • 5f441ee Fix multipart uploads on Minio (#2731)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 22.0.0

    chore(deps): bump arrow from 10.0.0 to 22.0.0

    Bumps arrow from 10.0.0 to 22.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    21.0.0 (2022-08-18)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • add into_inner method to ArrowWriter #2491 [parquet]
    • Remove byteorder dependency #2472 [parquet]
    • Return Structured ColumnCloseResult from GenericColumnWriter::close #2465 [parquet]
    • Push ChunkReader into SerializedPageReader #2463 [parquet]
    • Support SerializedPageReader::skip_page without OffsetIndex #2459 [parquet]
    • Support Time64/Time32 comparison #2457 [arrow]
    • Revise FromIterator for Decimal128Array to use Into instead of Borrow #2441 [parquet]
    • Support RowFilter withinParquetRecordBatchReader #2431 [parquet]
    • Remove the field StructBuilder::len #2429 [arrow]
    • Standardize creation and configuration of parquet --> Arrow readers ( ParquetRecordBatchReaderBuilder) #2427 [parquet]
    • Use OffsetIndex to Prune IO in ParquetRecordBatchStream #2426 [parquet]
    • Support peek_next_page and skip_next_page in InMemoryPageReader #2406 [parquet]
    • Support casting from Utf8/LargeUtf8 to Binary/LargeBinary #2402 [arrow]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 21.0.0

    chore(deps): bump arrow from 10.0.0 to 21.0.0

    Bumps arrow from 10.0.0 to 21.0.0.

    Changelog

    Sourced from arrow's changelog.

    21.0.0 (2022-08-18)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • add into_inner method to ArrowWriter #2491 [parquet]
    • Remove byteorder dependency #2472 [parquet]
    • Return Structured ColumnCloseResult from GenericColumnWriter::close #2465 [parquet]
    • Push ChunkReader into SerializedPageReader #2463 [parquet]
    • Support SerializedPageReader::skip_page without OffsetIndex #2459 [parquet]
    • Support Time64/Time32 comparison #2457 [arrow]
    • Revise FromIterator for Decimal128Array to use Into instead of Borrow #2441 [parquet]
    • Support RowFilter withinParquetRecordBatchReader #2431 [parquet]
    • Remove the field StructBuilder::len #2429 [arrow]
    • Standardize creation and configuration of parquet --> Arrow readers ( ParquetRecordBatchReaderBuilder) #2427 [parquet]
    • Use OffsetIndex to Prune IO in ParquetRecordBatchStream #2426 [parquet]
    • Support peek_next_page and skip_next_page in InMemoryPageReader #2406 [parquet]
    • Support casting from Utf8/LargeUtf8 to Binary/LargeBinary #2402 [arrow]
    • Support casting between Decimal128 and Decimal256 arrays #2375 [arrow]
    • Combine multiple selections into the same batch size in skip_records #2358 [parquet]
    • Add API to change timezone for timestamp array #2346 [arrow]
    • Change the output of read_buffer Arrow IPC API to return Result<_> #2342 [arrow]
    • Allow skip_records in GenericColumnReader to skip across row groups #2331 [parquet]
    • Optimize the validation of Decimal256 #2320 [arrow]
    • Implement Skip for DeltaBitPackDecoder #2281 [parquet]
    • Changes to ParquetRecordBatchStream to support row filtering in DataFusion #2270 [parquet]
    • Add ArrayReader::skip_records API #2197 [parquet]

    Fixed bugs:

    • Panic in SerializedPageReader without offset index #2503 [parquet]
    • MapArray columns don't handle null values correctly #2484 [arrow]
    • There is no compiler error when using an invalid Decimal type. #2440 [arrow]
    • Flight SQL Server sends incorrect response for DoPutUpdateResult #2403 [arrow-flight]
    • AsyncFileReaderNo Longer Object-Safe #2372 [parquet]
    • StructBuilder Does not Verify Child Lengths #2252 [arrow]

    Closed issues:

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 20.0.0

    chore(deps): bump arrow from 10.0.0 to 20.0.0

    Bumps arrow from 10.0.0 to 20.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    19.0.0 (2022-07-22)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • Use total_cmp from std #2130 [arrow]
    • Permit parallel fetching of column chunks in ParquetRecordBatchStream #2110 [parquet]
    • The GenericBinaryBuilder should use buffer builders directly. #2104 [arrow]
    • Pass generate_decimal256_case arrow integration test #2093 [arrow]
    • Rename weekday and weekday0 kernels to to num_days_from_monday and days_since_sunday #2065 [arrow]
    • Improve performance of filter_dict #2062 [arrow]
    • Improve performance of set_bits #2060 [arrow]
    • Lazily materialize the null buffer builder of BooleanBuilder #2058 [arrow]
    • BooleanArray::from_iter should omit validity buffer if all values are valid #2055 [arrow]
    • FFI_ArrowSchema should set DICTIONARY_ORDERED flag if a field's dictionary is ordered #2049 [arrow]
    • Support peek_next_page() and skip_next_page in SerializedPageReader #2043 [parquet]
    • Support FFI / C Data Interface for MapType #2037 [arrow]
    • The DecimalArrayBuilder should use FixedSizedBinaryBuilder #2026 [arrow]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 19.0.0

    chore(deps): bump arrow from 10.0.0 to 19.0.0

    Bumps arrow from 10.0.0 to 19.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    18.0.0 (2022-07-08)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • Add DataType::Dictionary support to subtract_scalar, multiply_scalar, divide_scalar #2019 [arrow]
    • Support DictionaryArray in add_scalar kernel #2017 [arrow]
    • Enable column page index read test for all types #2010 [parquet]
    • Simplify FixedSizeBinaryBuilder #2007 [arrow]
    • Support Decimal256Builder and Decimal256Array #1999 [arrow]
    • Support DictionaryArray in unary kernel #1989 [arrow]
    • Add kernel to quickly compute comparisons on Arrays #1987 [arrow]
    • Support DictionaryArray in divide kernel #1982 [arrow]
    • Implement Into<ArrayData> for T: Array #1979 [arrow]
    • Support DictionaryArray in multiply kernel #1972 [arrow]
    • Support DictionaryArray in subtract kernel #1970 [arrow]
    • Declare DecimalArray::length as a constant #1967 [arrow]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 18.0.0

    chore(deps): bump arrow from 10.0.0 to 18.0.0

    Bumps arrow from 10.0.0 to 18.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    17.0.0 (2022-06-24)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • add a small doc example showing ArrowWriter being used with a cursor #1927 [parquet]
    • Support cast to/from NULL and DataType::Decimal #1921 [arrow]
    • Add Decimal256 API #1913 [arrow]
    • Add DictionaryArray::key function #1911 [arrow]
    • Support specifying capacities for ListArrays in MutableArrayData #1884 [arrow]
    • Explicitly declare the features used for each dependency #1876 [parquet] [arrow] [arrow-flight]
    • Add Decimal128 API and use it in DecimalArray and DecimalBuilder #1870 [arrow]
    • PrimitiveArray::from_iter should omit validity buffer if all values are valid #1856 [arrow]
    • Add from(v: Vec<Option<&[u8]>>) and from(v: Vec<&[u8]>) for FixedSizedBInaryArray #1852 [arrow]
    • Add Vec-inspired APIs to BufferBuilder #1850 [arrow]
    • PyArrow intergation test for C Stream Interface #1847 [arrow]
    • Add nilike support in comparison #1845 [arrow]
    • Split up arrow::array::builder module #1843 [arrow]
    • Add quarter support in temporal kernels #1835 [arrow]
    • Rename ArrayData::validate_dictionary_offset to ArrayData::validate_values #1812 [arrow]

    ... (truncated)

    Commits
    • 330505c Prepare for 18.0.0 release: Update CHANGELOG and versions (#2023)
    • 5a76697 Fix record delimiting on row group boundaries (#2025) (#2027)
    • ca1bfb8 Add Decimal256Builder and Decimal256Array (#2000)
    • d70c2ea Consistent case in Index enumeration (#2029)
    • 373ac81 Add page index reader test for all types and support empty index. (#2012)
    • 9333a85 Don't generate empty google.protobuf.rs (#2005)
    • 58848c4 Fix parquet test_common feature flags (#2003)
    • d9dbf72 Fix several bugs in parquet writer statistics generation, add `EnabledStatist...
    • 7542f7d Add builder style APIs For Field: with_name, with_data_type and `with_n...
    • ef02bf8 Support dictionary array for divide kernel (#1983)
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 17.0.0

    chore(deps): bump arrow from 10.0.0 to 17.0.0

    Bumps arrow from 10.0.0 to 17.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    16.0.0 (2022-06-10)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • List equality method should work on empty offset ListArray #1817 [arrow]
    • Command line tool for convert CSV to Parquet #1797 [parquet]
    • IPC writer should write validity buffer for UnionArray in V4 IPC message #1793 [arrow]
    • Add function for row alignment with page mask #1790 [parquet]
    • Rust IPC Read should be able to read V4 UnionType Array #1788 [arrow]
    • combine_option_bitmap should accept arbitrary number of input arrays. #1780 [arrow]
    • Add substring_by_char kernels for slicing on character boundaries #1768 [arrow]
    • Support reading PageIndex from column metadata #1761 [parquet]
    • Support casting from DataType::Utf8 to DataType::Boolean #1740 [arrow]
    • Make current position available in FileWriter. #1691 [parquet]
    • Support writing parquet to stdout #1687 [parquet]

    Fixed bugs:

    • Incorrect Offset Validation for Sliced List Array Children #1814 [arrow]
    • Parquet Snappy Codec overwrites Existing Data in Decompression Buffer #1806 [parquet]
    • flight_data_to_arrow_batch does not support RecordBatches with no columns #1783 [arrow-flight]

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 1
  • chore(deps): bump arrow from 10.0.0 to 27.0.0

    chore(deps): bump arrow from 10.0.0 to 27.0.0

    Bumps arrow from 10.0.0 to 27.0.0.

    Changelog

    Sourced from arrow's changelog.

    Historical Changelog

    26.0.0 (2022-10-28)

    Full Changelog

    Breaking changes:

    Implemented enhancements:

    • Optimized way to count the numbers of true and false values in a BooleanArray #2963 [arrow]
    • Add pow to i256 #2954 [arrow]
    • Write Generic Code over [Large]BinaryArray and [Large]StringArray #2946 [arrow]
    • Add Page Row Count Limit #2941 [parquet]
    • prettyprint to show timezone offset for timestamp with timezone #2937 [arrow]
    • Cast numeric to decimal256 #2922 [arrow]
    • Add freeze_with_dictionary API to MutableArrayData #2914 [arrow]
    • Support decimal256 array in sort kernels #2911 [arrow]
    • support [+/-]hhmm and [+/-]hh as fixedoffset timezone format #2910 [arrow]
    • Cleanup decimal sort function #2907 [arrow]
    • replace from_timestamp by from_timestamp_opt #2892 [arrow]
    • Move Primitive arity kernels to arrow-array #2787 [arrow]
    • add overflow-checking for negative arithmetic kernel #2662 [arrow]

    Fixed bugs:

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • chore(deps): bump web-sys from 0.3.59 to 0.3.60

    chore(deps): bump web-sys from 0.3.59 to 0.3.60

    Bumps web-sys from 0.3.59 to 0.3.60.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • chore(deps): bump wasm-bindgen from 0.2.82 to 0.2.83

    chore(deps): bump wasm-bindgen from 0.2.82 to 0.2.83

    Bumps wasm-bindgen from 0.2.82 to 0.2.83.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • chore(deps): bump js-sys from 0.3.59 to 0.3.60

    chore(deps): bump js-sys from 0.3.59 to 0.3.60

    Bumps js-sys from 0.3.59 to 0.3.60.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
  • chore(deps): bump wasm-bindgen-test from 0.3.32 to 0.3.33

    chore(deps): bump wasm-bindgen-test from 0.3.32 to 0.3.33

    Bumps wasm-bindgen-test from 0.3.32 to 0.3.33.

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    dependencies 
    opened by dependabot[bot] 0
Owner
Dominik Moritz
Professor at CMU (@cmudig) and researcher at @apple. PhD in Computer Science from the University of Washington (@uwdata, @uwdb). Co-creator @vega, @streamlit.
Dominik Moritz
This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to the arrow format.

Arrow2-derive - derive for Arrow2 This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to th

Jorge Leitao 29 Dec 27, 2022
An experimental implementation of Arc against Apache Datafusion

box This is an experimental repository to perform a proof of concept replacement of the Apache Spark executor for Arc with Apache DataFusion. This is

tripl.ai 1 Nov 26, 2021
Pure Rust library for Apache ZooKeeper built on tokio

zookeeper-async Async Zookeeper client written 100% in Rust, based on tokio. This library is intended to be equivalent with the official (low-level) Z

Kamil Rojewski 16 Dec 16, 2022
🥸 Learning WebAssembly

learn-wasm # Initialize the project # https://github.com/lencx/create-xc-app npm init xc-app learn-wasm -t wasm-react Awesome Lists WebAssembly入门 Exam

Len C... 86 Dec 27, 2022
Async executor for WebAssembly

There are a number of async task executors available in Rust's ecosystem. However, most (if not all?) of them rely on primitives that might not be available or optimal for WebAssembly deployment at the time.

wasm.rs 65 Dec 31, 2022
An n-tuple pendulum simulator in Rust + WebAssembly (and JavaScript)

An n-tuple pendulum simulator in Rust + WebAssembly (and JavaScript) Remaking this n-tuple pendulum simulator moving the math to Rust ?? and WebAssemb

Travis Taylor 27 Feb 19, 2022
A rollup plugin that compile Rust code into WebAssembly modules

rollup-plugin-rust tl;dr -- see examples This is a rollup plugin that loads Rust code so it can be interop with Javascript base project. Currently, th

Fahmi Akbar Wildana 37 Aug 1, 2022
Continuous runtime observablity SDKs to monitor WebAssembly code.

Observe Observe is an observability SDK for WebAssembly. At the moment we support wasmtime hosts and we output opentelemetry data to stdout. We plan t

Dylibso 4 Jun 8, 2023
Cargo wrapper for working with Webassembly wasi(x).

cargo-wasix A cargo subcommand that wraps regular cargo commands for compiling Rust code to wasix, a superset of Websassembly wasi with additional fun

WASIX 19 Jun 12, 2023
Distributed compute platform implemented in Rust, and powered by Apache Arrow.

Ballista: Distributed Compute Platform Overview Ballista is a distributed compute platform primarily implemented in Rust, powered by Apache Arrow. It

Ballista 2.3k Jan 3, 2023
Distributed compute platform implemented in Rust, and powered by Apache Arrow.

Ballista: Distributed Compute Platform Overview Ballista is a distributed compute platform primarily implemented in Rust, powered by Apache Arrow. It

Ballista 2.3k Jan 3, 2023
A Rust DataFrame implementation, built on Apache Arrow

Rust DataFrame A dataframe implementation in Rust, powered by Apache Arrow. What is a dataframe? A dataframe is a 2-dimensional tabular data structure

Wakahisa 287 Nov 11, 2022
Official Rust implementation of Apache Arrow

Native Rust implementation of Apache Arrow Welcome to the implementation of Arrow, the popular in-memory columnar format, in Rust. This part of the Ar

The Apache Software Foundation 1.3k Jan 9, 2023
Apache Arrow DataFusion and Ballista query engines

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

The Apache Software Foundation 2.9k Jan 2, 2023
Fill Apache Arrow record batches from an ODBC data source in Rust.

arrow-odbc Fill Apache Arrow arrays from ODBC data sources. This crate is build on top of the arrow and odbc-api crate and enables you to read the dat

Markus Klein 21 Dec 27, 2022
Generated Ryst of Apache Arrow spec

Arrow generated IPC format The generated flatbuffers code for Rust. Note that these files suffered modifications because flatbuffers is unable to comp

null 14 Nov 30, 2022
Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

Apache Arrow Powering In-Memory Analytics Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enabl

The Apache Software Foundation 10.9k Jan 6, 2023
Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow

Welcome to arrowdantic Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow that can interoperate with Parquet

Jorge Leitao 52 Dec 21, 2022
Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.

Polars Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow Blazingly fast DataFrames in Rust, Python & Node.js Polars is

null 11.8k Jan 8, 2023
Fast regex in Rust for Apache Arrow, compiled to WASM

Rust regex in wasm I have been looking for a fast regular expression library in Javascript that runs on Apache Arrow for a few years. Arrow uses UTF-8

Nomic AI 3 May 3, 2023