Apache Arrow in WebAssembly

Dominik Moritz

Last update: Jan 1, 2023

Related tags

Utilities arrow-wasm

Overview

WASM Arrow

This package compiles the Rust library of Apache Arrow to WebAssembly. This might be a viable alternative to the pure JavaScript library. Right now, this library is incomplete and the API will change so we recommend using JavaScript library.

You can try this library in Observable at https://observablehq.com/@domoritz/apache-arrow-in-webassembly.

Documentation

Coming later. The API is somewhat similar to the Rust version (https://docs.rs/arrow/3.0.0/arrow/) although there are some differences to make the API more familiar to JavaScript users.

Building and testing

Node

Run with wasm-pack build --target nodejs && node examples/flights.js.

To use a debug build, run wasm-pack build --target nodejs --dev && node examples/flights.js.

Browser

Build with wasm-pack build --target web. Then run python3 -m http.server and open http://localhost:8000/examples/.

Publishing

Run npm publish to build a bundle and release it to NPM.

Linting

Run cargo fmt && cargo clippy before committing.

Check file size

We can check how large the WASM file is after compression (which every web server probably does).

gzip -9 <pkg//arrow_wasm_bg.wasm | wc -c

Comments

Improve passing buffer to wasm

Loading arrow from ipc will be a super common use case so it should be fast. We want to avoid as many memory copies as possible. We should look into what is happening and which copies we can avoid.

Try https://github.com/rustwasm/wasm-bindgen/issues/1643 and https://github.com/rustwasm/wasm-bindgen/issues/1993 and https://github.com/rustwasm/wasm-bindgen/issues/2456
enhancement

opened by domoritz 3
chore(deps): bump arrow from 10.0.0 to 26.0.0
Bumps arrow from 10.0.0 to 26.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

25.0.0 (2022-10-14)

Full Changelog

Breaking changes:

Make DecimalArray as PrimitiveArray #2857 [parquet] [arrow] (viirya)

fix timestamp parsing while no explicit timezone given #2814 [arrow] (waitingkuo)

Support Arbitrary Number of Arrays in downcast_primitive_array #2809 (tustvold)

Implemented enhancements:

Restore Integration test JSON schema serialization #2876 [arrow]

Fix various invalid_html_tags clippy error #2861 [parquet] [arrow] [arrow-flight]

Replace complicated temporal macro with generic functions #2851 [arrow]

Add NaN handling in dyn scalar comparison kernels #2829 [arrow]

Add overflow-checking variant of sum kernel #2821 [arrow]

Update to Clap 4 #2817 [parquet]

Safe API to Operate on Dictionary Values #2797 [arrow]

Add modulus op into ArrowNativeTypeOp #2753 [arrow]

Allow creating of TimeUnit instances without direct dependency on parquet-format #2708 [parquet]

Arrow Row Format #2677 [arrow]

Fixed bugs:

Don't try to infer nulls in CSV schema inference #2859 [arrow]

parquet::arrow::arrow_writer::ArrowWriter ignores page size properties #2853 [parquet]

Introducing ArrowNativeTypeOp made it impossible to call kernels from generics #2839 [arrow]

Unsound ArrayData to Array Conversions #2834 [parquet] [arrow]

... (truncated)

Commits

36b8dfc Add entry to changelog for 26.0.0 RC2 fix

fb2b49c Fix ignored limit on lexsort_to_indices (#2991)

7798043 Update version to 26.0.0 and add Changelog (#2962)

cbee739 Format Timestamps as RFC3339 (#2939)

843a2e5 Add BooleanArray::true_count and BooleanArray::false_count (#2957)

87ac05b Fix take string on sliced indices (#2960)

0a115d6 Fix GenericListArray::try_new_from_array_data error message (#526) (#2961)

b6f08a8 Add GenericByteArray (#2946) (#2947)

73416f8 Cast numeric to decimal256 (#2923)

b4872b7 Cleanup CI (#2933)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 25.0.0
Bumps arrow from 10.0.0 to 25.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

24.0.0 (2022-09-30)

Full Changelog

Breaking changes:

Cleanup ArrowNativeType (#1918) #2793 [parquet] [arrow] (tustvold)

Remove ArrowNativeType::FromStr #2775 [arrow] (tustvold)

Split out arrow-array crate (#2594) #2769 [arrow] (tustvold)

Add dyn_arith_dict feature flag #2760 [arrow] (tustvold)

Split out arrow-data into a separate crate #2746 [arrow] (tustvold)

Split out arrow-schema (#2594) #2711 [arrow] (tustvold)

Implemented enhancements:

Include field name in Parquet PrimitiveTypeBuilder error messages #2804 [parquet]

Add PrimitiveArray::reinterpret_cast #2785

BinaryBuilder and StringBuilder initialization parameters in struct_builder may be wrong #2783 [arrow]

Add divide scalar dyn kernel which produces null for division by zero #2767 [arrow]

Add divide dyn kernel which produces null for division by zero #2763 [arrow]

Improve performance of checked kernels on non-null data #2747 [arrow]

Add overflow-checking variants of arithmetic dyn kernels #2739 [arrow]

The binary function should not panic on unequaled array length. #2721 [arrow]

Fixed bugs:

min compute kernel is incorrect with sliced buffers in arrow 23 #2779 [arrow]

try_unary_dict should check value type of dictionary array #2754 [arrow]

... (truncated)

Commits

1eb19b5 Update version to 25.0.0 and update Changelog (#2877)

9c5ba92 Split out arrow-integration-test crate (#2868)

d67d5fb Decimal cleanup (#2637) (#2865)

d7f994c Take decimal as primitive (#2637) (#2869)

1b69dd7 Update test_parse_arg_compression_format_fail (#2867)

eeb1261 Make DecimalArray as PrimitiveArray (#2857)

fa1d079 Add interleave kernel (#1523) (#2838)

8adebca Validate ArrayData type when converting to Array (#2834) (#2835)

1397fb4 Don't try to infer nullability in CSV reader (#2860)

65d5576 Simplify OrderPreservingInterner allocation strategy (#2677) (#2827)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 24.0.0
Bumps arrow from 10.0.0 to 24.0.0.

Changelog

Sourced from arrow's changelog.

24.0.0 (2022-09-16)

Full Changelog

Breaking changes:

Move JSON Test Format To integration-testing #2724 [arrow] (tustvold)

Split out arrow-buffer crate (#2594) #2693 [arrow] (tustvold)

Simplify DictionaryBuilder constructors (#2684) (#2054) #2685 [parquet] [arrow] (tustvold)

Deprecate RecordBatch::concat replace with concat_batches (#2594) #2683 [arrow] (tustvold)

Add overflow-checking variant for primitive arithmetic kernels and explicitly define overflow behavior #2643 [arrow] (viirya)

Update thrift v0.16 and vendor parquet-format (#2502) #2626 [parquet] (tustvold)

Update flight definitions including backwards-incompatible change to GetSchema #2586 [arrow] [arrow-flight] (liukun4515)

Implemented enhancements:

Cleanup like and nlike utf8 kernels #2744 [arrow]

Speedup eq and neq kernels for utf8 arrays #2742 [arrow]

API for more ergonomic construction of RecordBatchOptions #2728 [arrow]

Automate updates to CHANGELOG-old.md #2726

Don't check the DivideByZero error for float modulus #2720 [arrow]

try_binary should not panic on unequaled array length. #2715 [arrow]

Add benchmark for bitwise operation #2714 [arrow]

Add overflow-checking variants of arithmetic scalar dyn kernels #2712 [arrow]

Add divide_opt kernel which produce null values on division by zero error #2709 [arrow]

Add DataType function to detect nested types #2704 [arrow]

Add support of sorting dictionary of other primitive types #2700 [arrow]

Sort indices of dictionary string values #2697 [arrow]

Support empty projection in RecordBatch::project #2690 [arrow]

Support sorting dictionary encoded primitive integer arrays #2679 [arrow]

Use BitIndexIterator in min_max_helper #2674 [arrow]

Support building comparator for dictionaries of primitive integer values #2672 [arrow]

Change max/min string macro to generic helper function min_max_helper #2657 [arrow]

Add overflow-checking variant of arithmetic scalar kernels #2651 [arrow]

Compare dictionary with binary array #2644 [arrow]

Add overflow-checking variant for primitive arithmetic kernels #2642 [arrow]

Use downcast_primitive_array in arithmetic kernels #2639 [arrow]

Support DictionaryArray in temporal kernels #2622 [arrow]

Inline Generated Thift Code Into Parquet Crate #2502 [parquet]

Fixed bugs:

Escape contains patterns for utf8 like kernels #2745 [arrow]

Float Array should not panic on DivideByZero in the Divide kernel #2719 [arrow]

DictionaryBuilders can Create Invalid DictionaryArrays #2684 [parquet] [arrow]

arrow crate does not build with features = ["ffi"] and default_features = false. #2670 [arrow]

Invalid results with RowSelector having row_count of 0 #2669 [parquet]

clippy error: unresolved import crate::array::layout #2659 [arrow]

Cast the numeric without the CastOptions #2648 [arrow]

Explicitly define overflow behavior for primitive arithmetic kernels #2641 [arrow]

... (truncated)

Commits

6477db1 Prepare for 24.0.0 (#2808)

e2bf158 add field name to parquet PrimitiveTypeBuilder error messages (#2805)

da8f742 Fix S3 query canonicalization (#2800) (#2801)

f845d6e Handle incomplete HTTP redirects missing LOCATION (#2795) (#2796)

178319b Replace checked casts with as for performance (#1918) (#2793)

7639f28 Add struct equality test case (#514) (#2791)

322d351 Add PrimitiveArray::reinterpret_cast (#2785) (#2786)

e97bd6f Disable test harness for string_dictionary_builder benchmark (#2788)

2ba1307 Move unary kernels to arrow-array (#2787) (#2789)

333633e fix(arrow-array): fix BinaryBuilder and StringBuilder initialization paramete...

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 23.0.0
Bumps arrow from 10.0.0 to 23.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

22.0.0 (2022-09-02)

Full Changelog

Breaking changes:

Use total_cmp for floating value ordering and remove nan_ordering feature flag #2614 [arrow] (viirya)

Gate dyn comparison of dictionary arrays behind dyn_cmp_dict #2597 [arrow] (tustvold)

Move JsonSerializable to json module (#2300) #2595 [arrow] (tustvold)

Decimal precision scale datatype change #2532 [parquet] [arrow] (psvri)

Refactor PrimitiveBuilder Constructors #2518 [parquet] [arrow] (psvri)

Refactoring DecimalBuilder constructors #2517 [arrow] (psvri)

Refactor FixedSizeBinaryBuilder Constructors #2516 [parquet] [arrow] (psvri)

Refactor BooleanBuilder Constructors #2515 [arrow] (psvri)

Refactor UnionBuilder Constructors #2488 [arrow] (psvri)

Implemented enhancements:

Add Macros to assist with static dispatch #2635 [arrow]

Support comparison between DictionaryArray and BooleanArray #2617 [arrow]

Use total_cmp for floating value ordering and remove nan_ordering feature flag #2613 [arrow]

Support empty projection in CSV, JSON readers #2603 [arrow]

Support SQL-compliant NaN ordering between for DictionaryArray and non-DictionaryArray #2599 [arrow]

Add dyn_cmp_dict feature flag to gate dyn comparison of dictionary arrays #2596 [arrow]

Add max_dyn and min_dyn for max/min for dictionary array #2584 [arrow]

Allow FlightSQL implementers to extend do_get() #2581 [arrow-flight]

Support SQL-compliant behavior on eq_dyn, neq_dyn, lt_dyn, lt_eq_dyn, gt_dyn, gt_eq_dyn #2569 [arrow]

Add sql-compliant feature for enabling sql-compliant kernel behavior #2568

Calculate sum for dictionary array #2565 [arrow]

... (truncated)

Commits

5a55406 update new arrow-buffer crate to 23.0.0 (#2748)

1da2bfb Update version to 23.0.0 and update CHANGELOG, add label_issue.py scrip...

968a767 Speedup string equal/not equal to empty string, cleanup like/ilike kernels, f...

f572ec1 Update try_binary and checked_ops, and remove math_checked_op (#2717)

43d912c Better construction of RecordBatchOptions (#2729)

0ebd71e Partially flatten arrow-buffer (#2737)

5238789 Automate updates to CHANGELOG-old.md (#2732)

eb9b456 benchmark: bitwise operation (#2718)

a7a9329 Update read parquet example in parquet/arrow home (#2730)

5f441ee Fix multipart uploads on Minio (#2731)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 22.0.0
Bumps arrow from 10.0.0 to 22.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

21.0.0 (2022-08-18)

Full Changelog

Breaking changes:

Return structured ColumnCloseResult (#2465) #2466 [parquet] (tustvold)

Push ChunkReader into SerializedPageReader (#2463) #2464 [parquet] (tustvold)

Revise FromIterator for Decimal128Array to use Into instead of Borrow #2442 [parquet] [arrow] (viirya)

Use Fixed-Length Array in BasicDecimal new and raw_value #2405 [arrow] (HaoYang670)

Remove deprecated ParquetWriter #2380 [parquet] (tustvold)

Remove deprecated SliceableCursor and InMemoryWriteableCursor #2378 [parquet] (tustvold)

Implemented enhancements:

add into_inner method to ArrowWriter #2491 [parquet]

Remove byteorder dependency #2472 [parquet]

Return Structured ColumnCloseResult from GenericColumnWriter::close #2465 [parquet]

Push ChunkReader into SerializedPageReader #2463 [parquet]

Support SerializedPageReader::skip_page without OffsetIndex #2459 [parquet]

Support Time64/Time32 comparison #2457 [arrow]

Revise FromIterator for Decimal128Array to use Into instead of Borrow #2441 [parquet]

Support RowFilter withinParquetRecordBatchReader #2431 [parquet]

Remove the field StructBuilder::len #2429 [arrow]

Standardize creation and configuration of parquet --> Arrow readers ( ParquetRecordBatchReaderBuilder) #2427 [parquet]

Use OffsetIndex to Prune IO in ParquetRecordBatchStream #2426 [parquet]

Support peek_next_page and skip_next_page in InMemoryPageReader #2406 [parquet]

Support casting from Utf8/LargeUtf8 to Binary/LargeBinary #2402 [arrow]

... (truncated)

Commits

e5b9d05 Update version to 22.0.0 and update CHANGELOG (#2612)

2b2c15b Add Linux, Mac and Windows .gitignore (#2630)

5a49cfe Add downcast macros (#2635) (#2636)

2d3a350 Use http with fake-gcs (#2632)

a9875f5 Document all arrow features in docs.rs (#2633) (#2634)

41a2d2d Document dyn_cmp_dict (#2624)

94bc726 Update quick-xml 0.24 (#2625)

b792ff7 Make JSON support optional via a feature flag (#2300) (#2601)

20282d1 Use required-features for CSV benchmarks/examples (#2602)

12ad8d7 Cast timestamp array to string array with timezone (#2608)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 21.0.0
Bumps arrow from 10.0.0 to 21.0.0.

Changelog

Sourced from arrow's changelog.

21.0.0 (2022-08-18)

Full Changelog

Breaking changes:

Return structured ColumnCloseResult (#2465) #2466 [parquet] (tustvold)

Push ChunkReader into SerializedPageReader (#2463) #2464 [parquet] (tustvold)

Revise FromIterator for Decimal128Array to use Into instead of Borrow #2442 [parquet] [arrow] (viirya)

Use Fixed-Length Array in BasicDecimal new and raw_value #2405 [arrow] (HaoYang670)

Remove deprecated ParquetWriter #2380 [parquet] (tustvold)

Remove deprecated SliceableCursor and InMemoryWriteableCursor #2378 [parquet] (tustvold)

Implemented enhancements:

add into_inner method to ArrowWriter #2491 [parquet]

Remove byteorder dependency #2472 [parquet]

Return Structured ColumnCloseResult from GenericColumnWriter::close #2465 [parquet]

Push ChunkReader into SerializedPageReader #2463 [parquet]

Support SerializedPageReader::skip_page without OffsetIndex #2459 [parquet]

Support Time64/Time32 comparison #2457 [arrow]

Revise FromIterator for Decimal128Array to use Into instead of Borrow #2441 [parquet]

Support RowFilter withinParquetRecordBatchReader #2431 [parquet]

Remove the field StructBuilder::len #2429 [arrow]

Standardize creation and configuration of parquet --> Arrow readers ( ParquetRecordBatchReaderBuilder) #2427 [parquet]

Use OffsetIndex to Prune IO in ParquetRecordBatchStream #2426 [parquet]

Support peek_next_page and skip_next_page in InMemoryPageReader #2406 [parquet]

Support casting from Utf8/LargeUtf8 to Binary/LargeBinary #2402 [arrow]

Support casting between Decimal128 and Decimal256 arrays #2375 [arrow]

Combine multiple selections into the same batch size in skip_records #2358 [parquet]

Add API to change timezone for timestamp array #2346 [arrow]

Change the output of read_buffer Arrow IPC API to return Result<_> #2342 [arrow]

Allow skip_records in GenericColumnReader to skip across row groups #2331 [parquet]

Optimize the validation of Decimal256 #2320 [arrow]

Implement Skip for DeltaBitPackDecoder #2281 [parquet]

Changes to ParquetRecordBatchStream to support row filtering in DataFusion #2270 [parquet]

Add ArrayReader::skip_records API #2197 [parquet]

Fixed bugs:

Panic in SerializedPageReader without offset index #2503 [parquet]

MapArray columns don't handle null values correctly #2484 [arrow]

There is no compiler error when using an invalid Decimal type. #2440 [arrow]

Flight SQL Server sends incorrect response for DoPutUpdateResult #2403 [arrow-flight]

AsyncFileReaderNo Longer Object-Safe #2372 [parquet]

StructBuilder Does not Verify Child Lengths #2252 [arrow]

Closed issues:

Combine DecimalArray validation #2447 [arrow]

... (truncated)

Commits

68934f0 Fix rustfmt commands (#2506)

7ce6ba4 Update create-tarball.sh and verify_release.sh script to skip object_store ...

dacd354 Prepare for release 21.0.0 -- update version and CHANGELOG.md (#2483)

526a2b8 Fix bug in page skipping (#2504)

c34f07f Fix MapArrayReader (#2484) (#1699) (#1561) (#2500)

f3afdd2 Add API to Retrieve Finished Writer from Parquet Writer (#2498)

15f42b2 RFC: Simplify decimal (#2440) (#2477)

e60eef3 parquet-read: add support to read parquet data from stdin (#2482)

a835ba0 mention object-store in front page doc (#2499)

97b4f65 Derive Copy,Clone for BasicDecimal (#2495)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 20.0.0
Bumps arrow from 10.0.0 to 20.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

19.0.0 (2022-07-22)

Full Changelog

Breaking changes:

Rename DecimalArray``/DecimalBuilder to Decimal128Array/Decimal128Builder #2101 [arrow]

Change builder append methods to be infallible where possible #2103 [parquet] [arrow] (jhorstmann)

Return reference from UnionArray::child (#2035) #2099 [arrow] (tustvold)

Remove preserve_order feature from serde_json dependency (#2095) #2098 [parquet] [arrow] (tustvold)

Rename weekday and weekday0 kernels to to num_days_from_monday and num_days_since_sunday #2066 [arrow] (alamb)

Remove null_count from write_batch_with_statistics #2047 [parquet] (tustvold)

Implemented enhancements:

Use total_cmp from std #2130 [arrow]

Permit parallel fetching of column chunks in ParquetRecordBatchStream #2110 [parquet]

The GenericBinaryBuilder should use buffer builders directly. #2104 [arrow]

Pass generate_decimal256_case arrow integration test #2093 [arrow]

Rename weekday and weekday0 kernels to to num_days_from_monday and days_since_sunday #2065 [arrow]

Improve performance of filter_dict #2062 [arrow]

Improve performance of set_bits #2060 [arrow]

Lazily materialize the null buffer builder of BooleanBuilder #2058 [arrow]

BooleanArray::from_iter should omit validity buffer if all values are valid #2055 [arrow]

FFI_ArrowSchema should set DICTIONARY_ORDERED flag if a field's dictionary is ordered #2049 [arrow]

Support peek_next_page() and skip_next_page in SerializedPageReader #2043 [parquet]

Support FFI / C Data Interface for MapType #2037 [arrow]

The DecimalArrayBuilder should use FixedSizedBinaryBuilder #2026 [arrow]

... (truncated)

Commits

30c94db Fix cargo publish (#2340)

83fab37 Remove vestigal object_store/.circleci/ (#2337)

9fde389 Update version to 20.0.0 and update CHANGELOG (#2323)

b1e2bd9 more const evaluations for list array (#2327)

3ed0e28 temporal conversion functions should work on negative input properly (#2326)

50d1e5f MINOR: make capitalization of CI jobs consistent (#2333)

87b19f8 MINOR: remove unused comment_bot.yml from CI scripts (#2334)

b8fd432 Don't hydrate string dictionaries when writing to parquet (#1764) (#2322)

6859efa Misc cleanup (#2330)

4a3919b Add typed dictionary (#2136) (#2297)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 19.0.0
Bumps arrow from 10.0.0 to 19.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

18.0.0 (2022-07-08)

Full Changelog

Breaking changes:

Fix several bugs in parquet writer statistics generation, add EnabledStatistics to control level of statistics generated #2022 [parquet] (tustvold)

Add page index reader test for all types and support empty index. #2012 [parquet] (Ted-Jiang)

Add Decimal256Builder and Decimal256Array; Decimal arrays now implement BasicDecimalArray trait #2000 [parquet] [arrow] (viirya)

Simplify ColumnReader::read_batch #1995 [parquet] [arrow] (tustvold)

Remove PrimitiveBuilder::finish_dict (#1978) #1980 [arrow] (tustvold)

Disallow cast from other datatypes to NullType #1942 [arrow] (liukun4515)

Add column index writer for parquet #1935 [parquet] (liukun4515)

Implemented enhancements:

Add DataType::Dictionary support to subtract_scalar, multiply_scalar, divide_scalar #2019 [arrow]

Support DictionaryArray in add_scalar kernel #2017 [arrow]

Enable column page index read test for all types #2010 [parquet]

Simplify FixedSizeBinaryBuilder #2007 [arrow]

Support Decimal256Builder and Decimal256Array #1999 [arrow]

Support DictionaryArray in unary kernel #1989 [arrow]

Add kernel to quickly compute comparisons on Arrays #1987 [arrow]

Support DictionaryArray in divide kernel #1982 [arrow]

Implement Into<ArrayData> for T: Array #1979 [arrow]

Support DictionaryArray in multiply kernel #1972 [arrow]

Support DictionaryArray in subtract kernel #1970 [arrow]

Declare DecimalArray::length as a constant #1967 [arrow]

... (truncated)

Commits

c3e019f Update version to 19.0.0 and add CHANGELOG (#2120)

567c457 Use total_cmp from std (#2131)

a9fa1b4 Simplify null mask preservation in parquet reader (#2116)

5e3facf fix clippy (#2124)

f62f26a Improve docstrings + examples for as_primitive_array cast functions (#2114)

db91031 Fix logical merge conflict (#2121)

1805b30 Update GenericBinaryBuilder to use buffer builders directly. (#2117)

fd38664 Pass decimal256 integration test (#2094)

576069a Add ArrayWriter indirection (#1764) (#2091)

be0d34d Add get_byte_ranges method to AsyncFileReader trait (#2115)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 18.0.0
Bumps arrow from 10.0.0 to 18.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

17.0.0 (2022-06-24)

Full Changelog

Breaking changes:

Add validation to RecordBatch for non-nullable fields containing null values #1890 [arrow] (andygrove)

Rename ArrayData::validate_dict_offsets to ArrayData::validate_values #1889 [arrow] (frolovdev)

Add Decimal128 API and use it in DecimalArray and DecimalBuilder #1871 [parquet] [arrow] (viirya)

Mark typed buffer APIs safe (#996) (#1027) #1866 [parquet] [arrow] (tustvold)

Implemented enhancements:

add a small doc example showing ArrowWriter being used with a cursor #1927 [parquet]

Support cast to/from NULL and DataType::Decimal #1921 [arrow]

Add Decimal256 API #1913 [arrow]

Add DictionaryArray::key function #1911 [arrow]

Support specifying capacities for ListArrays in MutableArrayData #1884 [arrow]

Explicitly declare the features used for each dependency #1876 [parquet] [arrow] [arrow-flight]

Add Decimal128 API and use it in DecimalArray and DecimalBuilder #1870 [arrow]

PrimitiveArray::from_iter should omit validity buffer if all values are valid #1856 [arrow]

Add from(v: Vec<Option<&[u8]>>) and from(v: Vec<&[u8]>) for FixedSizedBInaryArray #1852 [arrow]

Add Vec-inspired APIs to BufferBuilder #1850 [arrow]

PyArrow intergation test for C Stream Interface #1847 [arrow]

Add nilike support in comparison #1845 [arrow]

Split up arrow::array::builder module #1843 [arrow]

Add quarter support in temporal kernels #1835 [arrow]

Rename ArrayData::validate_dictionary_offset to ArrayData::validate_values #1812 [arrow]

... (truncated)

Commits

330505c Prepare for 18.0.0 release: Update CHANGELOG and versions (#2023)

5a76697 Fix record delimiting on row group boundaries (#2025) (#2027)

ca1bfb8 Add Decimal256Builder and Decimal256Array (#2000)

d70c2ea Consistent case in Index enumeration (#2029)

373ac81 Add page index reader test for all types and support empty index. (#2012)

9333a85 Don't generate empty google.protobuf.rs (#2005)

58848c4 Fix parquet test_common feature flags (#2003)

d9dbf72 Fix several bugs in parquet writer statistics generation, add `EnabledStatist...

7542f7d Add builder style APIs For Field: with_name, with_data_type and `with_n...

ef02bf8 Support dictionary array for divide kernel (#1983)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 17.0.0
Bumps arrow from 10.0.0 to 17.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

16.0.0 (2022-06-10)

Full Changelog

Breaking changes:

Seal ArrowNativeType and OffsetSizeTrait for safety (#1028) #1819 [arrow] (tustvold)

Improve API for csv::infer_file_schema by removing redundant ref #1776 [arrow] (tustvold)

Implemented enhancements:

List equality method should work on empty offset ListArray #1817 [arrow]

Command line tool for convert CSV to Parquet #1797 [parquet]

IPC writer should write validity buffer for UnionArray in V4 IPC message #1793 [arrow]

Add function for row alignment with page mask #1790 [parquet]

Rust IPC Read should be able to read V4 UnionType Array #1788 [arrow]

combine_option_bitmap should accept arbitrary number of input arrays. #1780 [arrow]

Add substring_by_char kernels for slicing on character boundaries #1768 [arrow]

Support reading PageIndex from column metadata #1761 [parquet]

Support casting from DataType::Utf8 to DataType::Boolean #1740 [arrow]

Make current position available in FileWriter. #1691 [parquet]

Support writing parquet to stdout #1687 [parquet]

Fixed bugs:

Incorrect Offset Validation for Sliced List Array Children #1814 [arrow]

Parquet Snappy Codec overwrites Existing Data in Decompression Buffer #1806 [parquet]

flight_data_to_arrow_batch does not support RecordBatches with no columns #1783 [arrow-flight]

... (truncated)

Commits

9f7b600 Update version and changelog for version 17.0.0 (#1926)

c5cf2f2 add readme (#1940)

d52be30 Set adjusted to UTC if UTC timezone (#1932) (#1937)

9e8c1bf Support casting NULL to/from Decimal (#1922)

a901927 Split up parquet::arrow::array_reader (#1483) (#1933)

27963e7 Update indexmap dependency (#1929)

f0df5e0 Add Decimal256 API (#1914)

f8afc14 Add ArrowWriter doctest (#1927) (#1930)

041e510 Complete and fixup split of arrow::array::builder module (#1843) (#1928)

c65bbc5 replace checked_add(sub).unwrap() with +(-) (#1924)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 1
chore(deps): bump arrow from 10.0.0 to 27.0.0
Bumps arrow from 10.0.0 to 27.0.0.

Changelog

Sourced from arrow's changelog.

Historical Changelog

26.0.0 (2022-10-28)

Full Changelog

Breaking changes:

Cast Timestamps to RFC3339 strings #2934

Remove Unused NativeDecimalType #2945 [arrow] (tustvold)

Format Timestamps as RFC3339 #2939 [arrow] (waitingkuo)

Update flatbuffers to resolve RUSTSEC-2021-0122 #2895 [arrow] (tustvold)

replace from_timestamp by from_timestamp_opt #2894 [arrow] (waitingkuo)

Implemented enhancements:

Optimized way to count the numbers of true and false values in a BooleanArray #2963 [arrow]

Add pow to i256 #2954 [arrow]

Write Generic Code over [Large]BinaryArray and [Large]StringArray #2946 [arrow]

Add Page Row Count Limit #2941 [parquet]

prettyprint to show timezone offset for timestamp with timezone #2937 [arrow]

Cast numeric to decimal256 #2922 [arrow]

Add freeze_with_dictionary API to MutableArrayData #2914 [arrow]

Support decimal256 array in sort kernels #2911 [arrow]

support [+/-]hhmm and [+/-]hh as fixedoffset timezone format #2910 [arrow]

Cleanup decimal sort function #2907 [arrow]

replace from_timestamp by from_timestamp_opt #2892 [arrow]

Move Primitive arity kernels to arrow-array #2787 [arrow]

add overflow-checking for negative arithmetic kernel #2662 [arrow]

Fixed bugs:

... (truncated)

Commits

94565bc Update version to 27.0.0 and add changelog (#3089)

561f63a Improved UX of creating TimestampNanosecondArray with timezones (#3088)

02a3f5c Move CSV test data (#3044) (#3051)

0139682 Remove unused range module (#3085)

5226258 Make RowSelection::intersection a member function (#3084)

885a361 feat: add OwnedRow (#3079)

9f14683 Move intersect_row_selections from datafusion to arrow-rs. (#3047)

4dd7fea Update parquet to depend on arrow subcrates (#3028)

132152c Update arrow-flight subcrates (#3044) (#3052)

8d364fe early type checks in RowConverter (#3080)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 0
chore(deps): bump web-sys from 0.3.59 to 0.3.60
Bumps web-sys from 0.3.59 to 0.3.60.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 0
chore(deps): bump wasm-bindgen from 0.2.82 to 0.2.83
Bumps wasm-bindgen from 0.2.82 to 0.2.83.

Commits

eba691f Bumped crate versions (#3079)

893812f web-sys: include VideoFrame in texImage* functions (#3077)

eac9722 Export initSync for no-module builds (#3075)

595b04b Trigger warnings for unused wasm-bindgen attributes (#3073)

f82f5c5 Refactor: normalized encoding of ImportModule (#3068)

edc5adf Put FontFace constructor to Immutable slice whitelist (#3072)

5c28993 Update WebGPU's webidl (#3067)

e47dda5 web-sys: Added Streams API (#3065)

699e788 Revert "web-sys: Added full Streams WebIDL (#3046)" (#3064)

cb94b43 Add TryFrom\<JsValue> for [iu](64|128) (#3058)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 0
chore(deps): bump js-sys from 0.3.59 to 0.3.60
Bumps js-sys from 0.3.59 to 0.3.60.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 0
chore(deps): bump wasm-bindgen-test from 0.3.32 to 0.3.33
Bumps wasm-bindgen-test from 0.3.32 to 0.3.33.

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies
opened by dependabot[bot] 0

Owner

Dominik Moritz

Professor at CMU (@cmudig) and researcher at @apple. PhD in Computer Science from the University of Washington (@uwdata, @uwdb). Co-creator @vega, @streamlit.

Dominik Moritz

GitHub

This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to the arrow format.

Arrow2-derive - derive for Arrow2 This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to th

Jorge Leitao

29 Dec 27, 2022

An experimental implementation of Arc against Apache Datafusion

box This is an experimental repository to perform a proof of concept replacement of the Apache Spark executor for Arc with Apache DataFusion. This is

tripl.ai

1 Nov 26, 2021

Pure Rust library for Apache ZooKeeper built on tokio

zookeeper-async Async Zookeeper client written 100% in Rust, based on tokio. This library is intended to be equivalent with the official (low-level) Z

Kamil Rojewski

16 Dec 16, 2022

🥸 Learning WebAssembly

learn-wasm # Initialize the project # https://github.com/lencx/create-xc-app npm init xc-app learn-wasm -t wasm-react Awesome Lists WebAssembly入门 Exam

Len C...

86 Dec 27, 2022

Async executor for WebAssembly

There are a number of async task executors available in Rust's ecosystem. However, most (if not all?) of them rely on primitives that might not be available or optimal for WebAssembly deployment at the time.

wasm.rs

65 Dec 31, 2022

An n-tuple pendulum simulator in Rust + WebAssembly (and JavaScript)

An n-tuple pendulum simulator in Rust + WebAssembly (and JavaScript) Remaking this n-tuple pendulum simulator moving the math to Rust ?? and WebAssemb

Travis Taylor

27 Feb 19, 2022

A rollup plugin that compile Rust code into WebAssembly modules

rollup-plugin-rust tl;dr -- see examples This is a rollup plugin that loads Rust code so it can be interop with Javascript base project. Currently, th

Fahmi Akbar Wildana

37 Aug 1, 2022

Continuous runtime observablity SDKs to monitor WebAssembly code.

Observe Observe is an observability SDK for WebAssembly. At the moment we support wasmtime hosts and we output opentelemetry data to stdout. We plan t

Dylibso

4 Jun 8, 2023

Cargo wrapper for working with Webassembly wasi(x).

cargo-wasix A cargo subcommand that wraps regular cargo commands for compiling Rust code to wasix, a superset of Websassembly wasi with additional fun

WASIX

19 Jun 12, 2023

Distributed compute platform implemented in Rust, and powered by Apache Arrow.

Ballista: Distributed Compute Platform Overview Ballista is a distributed compute platform primarily implemented in Rust, powered by Apache Arrow. It

Ballista

2.3k Jan 3, 2023

Distributed compute platform implemented in Rust, and powered by Apache Arrow.

Ballista: Distributed Compute Platform Overview Ballista is a distributed compute platform primarily implemented in Rust, powered by Apache Arrow. It

Ballista

2.3k Jan 3, 2023

A Rust DataFrame implementation, built on Apache Arrow

Rust DataFrame A dataframe implementation in Rust, powered by Apache Arrow. What is a dataframe? A dataframe is a 2-dimensional tabular data structure

Wakahisa

287 Nov 11, 2022

Official Rust implementation of Apache Arrow

Native Rust implementation of Apache Arrow Welcome to the implementation of Arrow, the popular in-memory columnar format, in Rust. This part of the Ar

The Apache Software Foundation

1.3k Jan 9, 2023

Apache Arrow DataFusion and Ballista query engines

DataFusion is an extensible query execution framework, written in Rust, that uses Apache Arrow as its in-memory format.

The Apache Software Foundation

2.9k Jan 2, 2023

Fill Apache Arrow record batches from an ODBC data source in Rust.

arrow-odbc Fill Apache Arrow arrays from ODBC data sources. This crate is build on top of the arrow and odbc-api crate and enables you to read the dat

Markus Klein

21 Dec 27, 2022

Generated Ryst of Apache Arrow spec

Arrow generated IPC format The generated flatbuffers code for Rust. Note that these files suffered modifications because flatbuffers is unable to comp

null

14 Nov 30, 2022

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

Apache Arrow Powering In-Memory Analytics Apache Arrow is a development platform for in-memory analytics. It contains a set of technologies that enabl

The Apache Software Foundation

10.9k Jan 6, 2023

Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow

Welcome to arrowdantic Arrowdantic is a small Python library backed by a mature Rust implementation of Apache Arrow that can interoperate with Parquet

Jorge Leitao

52 Dec 21, 2022

Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.

Polars Python Documentation | Rust Documentation | User Guide | Discord | StackOverflow Blazingly fast DataFrames in Rust, Python & Node.js Polars is

null

11.8k Jan 8, 2023

Fast regex in Rust for Apache Arrow, compiled to WASM

Rust regex in wasm I have been looking for a fast regular expression library in Javascript that runs on Apache Arrow for a few years. Arrow uses UTF-8

Nomic AI

3 May 3, 2023