🦖 Evolve your fixed length data files into Apache Arrow tables, fully parallelized!

Firelink Data

Last update: Dec 22, 2023

Related tags

Data processing rust cli arrow data-engineering apache-parquet apache-arrow

Overview

🦖 Evolve your fixed length data files into Apache Arrow tables, fully parallelized!

🔎 Overview

...

📦 Installation

The easiest way to install evolution on your system is by using the Cargo package manager.

$ cargo install evolution

Alternatively, you can build from source by cloning this repo and compiling using Cargo.

$ git clone https://github.com/firelink-data/evolution.git
$ cd evolution
$ cargo build --release

📋 License

All code is to be held under a general MIT license, please see LICENSE for specific information.

A new arguably faster implementation of Apache Spark from scratch in Rust

vega Previously known as native_spark. Documentation A new, arguably faster, implementation of Apache Spark from scratch in Rust. WIP Framework tested

2.1k Jan 5, 2023

A highly efficient daemon for streaming data from Kafka into Delta Lake

172 Dec 23, 2022

Perhaps the fastest and most memory efficient way to pull data from PostgreSQL into pandas and numpy. 🚀

flaco Perhaps the fastest and most memory efficient way to pull data from PostgreSQL into pandas and numpy. 🚀 Have a gander at the initial benchmarks

14 Oct 31, 2022

A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

5k Jan 9, 2023

New generation decentralized data warehouse and streaming data pipeline

184 Dec 22, 2022

This library provides a data view for reading and writing data in a byte array.

Docs This library provides a data view for reading and writing data in a byte array. This library requires feature(generic_const_exprs) to be enabled.

2 Nov 2, 2022

AppFlowy is an open-source alternative to Notion. You are in charge of your data and customizations

AppFlowy is an open-source alternative to Notion. You are in charge of your data and customizations. Built with Flutter and Rust.

30.7k Jan 7, 2023

Dig into ClickHouse with TUI interface. PRE ALPHA version, everything will be changed.

chdig Dig into ClickHouse with TUI interface. Motivation The idea is came from everyday digging into various ClickHouse issues. ClickHouse has a appro

5 Feb 10, 2023

A Rust crate that reads and writes tfrecord files

tfrecord-rust The crate provides the functionality to serialize and deserialize TFRecord data format from TensorFlow. Features Provide both high level

22 Nov 3, 2022

Comments

Update padder requirement from 0.1.0 to 1.0.0
Updates the requirements on padder to permit the latest version.

Release notes

Sourced from padder's releases.

v1.0.0

First major release 🔥⚡️

This is the first major release of padder. At face value, it offers any Rust developer an alternative to using the standard library macro format! which has known performance issues due to multiple allocations.

Padder allocates memory once, and then uses that memory for your formatting and padding needs. Check out the docs for examples and more information. The padding trait is currently implemented for string slices &str and generic vectors Vec<T>, with some trait bounds necessary.

Sadly no new contributors for this major release. However, feel free to contribute in any way if you see something that can be added or improved.

📋 Changelog:

[build] bump to major release v0.1.1 → v1.0.0 (firelink-data/padder#13)

[docs] udpate description in readme (firelink-data/padder#11)

[feat] trait bounds for char and u8 on Source trait (firelink-data/padder#10)

[docs] update readme short description (firelink-data/padder#9)

[crate] bump package version to 0.1.1 (firelink-data/padder#8)

[docs] fix typo in readme (firelink-data/padder#7)

[feat] document and improve padding functionality (firelink-data/padder#6)

[build] update package description (firelink-data/padder#5)

[docs] add install instructions and badges (firelink-data/padder#4)

[feat] padding, testing, and benchmarking (firelink-data/padder#3)

[git] dependabot config, ci and tests workflows (firelink-data/padder#2)

[build] add ci and tests workflows (firelink-data/padder#1)

Commits

See full diff in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot show <dependency name> ignore conditions will show all of the ignore conditions of the specified dependency

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies rust
opened by dependabot[bot] 0
Feature/cleanup slicer
SlicerProcessor is now a trait. An SampleSliceProcessor implementation is included that simply adds slices to an output file expected NOT to diff.

pub trait SlicerProcessor { fn set_line_break_handler(&mut self, fn_line_break:FnLineBreak); fn get_line_break_handler(&self) -> FnLineBreak;

fn process(&mut self,slices: Vec<&[u8]>) -> usize;

}
opened by Ignalina 0
Feature/arrow aggrigator

Would be good to have this in main so we dont diverge to much.

improved: made CLAP use 3 subcomands mock,slice,parse Partial impl of parse.

bad news: No unit test

opened by Ignalina 1

🦖 Evolve your fixed length data files into Apache Arrow tables, fully parallelized!

Related tags

Overview

🔎 Overview

📦 Installation

📋 License

You might also like...

A new arguably faster implementation of Apache Spark from scratch in Rust

A highly efficient daemon for streaming data from Kafka into Delta Lake

Perhaps the fastest and most memory efficient way to pull data from PostgreSQL into pandas and numpy. 🚀

A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

New generation decentralized data warehouse and streaming data pipeline

This library provides a data view for reading and writing data in a byte array.

AppFlowy is an open-source alternative to Notion. You are in charge of your data and customizations

Dig into ClickHouse with TUI interface. PRE ALPHA version, everything will be changed.

A Rust crate that reads and writes tfrecord files

Comments

Update padder requirement from 0.1.0 to 1.0.0

v1.0.0

First major release 🔥⚡️

📋 Changelog:

Feature/cleanup slicer

Feature/arrow aggrigator

Owner

Firelink Data

Apache Arrow is a multi-language toolbox for accelerated data interchange and in-memory processing

A Rust DataFrame implementation, built on Apache Arrow

Official Rust implementation of Apache Arrow

Apache Arrow DataFusion and Ballista query engines

Polars is a blazingly fast DataFrames library implemented in Rust using Apache Arrow Columnar Format as memory model.

An AWS Lambda for automatically loading JSON files as they're created into Delta tables

PostQuet: Stream PostgreSQL tables/queries to Parquet files seamlessly with this high-performance, Rust-based command-line tool.

Integration between arrow-rs and extendr

Arrow User-Defined Functions Framework on WebAssembly.

Apache TinkerPop from Rust via Rucaja (JNI)