Testing out if Rust can be used for a normal Data Engineering Pipeline.

Overview

RustForDataPipelines

Testing out if Rust can be used for a normal Data Engineering Pipeline.

Check out the full blog post here. https://www.confessionsofadataguy.com/using-rust-to-write-a-data-pipeline-thoughts-musings/

The idea was to try things like HTTP, File Systems, and Database calls (Postgres) out with Rust.

These are normal data engineering functions, is it too verbose to write data pipelines in Rust? The data flow is as follows

  • download .zip file.
  • unpack the .zip file into .csv, mess with files.
  • connect Rust to postgres
  • Process CSV file into postgres
  • Run some sql and push some rows
You might also like...
Cucumber testing framework for Rust. Fully native, no external test runners or dependencies.
Cucumber testing framework for Rust. Fully native, no external test runners or dependencies.

Cucumber testing framework for Rust An implementation of the Cucumber testing framework for Rust. Fully native, no external test runners or dependenci

Rust library that provides helpers for testing resilience of I/O operations.

partial-io Helpers for testing I/O behavior with partial, interrupted and blocking reads and writes. This library provides: PartialRead and PartialWri

botwork is a single-binary, generic and open-source automation framework written in Rust for acceptance testing & RPA

botwork botwork is a single-binary, generic and open-source automation framework written in Rust for acceptance testing, acceptance test driven develo

A dead simple functional testing tool for command line applications

Pharaoh : build that test pyramid! What it is Pharaoh is a dead simple, no permission needed, functional test runner for command line applications, wr

Snapshot testing for a herd of CLI tests

trycmd Snapshot testing for a herd of CLI tests trycmd aims to simplify the process for running a large collection of end-to-end CLI test cases, takin

Croc-look is a tool to make testing and debuging proc macros easier

croc-look croc-look is a tool to make testing and debuging proc macros easier by these two features Printing the implementation specific generated cod

A workflow tool for quickly running / testing something you are working on

runfast What is it? This is a program intended to be run in a project directory to set up a project run command, and remember it so we dont have to ty

A library and binary for testing unhooking ntdll by identifying hooks via in-memory disassembly
A library and binary for testing unhooking ntdll by identifying hooks via in-memory disassembly

(First Public?) Sample of unhooking ntdll (All Exports & IAT imports) hooks in Rust using in-memory disassembly, avoiding direct syscalls and all hooked functions (incl. hooked NtProtectVirtualMemory)

Concurrent and multi-stage data ingestion and data processing with Rust+Tokio

TokioSky Build concurrent and multi-stage data ingestion and data processing pipelines with Rust+Tokio. TokioSky allows developers to consume data eff

Owner
Daniel B
Data Engineer. Data lover. Data warehouse expert. Python and SQL are the spice of life.
Daniel B
A reverse-engineering of Speedball 2 for the Sega Megadrive

Reverse engineering of Speedball 2 for Sega Megadrive (Genesis) This repo is a reverse engineering of Speedball 2, using Ghidra to reverse the assembl

Simon Frankau 4 Jan 6, 2023
The application that can be used for personal usage to manage jira from terminal.

Jira Terminal This application can be used for personal usage to manage jira from terminal. Installation This application can be used in multiple plat

Amrit Ghimire 109 Dec 16, 2022
A high-performance WebSocket integration library for streaming public market data. Used as a key dependency of the `barter-rs` project.

Barter-Data A high-performance WebSocket integration library for streaming public market data from leading cryptocurrency exchanges - batteries includ

Barter 23 Feb 3, 2023
Basic template for an out-of-tree Linux kernel module written in Rust.

Rust out-of-tree module This is a basic template for an out-of-tree Linux kernel module written in Rust. Please note that: The Rust support is experim

Rust for Linux 118 Dec 26, 2022
A utility written in Rust for dumping binary information out of Mach-O files inspired by objdump

Mach-O Dump (macho-dump) An objdump like tool for exploring and manipulating Mach-O files. Note: This project is in an early stage and most of the fea

Irides 2 Oct 17, 2022
Works out if this is running from inside a shell, and if so, which one.

pshell pshell answers the question "Is my application running in a shell, and if so, which one". Example: you are installing something and want to mak

Alec Brown 2 Nov 3, 2022
🪐 An out-of-this-world greeter for your terminal

Draconis ?? An out-of-this-world greeter for your terminal Requirements pacman-contrib for pacman Important This program uses the openweathermap API f

Mars 11 Dec 17, 2022
🐎 Just a simple cross-platform neofetch for all the bronies out there.

⚠️ (WIP) This project is not ready for any serious use right now. A cross-platform command-line interface (CLI) tool written in Rust to display system

Jakub 4 Dec 15, 2022
Work out how far apart things should be (very quickly)

Kern determiner Work out how far apart things should be (very quickly) kerndeterminer is a Rust-based library for determining a kern value between two

Simon Cozens 10 Oct 29, 2022
A simple program for C program IO testing. Written in Rust

A simple program for C program IO testing. Written in Rust, using concurrency to speed up valgrind testing. Make sure to update settings at your first run of the program!

null 1 Feb 22, 2022