Efflux
Efflux is a set of Rust interfaces for MapReduce and Hadoop Streaming. It enables Rust developers to run batch jobs on Hadoop infrastructure whilst staying with the efficiency and safety they're used to.
Initially written to scratch a personal itch, this crate offers simple traits to mask the internals of working with Hadoop Streaming which lend themselves well to writing jobs quickly. Functionality is handed off to macros where possible to provide compile time guarantees, and any other functionality is kept simple to avoid overhead wherever possible.
Installation
Efflux is available on crates.io as a library crate, so you only need to add it as a dependency:
[dependencies]
efflux = "2.0"
You can then gain access to everything relevant using the prelude module of Efflux:
use efflux::prelude::*;
Usage
Efflux comes with a handy template to help generate new projects, using the kickstart tool. You can simply use the commands below and follow the prompt to generate a new project skeleton:
# install kickstart
$ cargo install kickstart
# create a project from the template
$ kickstart -s examples/template https://github.com/whitfin/efflux
If you'd rather not use the templating tool, you can always work from the examples found in this repository. A good place to start is the traditional wordcount example.
Testing
Testing your binaries is actually fairly simple, as you can simulate the Hadoop phases using a basic UNIX pipeline. The following example replicates the Hadoop job flow and generates output that matches a job executed with Hadoop itself:
# example Hadoop task invocation
$ hadoop jar hadoop-streaming-2.8.2.jar \
-input <INPUT> \
-output <OUTPUT> \
-mapper <MAPPER> \
-reducer <REDUCER>
# example simulation run via UNIX utilities
$ cat <INPUT> | <MAPPER> | sort -k1,1 | <REDUCER> > <OUTPUT>
This can be tested using the wordcount example to confirm that the outputs are indeed the same. There may be some cases where output differs, but it should be sufficient for many cases.
2 Jul 5, 2022
1 Feb 15, 2022
3 Feb 25, 2024
9.3k Jan 4, 2023
5.5k Jan 1, 2023
29 Mar 11, 2024
39 Aug 27, 2022
648 Dec 26, 2022
2 Feb 10, 2022
10 Dec 1, 2022
189 Jan 8, 2023
1.1k Jan 8, 2023
4 Jan 10, 2023
5 Jan 8, 2023
1.9k Dec 22, 2022
2 Jan 8, 2022
15 Jan 4, 2023
205 Dec 29, 2022
4 Feb 27, 2024
173 Dec 28, 2022
44 Oct 4, 2022
69 Dec 13, 2022
32 Dec 1, 2022
3 Dec 3, 2021
1.1k Jan 8, 2023
902 Jan 2, 2023
249 Dec 28, 2022
1 Oct 23, 2021
60 Oct 22, 2022