Parallel iterator processing library for Rust

Overview

Parallel iterator processing library for Rust

I keep needing one, so I wrote it.

See [IteratorExt] for supported operations.

In essence, if you have:

# fn step_a(x: usize) -> usize {
#   x * 7
# }
# 
# fn filter_b(x: &usize) -> bool {
#   x % 2 == 0
# }
# 
# fn step_c(x: usize) -> usize {
#   x + 1
# }
assert_eq!(
  (0..10)
    .map(step_a)
    .filter(filter_b)
    .map(step_c).collect::<Vec<_>>(),
    vec![1, 15, 29, 43, 57]
);

You can change it to:

use dpc_pariter::IteratorExt;
# fn step_a(x: usize) -> usize {
#   x * 7
# }
# 
# fn filter_b(x: &usize) -> bool {
#   x % 2 == 0
# }
# 
# fn step_c(x: usize) -> usize {
#   x + 1
# }
assert_eq!(
  (0..10)
    .map(step_a)
    .filter(filter_b)
    .parallel_map(step_c).collect::<Vec<_>>(),
    vec![1, 15, 29, 43, 57]
);

and it will run faster (conditions apply), because step_c will run in parallel on multiple-threads.

Notable features

  • order preserving
  • lazy, somewhat like a normal iterators
  • panic propagation
  • backpressure control
  • custom thread-pool support
  • other knobs to control performance & resource utilization

When to use and alternatives

This library is a good general purpose solution. When you have a chain of iterator steps, and would like to process one or some of them in parallel to speed it up, this should be a drop-in replacement.

Sending iterator items through channels is fast, but not free. Make sure to parallelize operations that are heavy enough to justify overhead of sending data through channels. E.g. operations involving IO or some CPU-heavy computation.

When you have a lot items already stored in a collection, that you want to "roll over and perform some mass computation" you probably want to use rayon instead. It's a library optimized for parallelizing processing of whole chunks of larger set of data. Because of that converting rayon's iterators back to ordered sequencial iterator is non-trivial.

There are alternative libraries somewhat like this, but I did not find any that I'd like API and/or implementation wise, so I wrote my own.

Status & plans

I keep needing this exact functionality, so I've cleaned up my ad-hoc code and put it in a proper library. I would actually prefer if someone else did it. :D I might add more features in the future, and I am happy to accept PRs.

I'm open to share ownership & maintenance, or even "donate it".

Owner
Dawid Ciężarkiewicz
Born 1985, programming since ~1993, on Linux ~1998, professionally ~2003, on github ~2008, with Rust ~2013. Deep&wide generalist, FOSS enthusiast.
Dawid Ciężarkiewicz
Parallel finance a decentralized lending protocol built on top of the Polkadot ecosystem. Our unique approach will allow users to earn "double interests" from staking and lending their tokens simultaneously.

Parallel Finance A new Cumulus-based Substrate node, ready for hacking ?? Getting Started Follow these steps to get started with the Cumulus Template

parallel-finance 47 Nov 20, 2021
A library to listen to global hotkeys in Rust

Rust Hotkey A library to listen to global hotkeys in Rust How to use See the examples folder for how to use this library. OS Support This lib aims to

James Birtles 31 Oct 10, 2021
minimalistic command launcher in rust

rrun Note: Apart from the occasional fix, this project is not actively developed anymore. rrun works fine and should run/compile for the time being on

null 98 Nov 17, 2021
Yet another fancy watcher. (Rust)

funzzy Yet another fancy watcher. (Inspired by antr / entr) Configure execution of different commands using semantic yaml. # .watch.yaml # list here a

Cristian Oliveira 158 Nov 17, 2021
A modern replacement for ps written in Rust

procs procs is a replacement for ps written in Rust. Documentation quick links Features Platform Installation Usage Configuration Features Output by t

null 2.4k Nov 21, 2021
A more intuitive version of du in rust

Dust du + rust = dust. Like du but more intuitive. Why Because I want an easy way to see where my disk is being used. Demo Install Cargo cargo install

andy.boot 3.2k Nov 26, 2021
Blazing 💥 fast terminal-ui for git written in rust 🦀

Blazing fast terminal client for git written in Rust Features Fast and intuitive keyboard only control Context based help (no need to memorize tons of

Stephan Dilly 6.5k Nov 25, 2021
A simple and fast download accelerator, written in Rust

zou A simple and fast download accelerator, written in Rust Zou is a Snatch fork by @k0pernicus. Snatch is a fast and interruptable download accelerat

Antonin Carette 165 Nov 17, 2021
Fuzzy Finder in rust!

Life is short, skim! Half of our life is spent on navigation: files, lines, commands… You need skim! It is a general fuzzy finder that saves you time.

Jinzhou Zhang 3k Nov 29, 2021
A bash-like Unix shell written in Rust

Cicada Unix Shell Cicada is a simple Unix shell written in Rust. Documents Install cicada Environment Variables Cicada Builtins Completion RC File His

Hugo Wang 864 Nov 22, 2021
Performs distributed command execution, written in Rust w/ Tokio

Concurr: Distributed and Concurrent Command Execution, in Rust This project is dual licensed under MIT and Apache 2.0. Originally inspired by the GNU

Michael Murphy 80 Oct 21, 2021
🔮 Futuristic take on hexdump, made in Rust.

hex (hx) Futuristic take on hexdump. hx accepts a file path as input and outputs a hexadecimal colorized view to stdout. $ hx tests/files/alphanumeric

Julian Sitkevich 302 Nov 22, 2021
Cross-platform Rust rewrite of the GNU coreutils

uutils coreutils uutils is an attempt at writing universal (as in cross-platform) CLI utilities in Rust. This repository is intended to aggregate GNU

null 9.1k Nov 24, 2021
A TUI system monitor written in Rust

NO LONGER MAINTAINED. For a similar program, check out https://github.com/ClementTsang/bottom. ytop Another TUI based system monitor, this time in Rus

Caleb Bassi 2k Nov 18, 2021
A rust layered configuration loader with zero-boilerplate configuration management.

salak A layered configuration loader with zero-boilerplate configuration management. About Features Placeholder Key Convension Cargo Features Default

Daniel YU 30 Oct 6, 2021
Untrusted IPC with maximum performance and minimum latency. On Rust, on Linux.

Untrusted IPC with maximum performance and minimum latency. On Rust, on Linux. When is this Rust crate useful? Performance or latency is crucial, and

null 36 Nov 5, 2021
An over-engineered rewrite of pipes.sh in Rust

pipes-rs An over-engineered rewrite of pipes.sh in Rust Installlation macOS Install using Homebrew or download manually from releases. $ brew install

Lucas 246 Nov 22, 2021
Trup-rewrite in rust! Finally!

Trup, but Rust! A Discord bot for the Unixporn community Now written in a good language! Dependencies Rust nightly sqlx-cli (if you need to change the

r/unixporn 70 Nov 30, 2021
A collection of small Rust programs for doing weird things

This is a repo of small programs, proof of concepts, or templates written in Rust that relate in some way to hacking and/or CTF. I think Rust is real

d3npa 21 Aug 5, 2021