gzp - Multi-threaded Compression

Overview

⛓️ gzp

Build Status license Version info

Multithreaded encoding.

Why?

This crate provides a near drop in replacement for Write that has will compress chunks of data in parallel and write to an underlying writer in the same order that the bytes were handed to the writer. This allows for much faster compression of data.

Supported Encodings:

Usage / Features

There are no features enabled by default. pargz_default enables pargz and flate2_default. flate2_default in turn uses the default backed for flate2, which, as of this writing is the pure rust backend. parsnap_default is just a wrapper over parsnap, which pulls in the snap dependency.

The following demonstrate common ways to override the features:

Simple way to enable both pargz and parsnap:

[dependencies]
gzp = { version = "*", features = ["pargz_default", "parsnap_default"] }

Use pargz to pull in flate2 and zlib-ng-compat to use the zlib-ng-compat backend for flate2 (most performant).

[dependencies]
gzp = { version = "*", features = ["pargz", "zlib-ng-compat"] }

To use Snap (could also use parsnap_default):

[dependencies]
gzp = { version = "*", no_default_features = true, features = ["parsnap"] }

To use both Snap and Gzip with specific backend:

[dependencies]
gzp = { version = "*", no_default_features = true, features = ["parsnap_default", "pargz", "zlib-ng-compat"] }

Examples

Simple example

use std::{env, fs::File, io::Write};

use gzp::pargz::ParGz;

fn main() {
    let file = env::args().skip(1).next().unwrap();
    let writer = File::create(file).unwrap();
    let mut par_gz = ParGz::builder(writer).build();
    par_gz.write_all(b"This is a first test line\n").unwrap();
    par_gz.write_all(b"This is a second test line\n").unwrap();
    par_gz.finish().unwrap();
}

An updated version of pgz.

use gzp::pargz::ParGz;
use std::io::{Read, Write};

fn main() {
    let chunksize = 64 * (1 << 10) * 2;

    let stdout = std::io::stdout();
    let mut writer = ParGz::builder(stdout).build();

    let stdin = std::io::stdin();
    let mut stdin = stdin.lock();

    let mut buffer = Vec::with_capacity(chunksize);
    loop {
        let mut limit = (&mut stdin).take(chunksize as u64);
        limit.read_to_end(&mut buffer).unwrap();
        if buffer.is_empty() {
            break;
        }
        writer.write_all(&buffer).unwrap();
        buffer.clear();
    }
    writer.finish().unwrap();
}

Same thing but using Snappy instead.

use gzp::parsnap::ParSnap;
use std::io::{Read, Write};

fn main() {
    let chunksize = 64 * (1 << 10) * 2;

    let stdout = std::io::stdout();
    let mut writer = ParSnap::builder(stdout).build();

    let stdin = std::io::stdin();
    let mut stdin = stdin.lock();

    let mut buffer = Vec::with_capacity(chunksize);
    loop {
        let mut limit = (&mut stdin).take(chunksize as u64);
        limit.read_to_end(&mut buffer).unwrap();
        if buffer.is_empty() {
            break;
        }
        writer.write_all(&buffer).unwrap();
        buffer.clear();
    }
    writer.finish().unwrap();
}

Notes

  • Files written with this are just Gzipped blocks catted together and must be read with flate2::bufread::MultiGzDecoder.

Future todos

  • Explore removing Bytes in favor of raw vec
  • Check that block is actually smaller than when it started
  • Update the CRC value with each block written
  • Add a BGZF mode + tabix index generation (or create that as its own crate)
  • Try with https://docs.rs/lzzzz/0.8.0/lzzzz/lz4_hc/fn.compress.html

Benchmarks

All benchmarks were run on the file in ./bench-data/shakespeare.txt catted together 100 times which creates a rough 550Mb file.

The primary takeaway here is that you probably want to give gzp at least 4 threads. 2 threads breaks even with the overhead of orchestrating the multi-threadedness, 4 gives a roughly 3x improvement.

benchmarks

Comments
  • gzp spawns one less threads than CPUs, which hurts performance

    gzp spawns one less threads than CPUs, which hurts performance

    I've run some tests comparing crabz to pigz using the benchmarking setup described in crabz readme. On a 4-core system with no hyperthreading crabz was measurably slower.

    I've profiled both using perf and it turned out that crabz spends the vast majority of the time in zlib compression, so parallelization overhead is not an issue. However, crabz only spawned 3 threads performing compression while pigz spawned 4 compression threads. After passing -p3 to pigz so that it would only spawn 3 compression threads, the compression time became identical to crabz.

    I suspect this is also why you're not seeing any parallelization gains on dual-core systems.

    Technical details

    crabz profile: https://share.firefox.dev/3zeVRxN pigz profile: https://share.firefox.dev/2WeYe4V

    crabz installed via cargo install crabz on a clean Ubuntu 20.04 installation, pigz installed via apt.

    $ hyperfine 'crabz -c 3 < /media/elementary/ssd/large-file.txt' 'pigz -3 < /media/elementary/ssd/large-file.txt'
    Benchmark #1: crabz -c 3 < /media/elementary/ssd/large-file.txt
      Time (mean ± σ):      4.642 s ±  0.351 s    [User: 11.326 s, System: 0.196 s]
      Range (min … max):    4.312 s …  5.465 s    10 runs
     
    Benchmark #2: pigz -3 < /media/elementary/ssd/large-file.txt
      Time (mean ± σ):      3.884 s ±  0.253 s    [User: 14.307 s, System: 0.167 s]
      Range (min … max):    3.556 s …  4.248 s    10 runs
     
    Summary
      'pigz -3 < /media/elementary/ssd/large-file.txt' ran
        1.20 ± 0.12 times faster than 'crabz -c 3 < /media/elementary/ssd/large-file.txt'
    
    opened by Shnatsel 27
  • Please support runtimes other than tokio

    Please support runtimes other than tokio

    I'd love to use gzp in libraries that are already using a different async runtime, and I'd like to avoid adding the substantial additional dependencies of a separate async runtime that isn't otherwise used by the library.

    Would you consider either supporting a thread pool like Rayon, or supporting other async runtimes?

    (If it helps, you could use a channel library like flume that works on any runtime, so that you only need a spawn function.)

    opened by joshtriplett 11
  • Avoid rustc version lock-in

    Avoid rustc version lock-in

    opened by drahnr 7
  • Error when writting with ` ZBuilder::<Bgzf, _>::new().num_threads(8)...`

    Error when writting with ` ZBuilder::::new().num_threads(8)...`

    Hello,

    I am getting an error whenever I use Zbuilder::<Bgzf, _> with more than one thread and I am having trouble figuring out what is going wrong. Here is the error message I am getting:

    thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: ChannelReceive(Disconnected)', /Users/mrvollger/.cargo/registry/src/github.com-1ecc6299db9ec823/gzp-0.9.2/src/par/compress.rs:301:27
    stack backtrace:
       0: rust_begin_unwind
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/std/src/panicking.rs:498:5
       1: core::panicking::panic_fmt
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/core/src/panicking.rs:107:14
       2: core::result::unwrap_failed
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/core/src/result.rs:1690:5
       3: core::result::Result<T,E>::unwrap
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/core/src/result.rs:1018:23
       4: <gzp::par::compress::ParCompress<F> as core::ops::drop::Drop>::drop
                 at /Users/mrvollger/.cargo/registry/src/github.com-1ecc6299db9ec823/gzp-0.9.2/src/par/compress.rs:301:13
       5: core::ptr::drop_in_place<gzp::par::compress::ParCompress<gzp::deflate::Bgzf>>
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/core/src/ptr/mod.rs:188:1
       6: core::ptr::drop_in_place<alloc::boxed::Box<dyn gzp::ZWriter>>
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/core/src/ptr/mod.rs:188:1
       7: core::ptr::drop_in_place<alloc::boxed::Box<dyn std::io::Write>>
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/core/src/ptr/mod.rs:188:1
       8: test::run_split_fastx
                 at ./src/test.rs:72:5
       9: test::main
                 at ./src/test.rs:77:5
      10: core::ops::function::FnOnce::call_once
                 at /rustc/a7e2e33960e95d2eb1a2a2aeec169dba5f73de05/library/core/src/ops/function.rs:227:5
    note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
    

    And here is some code and a test file that recreates this error for me:

    use gzp::deflate::Bgzf;
    use gzp::Compression;
    use gzp::ZBuilder;
    use needletail::{parse_fastx_file, parse_fastx_stdin, parser::LineEnding};
    use std::ffi::OsStr;
    use std::fs::File;
    use std::io::{BufWriter, Write};
    use std::path::{Path, PathBuf};
    
    const BUFFER_SIZE: usize = 128 * 1024;
    
    /// Uses the presence of a `.gz` extension to decide if compression is needed
    pub fn writer(filename: &str) -> Box<dyn Write> {
        let ext = Path::new(filename).extension();
        let path = PathBuf::from(filename);
        let buffer = Box::new(BufWriter::with_capacity(
            BUFFER_SIZE,
            File::create(path).expect("Error: cannot create output file"),
        ));
    
        if ext == Some(OsStr::new("gz")) {
            let writer = ZBuilder::<Bgzf, _>::new()
                .num_threads(8)
                .compression_level(Compression::new(6))
                .from_writer(buffer);
            Box::new(writer)
        } else {
            buffer
        }
    }
    
    /// Split a fasta file across outputs
    pub fn run_split_fastx(files: &[String], infile: &str) {
        // open the output files
        let mut outs = Vec::new();
        for f in files {
            let handle = writer(f);
            outs.push(handle);
        }
        // open reader
        let mut reader = if infile == "-" {
            parse_fastx_stdin().expect("Missing or invalid stdin for fastx parser.")
        } else {
            parse_fastx_file(infile).expect("Missing or invalid stdin for fastx parser.")
        };
        // iterate
        let mut out_idx = 0;
        let mut rec_num = 0;
        while let Some(record) = reader.next() {
            let seq_rec =
                record.unwrap_or_else(|_| panic!("Error reading record number {}", rec_num + 1));
            seq_rec
                .write(&mut outs[out_idx], Some(LineEnding::Unix))
                .unwrap_or_else(|_| panic!("Error writing record number {}", rec_num + 1));
    
            eprintln!("Wrote record number {}", rec_num + 1);
            out_idx += 1;
            rec_num += 1;
            if out_idx == outs.len() {
                out_idx = 0;
            }
        }
        // Close all the files.
        let mut n_out = 0;
        for mut out in outs {
            out.flush()
                .unwrap_or_else(|_| panic!("Error flushing output!"));
            n_out += 1;
            eprintln!("Finished output number {}", n_out);
        }
    }
    
    pub fn main() {
        let infile = "large.test.fa.gz";
        run_split_fastx(&["a.fa".to_string(), "b.fa.gz".to_string()], infile);
    }
    
    

    Hete is the fasta file used in this example code: large.test.fa.gz

    Any help would be greatly appreciated!

    Thanks! Mitchell

    opened by mrvollger 7
  • Performance comparison with rust-snappy

    Performance comparison with rust-snappy

    This looks really interesting, @sstadick! We (optionally) use compression to reduce the size of collated RAD files in our aleinv-fry project. Specifically, we make use of the Snappy frame encoding to allow multiple different threads to compress the data in parallel. It's not quite as fast as writing the raw data to disk, but the overhead is small and the compression benefits are pretty large.

    Do you have any idea how this compares to snap? Specifically, in the multithreaded case, I'd be curious of the time / size tradeoffs.

    Thanks! Rob

    opened by rob-p 6
  • Help with the bgzf reader?

    Help with the bgzf reader?

    Hello,

    I am newish to rust and I have what is probably a very simple question. How do I read in line by line a bgzipped file?

    I have this code which is largely borrowed from crabz (see bottom), but once I have the BgzfSyncReader I am not sure how to iterate over it, or manipulate it in any way.

    Thanks in advance! Mitchell

    /// Get a buffered input reader from stdin or a file
    fn get_input(path: Option<PathBuf>) -> Result<Box<dyn BufRead + Send + 'static>> {
        let reader: Box<dyn BufRead + Send + 'static> = match path {
            Some(path) => {
                if path.as_os_str() == "-" {
                    Box::new(BufReader::with_capacity(BUFFER_SIZE, io::stdin()))
                } else {
                    Box::new(BufReader::with_capacity(BUFFER_SIZE, File::open(path)?))
                }
            }
            None => Box::new(BufReader::with_capacity(BUFFER_SIZE, io::stdin())),
        };
        Ok(reader)
    }
    
    /// Example trying bgzip
    /// ```
    /// use rustybam::myio;
    /// let f = ".test/asm_small.paf.bgz";
    /// myio::test_gbz(f);
    /// ```
    pub fn test_bgz(filename: &str) {
        let ext = Path::new(filename).extension();
        eprintln!("{:?}", ext);
        let pathbuf = PathBuf::from(filename);
        let box_dny_bufread_send = get_input(Some(pathbuf)).unwrap();
        let gbzf_reader = BgzfSyncReader::new(box_dny_bufread_send);
        // How do I now loop over lines?
        for line in gbzf_reader {
            eprintln!(line);
        }
    }
    
    opened by mrvollger 5
  • Please support generating a single gzip-compatible or deflate-compatible stream

    Please support generating a single gzip-compatible or deflate-compatible stream

    Currently, pargz uses many GzEncoder instances, which each emit a gzip header, so decoding it requires MultiGzDecoder. If pargz instead used DeflateEncoder to emit raw deflate streams, with an optional gzip or zlib header in front, the resulting stream would be a single gzip/zlib/deflate file, and wouldn't require MultiGzDecoder.

    I'd love to be able to use pargz in file formats that expect raw deflate, or that expect zlib, or that expect gzip but don't handle multiple gzip headers.

    opened by joshtriplett 3
  • Expose FooterValues sum and amount

    Expose FooterValues sum and amount

    This is needed so we know how many bytes for the output buffer is needed. FooterValues is returned by Bgzf::bgzf.get_footer_values (in BlockFormatSpec).

    opened by nh13 0
  • Implement single threaded decompression for block formats

    Implement single threaded decompression for block formats

    Currently Mgzip and Bgzf both fall back to using MultiGzDecoder when 0 threads are selected for decompression.

    To better leverage libdeflater a single threaded decoder for block formats should be provided.

    enhancement 
    opened by sstadick 0
  • [feature] rework backend + oversubscribe writer

    [feature] rework backend + oversubscribe writer

    • Fix wording on thread allocation. num_threads now specifies how many compression threads to spawn and doesn't account for the writer thread since that can be oversubscribed.
    • Dropped rayon in favor of raw threads for performance
    • Made queue sizes larger

    Overall, this version is faster and leaner than the previous version.

    opened by sstadick 0
  • [feature] zlib, raw deflate, full deflate streams

    [feature] zlib, raw deflate, full deflate streams

    This PR reworks a lot of the internals of gzp. There is now one generic "engine" that operates over anything that implements FormatSpec. This now include Gzip, Zlib, Raw Deflate, and Snap.

    Additionally, instead of writing many gzip blocks with their own headers and no overlapping dicts, this now behaves like pigz and writes a single stream that uses the last 32K of the previous block as the dict for the next block.

    There are two features that still need to be implemented in future PRs:

    • Add the ability to set more metadata in the gzip and zlib headers
    • Drop the libz_sys req for zlib when using the rust backend and find an andler crate that has a combine method.
    opened by sstadick 0
  • [feature] Implement BufRead

    [feature] Implement BufRead

    Given that gzp does have internal buffer, it should implement BufRead so that double buffering isn't required to do things like iterate over lines.

    https://github.com/sstadick/gzp/issues/28

    opened by sstadick 0
  • Remove dependency on core_affinity crate

    Remove dependency on core_affinity crate

    core_affinity is unmaintained (last update was 2 years ago, last real update was 4 years ago)

    It links against System on macos as a framework, which is marked as "do not use" in apple's documentation: https://github.com/NixOS/nixpkgs/pull/143861#issuecomment-955619290, and is causing darwin builds of crabz to fail in nixpkgs

    Some possible alternatives I found: affinity

    enhancement good first issue 
    opened by figsoda 2
  • Add Intel ISA-L

    Add Intel ISA-L

    @ghuls have you ever done any benchmarking with ISA-L vs zlib-ng or libdeflate?

    I'm looking at wrapping GKL and thinking it may be a way to get ISA-L into gzp /crabz but can't find any reliable looking benchmarks indicating of that is even worth it.

    See https://www.reddit.com/r/rust/comments/qhaaju/help_wanted_opensource_genomics_project_in_rust/?utm_source=share&utm_medium=ios_app&utm_name=iossmf for the why on GKL

    opened by sstadick 2
  • error when target is i686-pc-windows-msvc

    error when target is i686-pc-windows-msvc

      = note: liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_alloc_decompressor referenced in function __ZN11libdeflater12Decompressor3new17hf31a74519cf4bcd1E
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_gzip_decompress referenced in function __ZN11libdeflater12Decompressor15gzip_decompress17hde9a1276df679f8cE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_zlib_decompress referenced in function __ZN11libdeflater12Decompressor15zlib_decompress17hce9268d56b6995e5E
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_deflate_decompress referenced in function __ZN11libdeflater12Decompressor18deflate_decompress17h5546d5b293d64f6cE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_free_decompressor referenced in function __ZN67_$LT$libdeflater..Decompressor$u20$as$u20$core..ops..drop..Drop$GT$4drop17h9c4c188330724ceeE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_alloc_compressor referenced in function __ZN11libdeflater10Compressor3new17h9374eb469501ac7aE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_deflate_compress_bound referenced in function __ZN11libdeflater10Compressor22deflate_compress_bound17h4313659dcc1e1e3fE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_deflate_compress referenced in function __ZN11libdeflater10Compressor16deflate_compress17hae2d861b3dc9d63dE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_zlib_compress_bound referenced in function __ZN11libdeflater10Compressor19zlib_compress_bound17h047ad95075c7dac8E
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_zlib_compress referenced in function __ZN11libdeflater10Compressor13zlib_compress17h5c62d3f235f4ec27E
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_gzip_compress_bound referenced in function __ZN11libdeflater10Compressor19gzip_compress_bound17h03ff8ed43415946cE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_gzip_compress referenced in function __ZN11libdeflater10Compressor13gzip_compress17h9aee05245d69c40aE
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_free_compressor referenced in function __ZN65_$LT$libdeflater..Compressor$u20$as$u20$core..ops..drop..Drop$GT$4drop17hf882442a05e00411E
              liblibdeflater-a26db91458bc2d2e.rlib(libdeflater-a26db91458bc2d2e.libdeflater.c676b24c-cgu.3.rcgu.o) : error LNK2019: unresolved external symbol _libdeflate_crc32 referenced in function __ZN11libdeflater3Crc6update17h1dbd33df57087956E
    
    opened by QiWei 2
  • [Idea] Make gzp usable as C library and/or python module

    [Idea] Make gzp usable as C library and/or python module

    Make gzp usable as C library and/or python module

    For example, there is a python wrapper around ISA-Lwhich uses theigzip` code to provide fast gzip decompression/compression (bad compression ratios) which mimics the standard gzip module to provide faster gzip (de)compression speed.

    python-isal
    
    Faster zlib and gzip compatible compression and decompression by providing Python bindings for the ISA-L library.
    
    This package provides Python bindings for the ISA-L library. The Intel(R) Intelligent Storage Acceleration Library (ISA-L) implements several key algorithms in assembly language. This includes a variety of functions to provide zlib/gzip-compatible compression.
    
    python-isal provides the bindings by offering three modules:
    
        isal_zlib: A drop-in replacement for the zlib module that uses ISA-L to accelerate its performance.
        igzip: A drop-in replacement for the gzip module that uses isal_zlib instead of zlib to perform its compression and checksum tasks, which improves performance.
        igzip_lib: Provides compression functions which have full access to the API of ISA-L's compression functions.
    
    isal_zlib and igzip are almost fully compatible with zlib and gzip from the Python standard library. There are some minor differences see: differences-with-zlib-and-gzip-modules.
    

    https://github.com/pycompression/python-isal

    It would be great if gzp could be exposed in a similar way, so more bioinformatics (python) programs can read/write gzip/bgzf files faster.

    opened by ghuls 1
Releases(v0.7.1)
  • v0.7.1(Aug 28, 2021)

  • v0.7.0(Aug 28, 2021)

  • v0.6.0(Aug 28, 2021)

  • v0.5.0(Aug 22, 2021)

    v0.5.0 is a large rework of gzp on several fronts. Overall, gzp now behaves more like pigz.

    • Single stream deflate encoding is used instead of Gzip blocks catted together. This means that both Zlib and RawDeflate formats are now supported.
    • The last 32K of the previously encoded block (in sequential order) is used as the dictionary for the current block to compress. This improves overall compression and matches pigz
    • There is now a single "engine" that drives parallel compression for all formats, including Snap. To add a new format you simply need to implement the FormatSpec trait.

    Performance stayed about the same through all of this. On toy benchmark programs (examples/test2.rs) gzp appears to be about 30% faster than pigz using equivalent compression levels, buffer sizes, and number of threads. This is very anecdotal and should be taken with many grains of salt.

    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Aug 15, 2021)

    This release moves gzp off of tokio and onto rayon as the backed threadpool. There was an error in the thread accounting the previous version that allowed tokio to spawn as many blocking threads as it liked making it look awesome in bechmarks. With that corrected there was no reason not to use rayon instead to slim down dependencies.

    Source code(tar.gz)
    Source code(zip)
Owner
Seth
Principal Bioinformatics Engineer @fulcrumgenomics
Seth
libbz2 (bzip2 compression) bindings for Rust

bzip2 Documentation A streaming compression/decompression library for rust with bindings to libbz2. # Cargo.toml [dependencies] bzip2 = "0.4" License

Alex Crichton 67 Dec 27, 2022
A Rust implementation of the Zopfli compression algorithm.

Zopfli in Rust This is a reimplementation of the Zopfli compression tool in Rust. I have totally ignored zopflipng. More info about why and how I did

Carol (Nichols || Goulding) 76 Oct 20, 2022
(WIP) Taking the pain away from file (de)compression

Ouch! ouch loosely stands for Obvious Unified Compression files Helper and aims to be an easy and intuitive way of compressing and decompressing files

Vinícius Miguel 734 Dec 30, 2022
Like pigz, but rust - a cross platform, fast, compression and decompression tool.

?? crabz Like pigz, but rust. A cross platform, fast, compression and decompression tool. Synopsis This is currently a proof of concept CLI tool using

Seth 232 Jan 2, 2023
Fastest Snappy compression library in Node.js

snappy !!! For [email protected] and below, please go to node-snappy. More background about the 6-7 changes, please read this, Thanks @kesla . ?? Help me to

LongYinan 103 Jan 2, 2023
Michael's Compression Algorithm

mca This repository contains a compression algorithm written by me (Michael Grigoryan). The algorithm is only able to compress and decompress text fil

Michael Grigoryan 1 Dec 19, 2022
A reimplementation of the Zopfli compression tool in Rust.

Zopfli in Rust This is a reimplementation of the Zopfli compression tool in Rust. Carol Nichols started the Rust implementation as an experiment in in

null 11 Dec 26, 2022
lzlib (lzip compression) bindings for Rust

lzip Documentation A streaming compression/decompression library for rust with bindings to lzlib. # Cargo.toml [dependencies] lzip = "0.1" License Lic

Firas Khalil Khana 8 Sep 20, 2022
Obvious Unified Compression Helper is a CLI tool to help you compress and decompress files of several formats

Ouch! ouch stands for Obvious Unified Compression Helper and is a CLI tool to help you compress and decompress files of several formats. Features Usag

null 734 Dec 30, 2022
Basic (and naïve) LZW and Huffman compression algorithms in Rust.

Naive implementation of the LZW and Huffman compression algorithms. To run, install the Rust toolchain. Cargo may be used to compile the source. Examp

Luiz Felipe Gonçalves 9 May 22, 2023
Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Google 3.2k Jan 6, 2023
A Rust on-site channel benchmarking helper. Inter-Process (async / busy) & Intra-Process (async single threaded / async multi threaded)

On-Site Rust Channel Benchmarking Helper Deploy on server to determine which public crates are the fastest for communicating in different architecture

null 23 Jul 9, 2024
A multi-threaded GTK 3 application for fetching the lyrics of the current playing song

Lyriek A multi-threaded GTK 3 application for fetching the lyrics of the current playing song. Installation Arch Linux yay -S lyriek Ubuntu apt insta

Bart Willems 18 Feb 9, 2022
Multi-threaded CLI torrent scraper for displaying searched for magnet links; tailored for use with plex & ssh.

magnetfinder Multi-threaded CLI torrent aggregator; scrapes torrent results from multiple websites and delivers them into a table in your terminal! Su

Ryan 59 Dec 10, 2022
simple multi-threaded port scanner written in rust

knockson simple multi-threaded port scanner written in rust Install Using AUR https://aur.archlinux.org/packages/knockson-bin/ yay -Syu knockson-bin M

Josh Münte 4 Oct 5, 2022
A fast, multi-threaded line counting utility written in Rust.

xloc A fast, multi-threaded line counting utility written in Rust. What is xloc A drop in replacement for bash's wc -l. Your project has x lines of co

null 1 Nov 15, 2021
Multi-threaded CLI torrent scraper for displaying searched for magnet links; tailored for use with plex & ssh.

magnetfinder Multi-threaded CLI torrent aggregator; scrapes torrent results from multiple websites and delivers them into a table in your terminal! Su

null 59 Dec 10, 2022
Multi-threaded Padding Oracle attacks against any service. Written in Rust.

rustpad is a multi-threaded successor to the classic padbuster, written in Rust. It abuses a Padding Oracle vulnerability to decrypt any cypher text or encrypt arbitrary plain text without knowing the encryption key!

Kibouo 76 Dec 16, 2022
Remedy is a multi-threaded rust-imap-maildir synchronization program

remedy Remedy is a multi-threaded rust-imap-maildir synchronization program. Please note that remedy is under heavy development. Current features: IMA

Tomas Jasek 2 Oct 9, 2022
A multi-threaded Twitch chat archiving and downloading tool.

Twitch Chat Downloader ??️ tcd is a multi-threaded Twitch Chat Downloader built in Rust ?? . Usage: tcd [OPTIONS] <--channel <CHANNEL>|--video <VIDEO>

Matthew Polak 6 Dec 19, 2022