A Rust implementation of the Zopfli compression algorithm.

Related tags

Compression zopfli

Zopfli in Rust

Build Status

This is a reimplementation of the Zopfli compression tool in Rust.

I have totally ignored zopflipng.

More info about why and how I did this can be found in the slides for a talk I gave about it.

How to build

To build the code, run:

$ cargo build --release

and the executable will be in target/release/zopfli.

This should work on stable or beta Rust.

You can also run make zopfli, which will run cargo build and then symlink target/release/zopfli to just zopfli in the project root; this is what the C library does and it was useful for scripting purposes during the rewrite process to keep the command and resulting artifacts the same.

Running the tests

There are some unit tests, mostly around the boundary package merge algorithm implementation in katajainen.rs, that can be run with:

$ cargo test

Golden master tests, to check that compressed files are exactly the same as the C implementation would generate, can be run using:

$ ./test/run.sh

and then checking that git reports no changes to the files in test/results.

Or you can run make test, which will run cargo test, then ./test/run.sh, and then will fail if there are any changed files according to git. Note that if you have uncommitted changes and you run this, your changes will cause this command to fail, but the tests actually passed.

  • Stack overflow in zopfli

    Stack overflow in zopfli

    First, thank you for your work creating this library. I am attempting to integrate zopfli into my rust application, oxipng, using the following function to call zopfli:

    pub fn zopfli_deflate(data: &[u8]) -> Result<Vec<u8>, PngError> {
        let mut output = Vec::with_capacity(max(1024, data.len() / 20));
        let options = zopfli::Options::default();
        match zopfli::compress(&options, &zopfli::Format::Zlib, data, &mut output) {
            Ok(_) => (),
            Err(_) => return Err(PngError::new("Failed to compress in zopfli")),

    Upon running my suite of integration tests, my test for zopfli runs for about 90 seconds then crashes with the following error:

    running 1 test
    test zopfli_mode has been running for over 60 seconds
    thread 'zopfli_mode' has overflowed its stack
    fatal runtime error: stack overflow
    error: process didn't exit successfully: `/home/soichiro/repos/oxipng/target/debug/deps/flags-c3b726640fb23cfd zopfli` (signal: 6, SIGABRT: process abort signal)

    If I run the test suite in release mode, the test crashes quickly with the following different error message (segfault):

    running 1 test
    error: process didn't exit successfully: `/home/soichiro/repos/oxipng/target/release/deps/flags-6d76a8384c81188c zopfli` (signal: 11, SIGSEGV: invalid memory reference)

    The input data to zopfli is uncompressed PNG image data of around 600kb. Unfortunately, running with RUST_BACKTRACE=1 doesn't produce any additional information or a backtrace. The only test that crashes or fails is the one that uses zopfli.

    opened by shssoichiro 6
  • Add streaming support

    Add streaming support

    These changes add streaming support to the crate, so entire files no longer have to be copied to main memory before being compressed. This streaming support is safer and easier to use than memmapping the file, a operation that is not entirely OS-agnostic and requires detailed analysis of how other processes may interact with the memmapped file to guarantee stability and safety. The crate now accepts any Read trait implementation for input data, which provides lots of flexibility.

    Compression should be unaffected, as the input data is still read in blocks of ZOPFLI_MASTER_BLOCK_SIZE like before, and a sliding window of the last ZOPFLI_WINDOW_SIZE bytes read is kept to initialize the dictionary for each master block. To test that the changes work okay, I ran the test/run.sh script, which compressed the test files with no issues. I got the original idea for this improvement from https://github.com/google/zopfli/issues/14#issuecomment-77830531.

    Although I tried to follow the style of the code that already exists and not introduce too much modern Rust features, MSRV is now bumped to 1.41. Also, the public interface exposed by this crate changed in a backwards-incompatible manner, but the crate major version is still 0, and semantic versioning conventions tell that public APIs in major version 0 are unstable, so I don't think this should be a problem.

    Resolves #18.

    opened by AlexTMjugador 3
  • Profile with oxipng images

    Profile with oxipng images

    In #36, @shssoichiro reported this lib taking 25 s in release mode with images in https://github.com/shssoichiro/oxipng/tree/master/tests/files :( :( :(

    opened by carols10cents 3
  • Improve the find_minimum function

    Improve the find_minimum function

    • The vp vector was ditched as it was redundant.
    • The p vector is now assigned using an iterator over the given range.
    • The two if statements checking the best_index value was fixed to avoid rewriting the same value.
    • The range is now calculated to avoid potentially calculating the range more than once.
    opened by mmstick 3
  • compilation fails with Debian Stable

    compilation fails with Debian Stable

    When running cargo install cargo-deb it fails with:

    error: failed to compile `cargo-deb v1.34.1`, intermediate artifacts can be found at `/tmp/cargo-installOXYI1N`
    Caused by:
      failed to parse manifest at `/home/markus/.cargo/registry/src/github.com-1ecc6299db9ec823/zopfli-0.5.0/Cargo.toml`
    Caused by:
      failed to parse the `edition` key
    Caused by:
      this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions.

    The failure happened on Debian stable amd64+arm32:

    > rustc --version                                                                                                                                                                                                           
    rustc 1.48.0
    > cargo --version                                                                                                                                                                                                           
    cargo 1.46.0
    opened by markus2330 2
  • zopfli::compress requires ownership of out parameter

    zopfli::compress requires ownership of out parameter

    Currently, the signature of the zopfli::compress method requires an owned struct that implements the Write trait to function, out. This means that out is dropped when the zopfli::compress method finishes execution, which is adequate when out is a file that shouldn't be touched after. However, as this means that the calling function no longer has the ownership of out after calling zopfli::compress, it can no longer use it, which is inadequate for use cases where the calling code needs to keep the ownership of the data sink.

    For instance, consider this sample and incomplete Rust code that doesn't compile, which is meant to compress in-memory data and keep its compressed version in memory:

    fn compress(data: &Vec<u8>) -> Vec<u8> {
    	let vec_buffer = Vec::with_capacity(data.len());
    		&zopfli::Options {
    			verbose: false, verbose_more: false,
    			numiterations: 8, blocksplittingmax: 15
    		data, vec_buffer

    As vec_buffer ownership is transferred in the call to zopfli::compress, the caller function can no longer return it.

    I've thought about writing the result of zopfli::compress to a temporary file and reading it to memory again later, but that's ugly. Also, using a Cursor or Rc doesn't really help. If zopfli::compress accepted a reference to out, or its result returned out, I think this issue could be solved, but I didn't dig enough in the source code of this repository to know how feasible that is.

    opened by AlexTMjugador 2
  • Update dependencies

    Update dependencies

    I've been testing cargo build -Z minimal-versions and found that the old crc may pull very old bitrotten versions. Latest versions of all deps work fine.

    opened by kornelski 1
  • Parallelize


    opened by carols10cents 1
Carol (Nichols || Goulding)
Carol (Nichols || Goulding)
A reimplementation of the Zopfli compression tool in Rust.

Zopfli in Rust This is a reimplementation of the Zopfli compression tool in Rust. Carol Nichols started the Rust implementation as an experiment in in

null 11 Dec 26, 2022
Michael's Compression Algorithm

mca This repository contains a compression algorithm written by me (Michael Grigoryan). The algorithm is only able to compress and decompress text fil

Michael Grigoryan 1 Dec 19, 2022
libbz2 (bzip2 compression) bindings for Rust

bzip2 Documentation A streaming compression/decompression library for rust with bindings to libbz2. # Cargo.toml [dependencies] bzip2 = "0.4" License

Alex Crichton 67 Dec 27, 2022
Like pigz, but rust - a cross platform, fast, compression and decompression tool.

?? crabz Like pigz, but rust. A cross platform, fast, compression and decompression tool. Synopsis This is currently a proof of concept CLI tool using

Seth 232 Jan 2, 2023
lzlib (lzip compression) bindings for Rust

lzip Documentation A streaming compression/decompression library for rust with bindings to lzlib. # Cargo.toml [dependencies] lzip = "0.1" License Lic

Firas Khalil Khana 8 Sep 20, 2022
Basic (and naïve) LZW and Huffman compression algorithms in Rust.

Naive implementation of the LZW and Huffman compression algorithms. To run, install the Rust toolchain. Cargo may be used to compile the source. Examp

Luiz Felipe Gonçalves 9 May 22, 2023
(WIP) Taking the pain away from file (de)compression

Ouch! ouch loosely stands for Obvious Unified Compression files Helper and aims to be an easy and intuitive way of compressing and decompressing files

Vinícius Miguel 734 Dec 30, 2022
gzp - Multi-threaded Compression

gzp - Multi-threaded Compression

Seth 123 Dec 28, 2022
Fastest Snappy compression library in Node.js

snappy !!! For [email protected] and below, please go to node-snappy. More background about the 6-7 changes, please read this, Thanks @kesla . ?? Help me to

LongYinan 103 Jan 2, 2023
Obvious Unified Compression Helper is a CLI tool to help you compress and decompress files of several formats

Ouch! ouch stands for Obvious Unified Compression Helper and is a CLI tool to help you compress and decompress files of several formats. Features Usag

null 734 Dec 30, 2022
A Brotli implementation in pure and safe Rust

Brotli-rs - Brotli decompression in pure, safe Rust Documentation Compression provides a <Read>-struct to wrap a Brotli-compressed stream. A consumer

Thomas Pickert 59 Oct 7, 2022
Zip implementation in Rust

zip-rs Documentation Info A zip library for rust which supports reading and writing of simple ZIP files. Supported compression formats: stored (i.e. n

null 549 Jan 4, 2023
Ribzip2 - A bzip2 implementation in pure Rust.

ribzip2 - a comprehensible bzip2 implementation ribzip2 is command line utility providing bzip2 compression and decompression written in pure Rust. It

null 16 Oct 24, 2022
A simple rust library to read and write Zip archives, which is also my pet project for learning Rust

rust-zip A simple rust library to read and write Zip archives, which is also my pet project for learning Rust. At the moment you can list the files in

Kang Seonghoon 2 Jan 5, 2022
Brotli compressor and decompressor written in rust that optionally avoids the stdlib

rust-brotli What's new in 3.2 into_inner conversions for both Reader and Writer classes What's new in 3.0 A fully compatible FFI for drop-in compatibi

Dropbox 659 Dec 29, 2022
DEFLATE, gzip, and zlib bindings for Rust

flate2 A streaming compression/decompression library DEFLATE-based streams in Rust. This crate by default uses the miniz_oxide crate, a port of miniz.

The Rust Programming Language 619 Jan 8, 2023
Snappy bindings for Rust

Snappy [ Originally forked from https://github.com/thestinger/rust-snappy ] Documentation Usage Add this to your Cargo.toml: [dependencies] snappy = "

Jeff Belgum 14 Jan 21, 2022
Tar file reading/writing for Rust

tar-rs Documentation A tar archive reading/writing library for Rust. # Cargo.toml [dependencies] tar = "0.4" Reading an archive extern crate tar; use

Alex Crichton 490 Dec 30, 2022
Pure Rust bzip2 decoder

bzip2-rs Pure Rust 100% safe bzip2 decompressor. Features Default features: Rust >= 1.34.2 is supported rustc_1_37: bump MSRV to 1.37, enable more opt

Paolo Barbolini 36 Jan 6, 2023