First, thank you for your work creating this library. I am attempting to integrate zopfli into my rust application, oxipng, using the following function to call zopfli:

pub fn zopfli_deflate(data: &[u8]) -> Result<Vec<u8>, PngError> {
    let mut output = Vec::with_capacity(max(1024, data.len() / 20));
    let options = zopfli::Options::default();
    match zopfli::compress(&options, &zopfli::Format::Zlib, data, &mut output) {
        Ok(_) => (),
        Err(_) => return Err(PngError::new("Failed to compress in zopfli")),
    };
    output.shrink_to_fit();
    Ok(output)
}

Upon running my suite of integration tests, my test for zopfli runs for about 90 seconds then crashes with the following error:

running 1 test
test zopfli_mode has been running for over 60 seconds

thread 'zopfli_mode' has overflowed its stack
fatal runtime error: stack overflow
error: process didn't exit successfully: `/home/soichiro/repos/oxipng/target/debug/deps/flags-c3b726640fb23cfd zopfli` (signal: 6, SIGABRT: process abort signal)

If I run the test suite in release mode, the test crashes quickly with the following different error message (segfault):

running 1 test
error: process didn't exit successfully: `/home/soichiro/repos/oxipng/target/release/deps/flags-6d76a8384c81188c zopfli` (signal: 11, SIGSEGV: invalid memory reference)

The input data to zopfli is uncompressed PNG image data of around 600kb. Unfortunately, running with RUST_BACKTRACE=1 doesn't produce any additional information or a backtrace. The only test that crashes or fails is the one that uses zopfli.

These changes add streaming support to the crate, so entire files no longer have to be copied to main memory before being compressed. This streaming support is safer and easier to use than memmapping the file, a operation that is not entirely OS-agnostic and requires detailed analysis of how other processes may interact with the memmapped file to guarantee stability and safety. The crate now accepts any Read trait implementation for input data, which provides lots of flexibility.

Compression should be unaffected, as the input data is still read in blocks of ZOPFLI_MASTER_BLOCK_SIZE like before, and a sliding window of the last ZOPFLI_WINDOW_SIZE bytes read is kept to initialize the dictionary for each master block. To test that the changes work okay, I ran the test/run.sh script, which compressed the test files with no issues. I got the original idea for this improvement from https://github.com/google/zopfli/issues/14#issuecomment-77830531.

Although I tried to follow the style of the code that already exists and not introduce too much modern Rust features, MSRV is now bumped to 1.41. Also, the public interface exposed by this crate changed in a backwards-incompatible manner, but the crate major version is still 0, and semantic versioning conventions tell that public APIs in major version 0 are unstable, so I don't think this should be a problem.

Resolves #18.

In #36, @shssoichiro reported this lib taking 25 s in release mode with images in https://github.com/shssoichiro/oxipng/tree/master/tests/files :( :( :(

• The vp vector was ditched as it was redundant.
• The p vector is now assigned using an iterator over the given range.
• The two if statements checking the best_index value was fixed to avoid rewriting the same value.
• The range is now calculated to avoid potentially calculating the range more than once.

When running cargo install cargo-deb it fails with:

error: failed to compile `cargo-deb v1.34.1`, intermediate artifacts can be found at `/tmp/cargo-installOXYI1N`

Caused by:
  failed to parse manifest at `/home/markus/.cargo/registry/src/github.com-1ecc6299db9ec823/zopfli-0.5.0/Cargo.toml`

Caused by:
  failed to parse the `edition` key

Caused by:
  this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions.

The failure happened on Debian stable amd64+arm32:

> rustc --version                                                                                                                                                                                                           
rustc 1.48.0
> cargo --version                                                                                                                                                                                                           
cargo 1.46.0

Currently, the signature of the zopfli::compress method requires an owned struct that implements the Write trait to function, out. This means that out is dropped when the zopfli::compress method finishes execution, which is adequate when out is a file that shouldn't be touched after. However, as this means that the calling function no longer has the ownership of out after calling zopfli::compress, it can no longer use it, which is inadequate for use cases where the calling code needs to keep the ownership of the data sink.

For instance, consider this sample and incomplete Rust code that doesn't compile, which is meant to compress in-memory data and keep its compressed version in memory:

fn compress(data: &Vec<u8>) -> Vec<u8> {
	let vec_buffer = Vec::with_capacity(data.len());
	zopfli::compress(
		&zopfli::Options {
			verbose: false, verbose_more: false,
			numiterations: 8, blocksplittingmax: 15
		},
		&zopfli::Format::Deflate,
		data, vec_buffer
	);
	vec_buffer
}

As vec_buffer ownership is transferred in the call to zopfli::compress, the caller function can no longer return it.

I've thought about writing the result of zopfli::compress to a temporary file and reading it to memory again later, but that's ugly. Also, using a Cursor or Rc doesn't really help. If zopfli::compress accepted a reference to out, or its result returned out, I think this issue could be solved, but I didn't dig enough in the source code of this repository to know how feasible that is.

I've been testing cargo build -Z minimal-versions and found that the old crc may pull very old bitrotten versions. Latest versions of all deps work fine.

Since zopfli splits the file into chunks before trying to compress them, that might be a good spot to fork a bunch of threads.

There's also a fork of zopfli that has implemented multi-threading in C, take a look to see how they're parallelizing the work.