Hi @kornelski, @shssoichiro, @sagacity, @mqudsi, @dfrankland, @bejolithic, and @AlexTMjugador,

I've invited you all to be admins on this repo today because you've all authored crates that depend on zopfli or significant contributions to the zopfli crate.

I started the Rust reimplementation as an experiment, and I should have acknowledged a long time ago that I wasn't really up for maintaining it beyond that.

However, you all are depending on this code! So now it's yours and you don't have to wait on me to review your PRs or fix bugs or cut new releases (crates.io invites are coming momentarily).

Please feel free to decide amongst yourselves what to do with this repo. I deliberately created a new repo rather than a fork of https://github.com/carols10cents/zopfli so that I can leave that as the experiment's archive, and this can be the canonical location for future development. I've already updated the URLs on crates.io.

I added a note to this repo's README explaining its origin, it'd be nice if that gets left in there but if you feel like taking it out, I understand :)

Enjoy your new puppy!!!!!! 🐶 🐶 🐶

Requiring input byte sources to implement the Seek trait is onerous for end-users, as most programs that generate DEFLATE streams do not impose seekability requirements. In the Unix world, it's common to pipe the output of a program as an input for a compression program, which is a non-seekable data source. In addition, achieving input seekability is often non-practical in network scenarios due to buffering resource requirements and other reasons.

By being smarter about how we use a sliding window, we can drop seekability and exact input size knowledge requirements from the API exposed by this crate, making it readily applicable for even more usage scenarios, without affecting compression. A high-level overview of the new technique is given through code comments.

A downside I can see of this change is that it requires a sliding window ZOPFLI_MASTER_BLOCK_SIZE bytes (≈ 1 MiB) bigger than before due to the need to temporarily store an additional uncompressed master block in memory. However, I think that the better API design makes this trade-off worth it: the additional memory usage is insignificant for the kind of computers that are most likely to run a compression algorithm as demanding as Zopfli anyway.

This is also a breaking change, as the compress_seekable function was removed from the public API. As a minor note, the sliding window for the deflate function is no longer allocated in the stack, which avoids running out of stack memory in practical scenarios I've encountered: allocating ZOPFLI_MASTER_BLOCK_SIZE bytes on the stack is a lot, and the cost of calling the memory allocator pales in comparison to actually compressing data with Zopfli.

Bejolithic is an owner of this crate, as of https://github.com/zopfli-rs/zopfli/issues/1.

I just happened to notice that all 3 other crates they own are copies of other crates, renamed, stripped of attribution and relicensed:

https://crates.io/crates/forage -> https://crates.io/crates/maimo: https://github.com/FuzzrNet/Forage/issues/6 https://crates.io/crates/wasmpng -> https://crates.io/crates/wasimage: https://github.com/datatrash/wasm-png/issues/1 https://crates.io/crates/bbcli -> https://crates.io/crates/wingcli: https://github.com/losfair/blueboat/issues/90

Probably innocuous, but thought it might be worth raising to nip in the bud potential for a supply-chain attack on users of this crate.

Correct some logic that was not taking into account the fact that the final block could also be the first block (with a block length of 0).

This technically closes #2 and prevents zopfli from crashing when given a single zero-length file as an input or as one of the inputs, except that I've manually verified a crash still occurs if explicitly using BlockType::Uncompressed or BlockType::Fixed. The patch works fine if using the default BlockType::Dynamic. I think those are separate bugs and would have no bearing on the correctness of this patch, but I'm not certain since this is my first time looking at the code.

(Come to think of it, I don't think the other block types are currently tested at all in any of the tests?)

Some work has been done in tidying up and improving the crate API lately:

• A crash when dealing with empty files was fixed (#3).
• The public API functions no longer require the byte source to implement Seek, or to provide the input data size beforehand (#7; this is a breaking change).
• Some useful, but previously private Zopfli algorithm options were exposed in the API (c789bc4f). In addition, the API is better documented now.
• The crate now uses log macros to print miscellaneous debug information that previously was conditionally printed to some standard stream, which is much more flexible and suitable for both library and binary dependent crates.
• The MSRV was documented and tested with cargo-msrv.
• A GitHub Actions workflow for CI was defined. It now runs the golden master tests on each commit, which increases our confidence in things working fine after every change.
• Clippy lints were fixed, and rustfmt was run through the codebase.
• The dependency declarations in Cargo.toml were tweaked to have precise versions. This has pros and cons (see this and this), but I think that this is the best approach overall when combined with not-so-frequent dependency upgrade automation, so the rest of the ecosystem has a reasonable time window to keep up.

I think that these changes are stable and relevant enough to justify a new release, but I'd also like to know what other maintainers think about it. Should we do it? :smile:

The Option struct, which can be used by client code to instruct how they want Zopfli to compress data, currently has most of its fields private:

https://github.com/zopfli-rs/zopfli/blob/29b9589d459905af24aecc61c1406b3d0d3c724a/src/lib.rs#L29-L46

However, library users may find it useful to change the default values for these private options. For example, lowering numiterations is immensely needed when dealing with big files, because otherwise the optimization could take so long.

I've been using a patch that just makes these fields pub for some time without problems, but to upstream this improvement it would be nice to add some range checks: it does not make sense to set numiterations or blocksplittingmax to too high or negative values, for example.

@shssoichiro may be interested in this improvement, as tweaking these parameters may improve performance and/or compression in OxiPNG. For example, ZopfliPNG chooses the number of iterations like this:

  options.numiterations = insize < 200000
      ? png_options->num_iterations : png_options->num_iterations_large;

Hi! Attempting to compress empty data (my example uses compress_seekable but I have tested a file also) causes a panic.

fn main() {
    let cursor = std::io::Cursor::new(&[]);
    let mut out = Vec::new();
    zopfli::compress_seekable(
        &zopfli::Options::default(),
        &zopfli::Format::Gzip,
        cursor,
        &mut out,
    );
}

thread 'main' panicked at 'attempt to subtract with overflow', /home/sky/git/zopfli/src/deflate.rs:311:19
stack backtrace:
   0: rust_begin_unwind
             at /rustc/45e2c2881d11324d610815bfff097e25c412199e/library/std/src/panicking.rs:584:5
   1: core::panicking::panic_fmt
             at /rustc/45e2c2881d11324d610815bfff097e25c412199e/library/core/src/panicking.rs:142:14
   2: core::panicking::panic
             at /rustc/45e2c2881d11324d610815bfff097e25c412199e/library/core/src/panicking.rs:48:5
   3: zopfli::deflate::calculate_block_symbol_size_small
             at /home/sky/git/zopfli/src/deflate.rs:311:19
   4: zopfli::deflate::calculate_block_symbol_size_given_counts
             at /home/sky/git/zopfli/src/deflate.rs:345:9
   5: zopfli::deflate::try_optimize_huffman_for_rle
             at /home/sky/git/zopfli/src/deflate.rs:856:20
   6: zopfli::deflate::get_dynamic_lengths
             at /home/sky/git/zopfli/src/deflate.rs:905:5
   7: zopfli::deflate::calculate_block_size
             at /home/sky/git/zopfli/src/deflate.rs:836:31
   8: zopfli::squeeze::lz77_optimal
             at /home/sky/git/zopfli/src/squeeze.rs:506:20
   9: zopfli::deflate::blocksplit_attempt
             at /home/sky/git/zopfli/src/deflate.rs:1147:17
  10: zopfli::deflate::deflate_part
             at /home/sky/git/zopfli/src/deflate.rs:164:31
  11: zopfli::deflate::deflate
             at /home/sky/git/zopfli/src/deflate.rs:104:9
  12: zopfli::gzip::gzip_compress
             at /home/sky/git/zopfli/src/gzip.rs:49:5
  13: zopfli::compress
             at /home/sky/git/zopfli/src/lib.rs:98:25
  14: zopfli::compress_seekable
             at /home/sky/git/zopfli/src/lib.rs:83:5
  15: scratch::main
             at ./src/main.rs:4:5
  16: core::ops::function::FnOnce::call_once
             at /rustc/45e2c2881d11324d610815bfff097e25c412199e/library/core/src/ops/function.rs:227:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

Welcome to Renovate! This is an onboarding PR to help you understand and configure settings before regular Pull Requests begin.

🚦 To activate Renovate, merge this Pull Request. To disable Renovate, simply close this Pull Request unmerged.

Detected Package Files

• Cargo.toml (cargo)
• .github/workflows/ci.yml (github-actions)

Configuration

🔡 Renovate has detected a custom config for this PR. Feel free to ask for help if you have any doubts and would like it reviewed.

Important: Now that this branch is edited, Renovate can't rebase it from the base branch any more. If you make changes to the base branch that could impact this onboarding PR, please merge them manually.

What to Expect

It looks like your repository dependencies are already up-to-date and no Pull Requests will be necessary right away.

❓ Got questions? Check out Renovate's Docs, particularly the Getting Started section. If you need any further assistance then you can also request help here.

This PR has been generated by Mend Renovate. View repository job log here.

This SIMD accelerated crc library may be able to improve performance: https://github.com/srijs/rust-crc32fast Not sure if this is related to the performance difference between this and zopfli-rs?

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

This repository currently has no open or pending branches.

Detected dependencies

• [ ] Check this box to trigger a request for Renovate to run again on this repository