Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Okay, as mentioned in #143 I've taken a shot at updating the CMake script to try to address some of the issues raised there. Specifically:

• Add install target
• Version is only set once
• The default build type now uses CFLAGS etc.

I also improved support for using Zopfli as a subproject from other CMake based projects.

Some issues/things to discuss that remain:

• I added a workaround for the issue about symbol export when building as a DLL with MSVC, it only works for recent versions of CMake. Actually fixing this would require adding macros to the public headers to optionally export symbols, and I don't know if the devs are interested in this (see for instance this description).
• libm is still included by default on UNIX, somebody with more knowledge about this will have to fix it if needed.
• Leaving the default build type alone allows building using CFLAGS etc., but means the default build will be unoptimized, perhaps the docs should suggest building with -DCMAKE_BUILD_TYPE=Release?

• Fix incorrect Deflate stream reporting when zlib or gz container is used - use offset to record outsize before compressing and then use that offset to properly display compressed stream size,
• Fix SIGSEGV when file can't be opened for writing binary ("wb") by displaying error message and exiting with EXIT_FAILURE. This will help person understand that he/she is working in read-only directory or forgot to sudo,
• Fix output to stdout on Windows to not convert new lines, instead output to terminal as binary (I'm not sure if changing back to text mode is necessary but did it just to be safe and not break terminal output after zopfli closes),
• Fix iterations error message not ending with new line separator making some terminals display command prompt at the end of this message when zopfli exist (for example Linux),
• Fix SIGSEGV when there is not enough memory to allocate cache, display error and exit instead.

Please consider tagging a release every now and then as per http://semver.org/.

This would be useful for various obvious reasons, but I’m asking this 
specifically because it would allow me to create a 
[Homebrew](http://mxcl.github.io/homebrew/) formula so people using OS X can 
install Zopfli even more easily. (All they’d have to do is enter `brew 
install zopfli` in their shell.)

How about tagging the current release as v0.1.0 and then go from there?

Note: you may also want to display the version number in the help screen 
(`zopfli -h`).

Original issue reported on code.google.com by [email protected] on 19 Apr 2013 at 7:53

I can see there have been quite a few changes since 1.0.0 release. Now that 2 years have past, why not make a 1.0.1 release yet, as the Makefile implies?

Original file - 2 881 698 byte

Kzip -n24 - kzip.png.gz - 587 696 byte

T Boundary   Tokens   h.size   b.size
1        0       38        3      326
2       28      616      239     2430
2    22cb5     2502      517    28190
2    2d9e5    13143      695   185804
2    52a2e    35256      873   511584
2    94049    10559      761   155757
2    aabd4    12891      765   179466
2    c4ec0    33708      860   458594
2   105d5c    15400      760   205492
2   1215c3    29998      881   404564
2   14d740    31785      824   445153
2   17a200     5742      685    71954
2   1889cb     3460      592    47216
2   190f21    15080      752   221017
2   1b5741    11789      728   170790
2   1caa76    23660      744   341586
2   1ef9f2    17853      777   254185
2   20f192     3017      547    33958
2   21d031    13763      824   199319
2   23bfad    16925      770   248473
2   259e4f    12317      758   177183
2   26fecf    12971      757   178228
2   284496    16962      668   224078
2   2a35d7      535      322     2827
4748174 bits long (24 blocks)

zopfli -i15 - zopfli.png.gz - 593 540 byte (options->blocksplittingmax = 0). Ratio 0,98%

T Boundary   Tokens   h.size   b.size
2        0     1608      898    14389
2    2913a      268      718     3849
2    2ac7e    19745      892   278331
2    5de38    25482      957   365386
2    8bd96    15297      924   221828
2    ab534    29291      878   398858
2    e693b    16247      938   219429
2   104c14    13909      900   183257
2   11d277    29170      891   386252
2   148fbc    37454      899   509544
2   17d0d0     1611      824    23185
2   185e84     1565      941    18288
2   1884b2    13165      944   185803
2   1aa9fd    40508      909   579102
2   1ee8d5    18498      950   260705
2   20f01a      859      807    10417
2   210ccc      343      390     3814
2   21a3c0      939      801    10174
2   21bfec     8627      937   121285
2   230b86    42250      842   602616
2   27bdde    11512      924   153738
2   28d85a    11109      918   148241
2   2a35d7       77      346     1094
2   2a5203      458      303     1837
4701422 bits long (24 blocks)

Tell me, what to do to improve the compression rate of this file?

https://github.com/Mattlk13/zopfli/pull/3 introduced a compile error on zopfli\src\zopflipng\lodepng\lodepng_util.cpp, throwing an error C2124: divide or mod by zero. See this AppVeyor build as an example.

(ClCompile target) -> 
  C:\projects\zopfli\src\zopflipng\lodepng\lodepng_util.cpp(299,1): error C2124: divide or mod by zero [C:\projects\zopfli\cmake_build\libzopflipng.vcxproj]
    69 Warning(s)
    1 Error(s)

There are also a lot of open warnings that should be addressed.

Make sure qsort uses ascending ordering of counts when weights are equal as per GCC 4.8. This makes zopfli more portable on different set of compilers that support qsort.

This fixes issue #95 .

If sorting counts through reverse ordering is desired, modify accordingly.

I found an issue with Zopfli (the same behavior happens in my fork) if anybody is interested.

When I compile it using mingw64 GCC 5.3 in MSYS2 (doesn't matter if it's x86 or x64 version, also O0 doesn't make any difference as well) the Windows' binary produces different block slit points and also the progress of iterations progresses differently (when I pass exact same block split points in my fork) than it does with Linux' version compiled with GCC 4.8.x.

Not sure if it's some kind of bug in GCC 4.8.x or regression in GCC 5.3. Also I'm not sure which one provides VALID compilation. Though I know for sure that with older toolchains I used prior to MSYS2 there were no differences.

Example of GCC 5.3 split points produced: 2882 17106 18913 (hex: b42 42d2 49e1) Example of GCC 4.8 split points produced: 2853 17304 18923 (hex: b25 4398 49eb)

So far I can't figure if there is anything in the code that can be altered since even after enabling all possible warnings and trying code changes to make everything nice according to them, nothing helps.

So if anybody has any more intel, this could be a nice lesson. :)

On a side note. On ARMv7 device (Odroid U3) running my fork I got "ZopfliVerifyLenDist: Assertion `data[pos - dist + i] == data[pos + i]' failed." once. Not sure if it has anything to do with GCC 4.8.x or it's just that some of switches in my fork don't play along with each other on certain data (--brotli --lazy while certain static block split points were passed with --cbs command).

Apparently, the zopfli-1.0.1 tag is not properly picked up by git describe due to not being an annotated tag, which may trip some packaging scripts. The snippet below demonstrates how this discrepancy occurs even in a freshly cloned git repository:

✓ fratti@archbook /tmp $ git clone https://github.com/google/zopfli.git
Cloning into 'zopfli'...
remote: Counting objects: 385, done.
remote: Total 385 (delta 0), reused 0 (delta 0), pack-reused 385
Receiving objects: 100% (385/385), 254.24 KiB | 0 bytes/s, done.
Resolving deltas: 100% (215/215), done.
Checking connectivity... done.
✓ fratti@archbook /tmp $ cd zopfli/
✓ fratti@archbook /tmp/zopfli $ git tag --list
zopfli-1.0.0
zopfli-1.0.1
✓ fratti@archbook /tmp/zopfli $ git describe
zopfli-1.0.0-49-g0aa5474
✓ fratti@archbook /tmp/zopfli $ git describe --tags
zopfli-1.0.1-9-g0aa5474

Essentially, you forgot the -a in git tag -a zopfli-1.0.1 89cf773, creating a lightweight instead of an annotated tag, which is usually meant to be used for temporary tags.

I have a custom PNG optimiser for large files, which uses libzopfli for deflate compression. To improve speed, image data is split into chunks, and compression run on multiple cores. This works very well, scales perfectly, and causes only a negligible size increase.

I'm attaching a patch here for anyone interested, or perhaps to be included into libzopfli itself. I think it's the least intrusive way to add this, but perhaps not the most elegant, so I'm not submitting a PR.

diff --git a/src/zopfli/deflate.c b/src/zopfli/deflate.c
index 4d124f4..7628658 100644
--- a/src/zopfli/deflate.c
+++ b/src/zopfli/deflate.c
@@ -834,18 +834,28 @@ void ZopfliDeflate(const ZopfliOptions* options, int btype, int final,
                    unsigned char* bp, unsigned char** out, size_t* outsize) {
  size_t offset = *outsize;
 #if ZOPFLI_MASTER_BLOCK_SIZE == 0
-  ZopfliDeflatePart(options, btype, final, in, 0, insize, bp, out, outsize);
+  ZopfliDeflatePart(options, btype, final == 1, in, 0, insize, bp, out, outsize);
 #else
   size_t i = 0;
   do {
     int masterfinal = (i + ZOPFLI_MASTER_BLOCK_SIZE >= insize);
-    int final2 = final && masterfinal;
+    int final2 = final == 1 && masterfinal;
     size_t size = masterfinal ? insize - i : ZOPFLI_MASTER_BLOCK_SIZE;
     ZopfliDeflatePart(options, btype, final2,
                       in, i, i + size, bp, out, outsize);
     i += size;
   } while (i < insize);
 #endif
+
+  /* Z_SYNC_FLUSH */
+  if (final == 2) {
+    AddBits(0, 3, bp, out, outsize);
+    ZOPFLI_APPEND_DATA(0, out, outsize);
+    ZOPFLI_APPEND_DATA(0, out, outsize);
+    ZOPFLI_APPEND_DATA(0xFF, out, outsize);
+    ZOPFLI_APPEND_DATA(0xFF, out, outsize);
+  }
+
   if (options->verbose) {
     fprintf(stderr,
             "Original Size: %lu, Deflate: %lu, Compression: %f%% Removed\n",

To use this, call ZopfliDeflate with 1 passed for final on the last chunk, and 2 on all other chunks.

PS. The documentation from the zlib manual.

If the parameter flush is set to Z_SYNC_FLUSH, all pending output is flushed to the output buffer and the output is aligned on a byte boundary, so that the decompressor can get all input data available so far. [...] This completes the current deflate block and follows it with an empty stored block that is three bits plus filler bits to the next byte, followed by four bytes (00 00 ff ff).

Split ZopfliInitHash to:

• ZopfliAllocHash - allocate hash memory,
• ZopfliResetHash - reset hash values.

Allocate Hash outside of ZopfliLZ77Greedy and ZopfliLZ77OptimalRun that pass it further to functions previously allocating them. Do the same for costs malloc'd array.

Reason for this change:

• the size of malloc doesn't change,
• speed up Zopfli^,
• fix crash on certain devices^^.

^ speeds up Zopfli (especially on smaller blocks) by reducing amount of sys time from ~7s to 0.1s on x64 Linux for ~5m compression time and from ~1m to 0.1s on ARMv7 Linux for 13m compression time. ^^ fixes a large amount of iterations crash on some ARM devices that due to architecture or older kernel (not sure which) don't handle too aggressive heap allocation and freeing well.

PS. You don't want to know how many hours I wasted guessing what was wrong on Odroid U3 when gdb showed false-positives - without pthreads it was running into assert in ZopfliVerifyLenDist, with pthreads it was crashing on free in ZopfliCleanHash. :)

PS2: TCmalloc didn't help on Odroid U3, still crash due to heap problems occured, before this fix.

I'm using pigz to compress a qcow2 QEMU QCOW2 Image (v3) file which happens to be very huge. I can't share the file because it might contain sensitive information I don't want to disclose here. The file size is 42949672960 bytes.

pigz: zopfli/src/zopfli/tree.c:100: ZopfliCalculateBitLengths: Assertion `!error' failed.
Aborted (core dumped)

Obviously, a stack-trace would be a lot beneficial here, but it happens to be very slow on my personal machine, although, I can manage to run it on a local server, if it is really required.

Hello. Can you document, at least shortly, parameters of ZopfliDeflate() and ZopfliDeflatePart() functions? In source file would be enough.

They are most used in all applications utilizing zopfli but they are bit mysterious and when one tries to understand mechanics behind these functions it's more less guessing game.

Regards.

The last release was two years and two days ago. The main branch contains fixes for issues like #176 that would be nice to get into a release (the incorrect Decoding error 58: invalid ADLER32 encountered error).

Linux distributions keep shipping the last stable releases even though the main branch is in a better state than the last release.

Ping @lvandeve.

Using the file

https://gitlab.gnome.org/GNOME/gnome-backgrounds/-/blob/main/backgrounds/adwaita-day.png

And Zopflipng 1.0.3 with no further options than input/output

The result is: Error: verification of result failed, keeping original. Error: 58.

This adds two public targets:

• :zopfli.
• :zopflipng.

Which enable easily adding Zopfli in bazel-enabled repositories:

1. Add this repo through http_archive.
2. use @zopfli//:zopfli. My use case is to zopfli some CSS at build time:

genrule(
    name = "all-min.css_gz",
    srcs = ["gen/all-min.css"],
    outs = ["all-min.css.gz"],
    cmd = "$(location @zopfli//:zopfli) -c $< > $@",
    tools = ["@zopfli"],
)

Everything is compiled statically (i.e. zopflipng statically links zopfli) for the sake of simplicity.