This is a followup to #34. @japaric, @TPS suggested that you could help. Do you have experience with deploying rust-everywhere for Windows on Travis CI? The Travis docs seem to indicate that Windows isn't a supported platform. Is the workaround then to use Appveyor for Windows builds?

So this PR might look a bit weird, but hopefully still acceptable. There are two things going on here:

1. A custom alternative to std::thread::spawn based on rayon + crossbeam-channel. See the comment on the function for motivation and more details, but TL;DR is that it makes possible to run OxiPNG with threads even on a Wasm target! (in engines that support Wasm threads). Working variant of Squoosh.app that makes use of this patch and runs OxiPNG+threads here: https://multithread--squoosh.netlify.app/editor
2. Since the part 1 requires crossbeam-channel anyway, I switched to it from std::sync::mspc, too, for the better performance as well as API improvements. For example, before this, evaluator had to use sync_channel(4) - a blocking synchronous channel with a magical constant maximum of 4 - because the async version of std::sync::mpsc::Sender can't be shared between threads (it doesn't implement Sync trait). However, with crossbeam it's not a problem, and it becomes possible to use asynchronous unbounded channel instead.

Currently --help says:

Optimization levels:
    -o 0  =>  --zc 3 --nz                  (0 or 1 trials)
    -o 1  =>  --zc 9                       (1 trial, determined heuristically)
    -o 2  =>  --zc 9 --zs 0-3 -f 0,5       (8 trials)
    -o 3  =>  --zc 9 --zs 0-3 -f 0-5       (24 trials)
    -o 4  =>  --zc 9 --zs 0-3 -f 0-5 -a    (24 trials + 6 alpha trials)
    -o 5  =>  --zc 3-9 --zs 0-3 -f 0-5 -a  (96 trials + 6 alpha trials)
    -o 6  =>  --zc 1-9 --zs 0-3 -f 0-5 -a  (180 trials + 6 alpha trials)

However, actual code for preset 4 is equivalent to preset 3 (no alpha optimisation):

    fn apply_preset_3(mut self) -> Self {
        for i in 1..5 {
            self.filter.insert(i);
        }
        self
    }

    fn apply_preset_4(self) -> Self {
        self.apply_preset_3()
    }

I suppose this is the outcome of a decision not to perform alpha optimisations by default, made in https://github.com/shssoichiro/oxipng/pull/187.

Which is fine, but makes state of this preset fairly confusing. Should it be removed at this point?

Also, docs for higher opt levels should probably be updated as well to exclude -a?

Trying to run e.g.

cargo run -- -o 6 -P tests/files/grayscale_alpha_16_reduce_alpha_right.png --alpha

on current master (39ff7cfd5c8403af82df097f7a227ebac370254a) seems to produce non-deterministic results.

Sometimes I'm getting

Processing: tests/files/grayscale_alpha_16_reduce_alpha_right.png
    200x152 pixels, PNG format
    2x16 bits/pixel, GrayscaleAlpha
    IDAT size = 32322 bytes
    File size = 32400 bytes
Trying: 216 combinations
Found better combination:
    zc = 9  zs = 1  f = 5        20295 bytes
    IDAT size = 20295 bytes (12027 bytes decrease)
    file size = 20373 bytes (12027 bytes = 37.12% decrease)
Running in pretend mode, no output

and sometimes (more frequently) a worse result

Processing: tests/files/grayscale_alpha_16_reduce_alpha_right.png
    200x152 pixels, PNG format
    2x16 bits/pixel, GrayscaleAlpha
    IDAT size = 32322 bytes
    File size = 32400 bytes
Trying: 216 combinations
Found better combination:
    zc = 9  zs = 1  f = 5        20335 bytes
    IDAT size = 20335 bytes (11987 bytes decrease)
    file size = 20413 bytes (11987 bytes = 37.00% decrease)
Running in pretend mode, no output

Originally posted as comments on https://github.com/shssoichiro/oxipng/commit/239ca81db7b45b3ac731a8d8b2746e760500d0a9#commitcomment-37751319.

$ cd /tmp
$ wget -q 'https://raw.githubusercontent.com/petvas/i3lock-blur/master/lock.png'
$ cp lock.png{,.orig}

$ optipng -v
OptiPNG version 0.7.7
Copyright (C) 2001-2017 Cosmin Truta and the Contributing Authors.

This program is open-source software. See LICENSE for more details.

Portions of this software are based in part on the work of:
  Jean-loup Gailly and Mark Adler (zlib)
  Glenn Randers-Pehrson and the PNG Development Group (libpng)
  Miyasaka Masaru (BMP support)
  David Koblas (GIF support)

Using libpng version 1.6.37 and zlib version 1.2.11

$ optipng -o4 lock.png
** Processing: lock.png
512x512 pixels, 4x8 bits/pixel, RGB+alpha
Reducing image to 8 bits/pixel, 233 colors (232 transparent) in palette
Input IDAT size = 13216 bytes
Input file size = 13359 bytes

Trying:
  zc = 9  zm = 8  zs = 0  f = 0		IDAT size = 6592
  zc = 9  zm = 8  zs = 3  f = 0		IDAT size = 6251

Selecting parameters:
  zc = 9  zm = 8  zs = 3  f = 0		IDAT size = 6251

Output IDAT size = 6251 bytes (6965 bytes decrease)
Output file size = 7336 bytes (6023 bytes = 45.09% decrease)

$ oxipng --version
oxipng 2.3.0

$ cp lock.png{.orig,}
$ oxipng -o6 lock.png
Processing: lock.png
    512x512 pixels, PNG format
    4x8 bits/pixel, RGBA
    IDAT size = 13216 bytes
    File size = 13359 bytes
Trying: 216 combinations
Found better combination:
    zc = 9  zs = 0  f = 0        8927 bytes
    IDAT size = 8927 bytes (4289 bytes decrease)
    file size = 9055 bytes (4304 bytes = 32.22% decrease)
Output: lock.png

In keeping with being able to use oxipng as a library, there should be a way to alter an image already in-memory. This allows end users working with in-memory representations -- say, external web servers providing optimization services -- to easily take advantage of oxipng.

Processing: Mona_Lisa-PNG8-Scatter-BW.png
    404x600 pixels, PNG format
    1x8 bits/pixel, Grayscale
    IDAT size = 18171 bytes
    File size = 18242 bytes
Reducing image to 1x1 bits/pixel, Grayscale
Trying: 6 combinations
Found better combination:
    zc = 0  zs = 0  f = 0        16966 bytes
    IDAT size = 16966 bytes (1205 bytes decrease)
    file size = 17037 bytes (1205 bytes = 6.61% decrease)
The resulting image is corrupted and will not be outputted.
This is a bug! Please report it at https://github.com/shssoichiro/oxipng/issues
The resulting image is corrupted

I don't know what's happening here, but I ran the command line oxipng -s -o 6 -Z --fix *BW.png in an attempt to figure out why earlier runs (without -Z, as well as with and without -D) had some corrupt images, but only for monochrome black and white PNGs (1-bit). All of the attached PNGs had corrupt output when given the above command line. I an on Windows 10 64-bit, and used a just-downloaded 5.0.1 release. Zopfli compression claimed it had significant size reductions, but then also that the output was corrupted and so was not saved. I don't have a Rust dev environment set up, and I suspect compiling oxipng from source would be a challenge for me, so I guess I'm just reporting what it says to report, and providing some of the images that failed.

EDIT: I originally uploaded GIFs, not PNGs, though the contents were identical. Sorry about that!

This PR introduces an entirely different approach to alpha optimisation, which is more effective than before and much faster. I think this is quite cool, so I'll start by showing off some benchmarks.

Set of non-reducible RGBA files totalling 15,737,898 bytes, compressed using -o3: | Test | Size | Time | |-|-|-| | Without -a | 9,868,361 | 20.463 | | With -a, master | 8,813,177 | 24.155 | | With -a, PR | 8,754,222 | 20.488 |

Following comments on #468, I continued searching for better alpha techniques and eventually came across CryoPNG. The key here is to alter the data such that it becomes zero when filtered. While CryoPNG does this assuming a single delta filter for the whole image, we can take it to the next level by integrating the technique into the actual filtering stage. This allows it to work for heuristic filters and actually help guide the heuristic selection.

The speed advantage is that there's no need to do any evaluator trials. Being tailored to the specific filter being applied, the output is near-optimal - this doesn't mean it's the best possible but does mean it would be difficult to improve upon. As such, the original image data is no longer considered - there are still images where the original data could be better but these would be quite rare and it would be impractical to evaluate in the new setup.

An example of what the technique looks like:

I've removed all the redundant tests and associated files (and also some redundant deflate benches leftover from before the switch to libdeflate).

Using the newly-released 6.0.0 version on Windows 11 (64-bit) on the attached PNG results in the following:

> oxipng -Z -o max cover.png

Processing: cover.png
    2981x2981 pixels, PNG format
    3x8 bits/pixel, RGB
    IDAT size = 6087921 bytes
    File size = 6088031 bytes
Trying: 6 combinations
Found better combination:
    zc = 0  zs = 0  f = 4        5897004 bytes
    IDAT size = 5897004 bytes (190917 bytes decrease)
    file size = 5897114 bytes (190917 bytes = 3.14% decrease)
The resulting image is corrupted and will not be outputted.
This is a bug! Please report it at https://github.com/shssoichiro/oxipng/issues
The resulting image is corrupted

As per instructions, here I am.

The problematic image is available at https://user-images.githubusercontent.com/1986890/188520158-84a1a201-a63f-4884-b880-19980932551b.png

FWIW, I had just successfully compressed this image (same command) with oxipng 5.0.1 a few days ago.

The -o max flag doesn't make a difference, either (same error), but I don't get the error in normal mode (it can't achieve better compression and doesn't even try, I guess).

Looks like https://github.com/shssoichiro/oxipng/commit/4ef92089d5743e896c920d6fa364ff66ab22c346 landed yesterday and removes the non-parallel mode added in #194.

These conditional compilation attributes and a separate synchronous code are important for environments like WebAssembly (which is what they were added for in the first place), where thread::spawn is not available and errors out at runtime.

After this change, it's invoked unconditionally again.

Although the Cloudflare fork of zlib may introduce performance improvements, there are issues building it with MinGW toolchains on Windows, as evidenced by this StackOverflow question.

However, it is not always possible to switch to the MSVC toolchain on Windows, because that may break other native C libraries which have trouble with MSVC instead, and just compiling to a 32-bit target can get pretty messy too.

I've opened an issue to get this error fixed upstream, in the cloudflare-zlib-sys crate, but several months have passed with no answer.

Therefore, make the Cloudflare zlib dependency optional, so users will have to explicitly opt in for it.

This is a draft proposal for a raw API, as discussed in #376. Although it's not final yet, I wanted to get a PR up now before the holidays and hopefully get some feedback on it. One question I have is, should the input PngImage be an & reference? More refactoring may be required to allow this, as I don't think you can create an Arc from a shared reference without cloning.

Hello, I used to use optipng on the output of long image processing pipelines and when doing so, I chose the simplest possible interchange format to avoid any useless encoding cost. Usually, it was PPM (since I knew my input didn't have an alpha channel).

Would it be possible for oxipng to support at least one such format? Netpbm would probably be the most suitable and the image crate seems to have support for some of them (https://docs.rs/image/latest/image/codecs/pnm/index.html).

For an idea of the "cost" of using PNG in this situation (multiplied by num_images / num_cpu):

$ magick identify /tmp/tmp.ppm
/tmp/tmp.ppm PPM 5551x6380 5551x6380+0+0 8-bit sRGB 101.324MiB 0.120u 0:00.087
$ time magick convert -define png:compression-level=0 /tmp/tmp.ppm /tmp/tmp.png
magick convert -define png:compression-level=0 /tmp/tmp.ppm /tmp/tmp.png  1.72s user 0.16s system 182% cpu 1.033 total
$ time magick convert /tmp/tmp.ppm /tmp/tmp2.ppm
magick convert /tmp/tmp.ppm /tmp/tmp2.ppm  0.19s user 0.11s system 99% cpu 0.300 total
$ time magick convert /tmp/tmp.ppm /tmp/tmp.pam
magick convert /tmp/tmp.ppm /tmp/tmp.pam  0.18s user 0.11s system 99% cpu 0.289 total
$ time magick convert /tmp/tmp.ppm /tmp/tmp.miff
magick convert /tmp/tmp.ppm /tmp/tmp.miff  0.19s user 0.11s system 99% cpu 0.302 total

ℹ️ Compiling your crate in development mode...

[INFO]: 🎯 Checking for the Wasm target... [INFO]: 🌀 Compiling to Wasm... Compiling libdeflate-sys v0.10.0 The following warnings were emitted during compilation:

warning: In file included from libdeflate/lib/arm/cpu_features.c:38: warning: In file included from libdeflate/lib/arm/../cpu_features_common.h:38: warning: libdeflate/lib/arm/../lib_common.h:64:10: fatal error: 'string.h' file not found warning: #include <string.h> warning: ^~~~~~~~~~ warning: 1 error generated.

error: failed to run custom build command for libdeflate-sys v0.10.0

Caused by: process didn't exit successfully: /Users/ruansan/Development/kg/rust-projects/png-optimize/target/debug/build/libdeflate-sys-a95b79cdfa5005fb/build-script-build (exit status: 1) --- stdout TARGET = Some("wasm32-unknown-unknown") OPT_LEVEL = Some("0") HOST = Some("x86_64-apple-darwin") CC_wasm32-unknown-unknown = None CC_wasm32_unknown_unknown = None TARGET_CC = None CC = None CFLAGS_wasm32-unknown-unknown = None CFLAGS_wasm32_unknown_unknown = None TARGET_CFLAGS = None CFLAGS = None CRATE_CC_NO_DEFAULTS = None DEBUG = Some("true") running: "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "libdeflate" "-o" "/Users/ruansan/Development/kg/rust-projects/png-optimize/target/wasm32-unknown-unknown/debug/build/libdeflate-sys-c7005d76609b5c0d/out/lib/libdeflate/lib/arm/cpu_features.o" "-c" "libdeflate/lib/arm/cpu_features.c" cargo:warning=In file included from libdeflate/lib/arm/cpu_features.c:38: cargo:warning=In file included from libdeflate/lib/arm/../cpu_features_common.h:38: cargo:warning=libdeflate/lib/arm/../lib_common.h:64:10: fatal error: 'string.h' file not found cargo:warning=#include <string.h> cargo:warning= ^~~~~~~~~~ cargo:warning=1 error generated. exit status: 1

--- stderr

error occurred: Command "clang" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "--target=wasm32-unknown-unknown" "-I" "libdeflate" "-o" "/Users/ruansan/Development/kg/rust-projects/png-optimize/target/wasm32-unknown-unknown/debug/build/libdeflate-sys-c7005d76609b5c0d/out/lib/libdeflate/lib/arm/cpu_features.o" "-c" "libdeflate/lib/arm/cpu_features.c" with args "clang" did not execute successfully (status code exit status: 1).

Error: Compiling your crate to WebAssembly failed Caused by: failed to execute cargo build: exited with exit status: 101 full command: "cargo" "build" "--lib" "--target" "wasm32-unknown-unknown"

Currently pngcrush at max settings always compresses a little more than oxipng at max settings.

Source used is “Cosmic Cliffs” in the Carina Nebula (NIRCam and MIRI Composite Image) from webbtelescope.org.

λ wget "https://stsci-opo.org/STScI-01G8GYD8B1H306SSPR4MY6NGN0.png"
--2022-07-23 12:53:58--  https://stsci-opo.org/STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Resolving stsci-opo.org (stsci-opo.org)... 65.9.95.4, 65.9.95.10, 65.9.95.43, ...
Connecting to stsci-opo.org (stsci-opo.org)|65.9.95.4|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 50166110 (48M) [image/png]
Saving to: ‘STScI-01G8GYD8B1H306SSPR4MY6NGN0.png’

STScI-01G8GYD8B1H306SSPR4MY6NGN0.png            100%[======================================================================================================>]  47.84M  5.78MB/s    in 8.0s    

2022-07-23 12:54:08 (5.96 MB/s) - ‘STScI-01G8GYD8B1H306SSPR4MY6NGN0.png’ saved [50166110/50166110]

λ pngcrush -brute -rem alla -rem allb -rem text -reduce -check STScI-01G8GYD8B1H306SSPR4MY6NGN0.png pngcrush.png
  Recompressing IDAT chunks in STScI-01G8GYD8B1H306SSPR4MY6NGN0.png to pngcrush.png
   Total length of data found in critical chunks            =  50145112
pngcrush: iCCP: known incorrect sRGB profile
   Best pngcrush method        =  16 (ws 15 fm 5 zl 2 zs 2) =  45336617
CPU time decode 101.124237, encode 2127.660889, other 0.256804, total 2229.506836 sec

λ oxipng --opt max --strip all --alpha --interlace 0 --threads 8 --out oxipng.png STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
Processing: STScI-01G8GYD8B1H306SSPR4MY6NGN0.png
    11264x3904 pixels, PNG format
    3x8 bits/pixel, RGB
    IDAT size = 50144491 bytes
    File size = 50166110 bytes
Trying: 216 combinations
Found better combination:
    zc = 1  zs = 2  f = 5        45398613 bytes
    IDAT size = 45398613 bytes (4745878 bytes decrease)
    file size = 45398670 bytes (4767440 bytes = 9.50% decrease)
Output: oxipng.png

λ ls -l *.png
-rw-r--r-- 1 txtsd users 45398670 Jul 23 12:59 oxipng.png
-rw-r--r-- 1 txtsd users 45336617 Jul 23 13:35 pngcrush.png
-rw-r--r-- 1 txtsd users 50166110 Jul 21 23:25 STScI-01G8GYD8B1H306SSPR4MY6NGN0.png

I absolutely love oxipng, it is my new favorite PNG optimizer.

I have been using it like this (Is it possible to get more compression? did I miss any of the advanced options?):

oxipng -o max --strip all -a -Z *.png

I am wondering if there is a tool that does something similar for .ico files? (I still have one of these for old browser compatibility)