Some PistonDeveloper projects are implicit referencing miniz-sys. On my Desktop i run into problems compiling them (named: conrod examples and imgproc examples).

OS: Windows 8.1 (64Bit) rustc: 1.5.0 (Stable)

failed to run custom build command for `miniz-sys v0.1.7`
Process didn't exit successfully: `C:\Users\username\Documents\GitHub\conrod\target\
release\build\miniz-sys-f1c39e8e406fa25f\build-script-build` (exit code: 101)
--- stdout
TARGET = Some("x86_64-pc-windows-gnu")
OPT_LEVEL = Some("3")
PROFILE = Some("release")
TARGET = Some("x86_64-pc-windows-gnu")
debug=false opt-level=3
HOST = Some("x86_64-pc-windows-gnu")
TARGET = Some("x86_64-pc-windows-gnu")
TARGET = Some("x86_64-pc-windows-gnu")
HOST = Some("x86_64-pc-windows-gnu")
CC_x86_64-pc-windows-gnu = None
CC_x86_64_pc_windows_gnu = None
HOST_CC = None
CC = None
TARGET = Some("x86_64-pc-windows-gnu")
HOST = Some("x86_64-pc-windows-gnu")
CFLAGS_x86_64-pc-windows-gnu = None
CFLAGS_x86_64_pc_windows_gnu = None
HOST_CFLAGS = None
CFLAGS = None
running: "gcc.exe" "-O3" "-ffunction-sections" "-fdata-sections" "-m64" "-o" "C:
\\Users\\username\\Documents\\GitHub\\conrod\\target\\release\\build\\miniz-sys-f1c3
9e8e406fa25f\\out\\miniz.o" "-c" "miniz.c"
ExitStatus(ExitStatus(1))

command did not execute successfully, got: exit code: 1

--- stderr
gcc.exe: error: CreateProcess: No such file or directory
thread '<main>' panicked at 'explicit panic', C:\Users\username\.cargo\registry\src\
github.com-0a35038f75765ae4\gcc-0.3.21\src\lib.rs:772

My %PATH% (relevant parts) looks like this:

C:\Program Files\Rust stable 1.5\bin;C:\Program Files\Rust stable 1.5\bin\rustlib\x86_64-pc-windows-gnu\lib\;C:\Program Files\Rust stable 1.5\bin\rustlib\x86_64-pc-windows-gnu\bin\;C:\Program Files\Rust stable1.5\bin\;

manual executing build-scrip-build.exe:

C:\Users\username\Documents\GitHub\conrod\target\debug\build\miniz-sys-f1c39e8e406fa
25f>build-script-build.exe
thread '<main>' panicked at 'called `Option::unwrap()` on a `None` value', ../sr
c/libcore\option.rs:366

gcc -v output:

Using built-in specs.
COLLECT_GCC=gcc
Target: x86_64-w64-mingw32
Configured with: ../../../src/gcc-4.9.1/configure --host=x86_64-w64-mingw32 --bu
ild=x86_64-w64-mingw32 --target=x86_64-w64-mingw32 --prefix=/mingw64 --with-sysr
oot=/c/mingw491/x86_64-491-win32-seh-rt_v3-rev1/mingw64 --with-gxx-include-dir=/
mingw64/x86_64-w64-mingw32/include/c++ --enable-shared --enable-static --disable
-multilib --enable-languages=ada,c,c++,fortran,objc,obj-c++,lto --enable-libstdc
xx-time=yes --enable-threads=win32 --enable-libgomp --enable-libatomic --enable-
lto --enable-graphite --enable-checking=release --enable-fully-dynamic-string --
enable-version-specific-runtime-libs --disable-isl-version-check --disable-cloog
-version-check --disable-libstdcxx-pch --disable-libstdcxx-debug --enable-bootst
rap --disable-rpath --disable-win32-registry --disable-nls --disable-werror --di
sable-symvers --with-gnu-as --with-gnu-ld --with-arch=nocona --with-tune=core2 -
-with-libiconv --with-system-zlib --with-gmp=/c/mingw491/prerequisites/x86_64-w6
4-mingw32-static --with-mpfr=/c/mingw491/prerequisites/x86_64-w64-mingw32-static
 --with-mpc=/c/mingw491/prerequisites/x86_64-w64-mingw32-static --with-isl=/c/mi
ngw491/prerequisites/x86_64-w64-mingw32-static --with-cloog=/c/mingw491/prerequi
sites/x86_64-w64-mingw32-static --enable-cloog-backend=isl --with-pkgversion='x8
6_64-win32-seh-rev1, Built by MinGW-W64 project' --with-bugurl=http://sourceforg
e.net/projects/mingw-w64 CFLAGS='-O2 -pipe -I/c/mingw491/x86_64-491-win32-seh-rt
_v3-rev1/mingw64/opt/include -I/c/mingw491/prerequisites/x86_64-zlib-static/incl
ude -I/c/mingw491/prerequisites/x86_64-w64-mingw32-static/include' CXXFLAGS='-O2
 -pipe -I/c/mingw491/x86_64-491-win32-seh-rt_v3-rev1/mingw64/opt/include -I/c/mi
ngw491/prerequisites/x86_64-zlib-static/include -I/c/mingw491/prerequisites/x86_
64-w64-mingw32-static/include' CPPFLAGS= LDFLAGS='-pipe -L/c/mingw491/x86_64-491
-win32-seh-rt_v3-rev1/mingw64/opt/lib -L/c/mingw491/prerequisites/x86_64-zlib-st
atic/lib -L/c/mingw491/prerequisites/x86_64-w64-mingw32-static/lib '
Thread model: win32

For some reason I cannot compile this project: https://github.com/viperscape/font-atlas-example/tree/master/atlas-gen, specifically that sub-project atlas-gen. Below is the output during linking. I am using mingw gcc from cygwin (x86_64-w64-mingw32-gcc) with rust-nightly 32bit. It's very possible that this is an issue on my end, as I was able to compile this yesterday; I'm not sure what changed :-( I'm hesitant to install mingw directly because it's housed on sourceforge, which I try to avoid now a days. Any clues to what is going on here? Thanks!

note: C:\Users\chris\font-atlas-example\atlas-gen\target\debug\deps\libminiz_sys-d19b88f9ef21a81d.rlib(miniz.o): In function `tinfl_decompress':
/cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:1707: undefined reference to `__assert_func'
C:\Users\chris\font-atlas-example\atlas-gen\target\debug\deps\libminiz_sys-d19b88f9ef21a81d.rlib(miniz.o): In function `tdefl_start_dynamic_block':
/cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2024: undefined reference to `__assert_func'
/cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2026: undefined reference to `__assert_func'
/cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2027: undefined reference to `__assert_func'
/cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2030: undefined reference to `__assert_func'
C:\Users\chris\font-atlas-example\atlas-gen\target\debug\deps\libminiz_sys-d19b88f9ef21a81d.rlib(miniz.o):/cygdrive/c/Users/Chris/.cargo/registry/src/github.com-121aea75f9ef2ce2/miniz-sys-0.1.6/miniz.c:2031: more undefined references to `__assert_func' follow

$ gcc --version
gcc (GCC) 4.9.2
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$ rustc --version
rustc 1.5.0-nightly (168a23ebe 2015-10-01)

$ cargo --version
cargo 0.6.0-nightly (7f21e73 2015-10-01)

flate2 gets its actual compression from either libz or miniz, both C libraries. Ultimately, a pure-Rust stack is better for Rust, mostly because of build simplicity.

Write a miniz replacement in Rust, publish it to crates.io and add another compile-time feature to select it.

Then make the performance better than miniz and make it the default.

The write/read/bufread modules all repeat the same type names:

pub use gz::EncoderReader as flate2::read::GzEncoder;
pub use deflate::EncoderReader as flate2::read::DeflateEncoder;

Unfortunately the rustdoc for flate2::read::GzEncoder shows the private names:

impl<R: Read> EncoderReader<R>
fn new(r: R, level: Compression) -> EncoderReader<R>

This is a rustdoc bug but even aside from that, this pattern is confusing to people browsing the code because what they see clicking through a [src] link does not match how they will be using the library. Let's try to make the real type names match the public names they are exported as.

Relevant API guideline: https://github.com/brson/rust-api-guidelines/issues/5 Rustdoc bug: https://github.com/rust-lang/rust/issues/41072

As explained to me in http://stackoverflow.com/a/28641354/506962

However, it [ByRefReader] does not work when reading. It looks like reader::ZlibDecoder might consume all the way to the end of the underlying Reader. This could potentially be a bug or an oversight in the flate2 library.

Quadratic complexity happens when the underlying reader supplies bytes one at a time and would block. And if header flag FNAME is set and we never have a null byte to end the filename, read_gz_header keeps reading all of the partial filename and computing the CRC...

The solution is to keep a more complex structure than a simple buffer for what has already been parsed in the header (as is done in C zlib)

@alexcrichton I will put some comments in the code for your review

I'm working on an application that pulls parts of compressed files out of S3. I don't want to pull the entire file out of s3 due to file sizes. I should be able to pass part of the file to GzDecode and decompress just the chunk.

let client = S3Client::new(Region::default());
let req = GetObjectRequest {
    bucket: self.bucket.clone(),
    key: self.key.clone(),
    version_id: Some(self.version_id.clone()),
    part_number: Some(part),
    ..Default::default()
};

let response = client.get_object(req).await;
if response.is_err() {
    let err = response.err().unwrap();
    error!("{}", err);
    panic!("Unable to fetch object from S3");
}
let response = response.unwrap();
// More Code Here...

let body = response.body.unwrap();
let mut buff = BytesMut::with_capacity(512);
match body.into_async_read().read_buf(&mut buff).await {
// More Code Here...
let frozen_bytes = buff.to_vec();
let mut deflater = GzDecoder::new(&frozen_bytes[..]);
let mut s = String::new();
let read_response_length = deflater.read_to_string(&mut s);

However, I get a corrupt deflate stream error. Is it not possible to pass only part of a gzip compressed file to the GzDecoder?

This aligns build script conditions with all the #[cfg(...)] conditions in that miniz-sys shouldn't be attempted to be compiled on WASI target.

In particular, this previously caused library to fail to compile to wasm32-unknown-wasi target from a Windows host.

error[E0432]: unresolved import `libc::c_int`
   --> src/mem.rs:316:13
    |
316 |         use libc::c_int;
    |             ^^^^^^^^^^^ no `c_int` in the root

error[E0432]: unresolved imports `libc::c_int`, `libc::c_uint`, `libc::c_void`, `libc::size_t`
  --> src/ffi/c.rs:10:16
   |
10 | pub use libc::{c_int, c_uint, c_void, size_t};
   |                ^^^^^  ^^^^^^  ^^^^^^  ^^^^^^ no `size_t` in the root
   |                |      |       |
   |                |      |       no `c_void` in the root
   |                |      no `c_uint` in the root
   |                no `c_int` in the root

error[E0432]: unresolved imports `libc::c_char`, `libc::c_int`
   --> src/ffi/c.rs:381:16
    |
381 |     use libc::{c_char, c_int};
    |                ^^^^^^  ^^^^^ no `c_int` in the root
    |                |
    |                no `c_char` in the root

I was working with a dump of wikipedia data: https://dumps.wikimedia.org/enwiki/latest/enwiki-latest-abstract.xml.gz (~760M) and found that flate2 could not read the file properly.

When I would gunzip and then re-gzip the file (e.g. gunzip -c enwiki-latest-abstract.xml.gz | gzip > enwiki-latest-abstract.xml.gz) the Rust code would read the file properly.

I'm really not sure what magic bytes flate2 might not be handling properly, as far as I can tell it's a well formed gzip file :shrug:

My code looks something like this:

        use std::io::BufReader;
        let file = File::open(gzip_xml)?;
        let gz = GzDecoder::new(BufReader::new(file));
        println!("header: {:?}", gz.header());
        let mut reader = BufReader::new(gz);
        let mut line = String::new();

        use std::io::BufRead;
        let mut count = 0;
        loop {
            if count > 10 {
                break;
            }
            count += 1;
            let mut line = String::new();
            let len = reader.read_line(&mut line)?;
            println!("{} - {}", len, line);
        }

What I noticed was interesting is in the original downloaded file, the "header" that I printed had an empty filename field, the re-gzipped archive has that however. :confused:

Suddenly got this error, seems related to the new release. Note that I'm not using this crate directly. Any ideas?

Compiling flate2 v1.0.10 error[E0658]: use of unstable library feature 'rustc_private': this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via Cargo.toml instead? --> /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.10/src/lib.rs:89:1 | 89 | extern crate libc; | ^^^^^^^^^^^^^^^^^^ | = note: for more information, see https://github.com/rust-lang/rust/issues/27812

error[E0658]: use of unstable library feature 'rustc_private': this crate is being loaded from the sysroot, an unstable location; did you mean to load this crate from crates.io via Cargo.toml instead? --> /home/rust/.cargo/registry/src/github.com-1ecc6299db9ec823/flate2-1.0.10/src/ffi.rs:52:20 | 52 | pub use libc::{c_int, c_uint, c_void, size_t}; | ^^^^^ | = note: for more information, see https://github.com/rust-lang/rust/issues/27812

The combine function for the CRC hash can cause integer overflow if the amt variable is close to u32::MAX. To fix this I changed the regular add to wrapping add, the same way the update function works.

Greetings!

It would be very nice to have a method to get the current position in compressed file: this way it's possible to calculate/display progress of file processing since we know the size of compressed file, not the uncompressed.

This is what gzoffset does in zlib C library.

Add CIFuzz workflow action to have fuzzers build and run on each PR.

This is a service offered by OSS-Fuzz where flate2-rs already runs. CIFuzz can help detect regressions and catch fuzzing build issues early, and has a variety of features (see the URL above). In the current PR the fuzzers gets build on a pull request and will run for 300 seconds.

Part of #178.

I removed the text that says "the specification, however, allows ..." and the comment about bioinformatics because they make it sound like MultiGzDecoder is a rare thing that you should only enable if you need it, but it's actually the correct choice for almost all cases since it implements what the RFC considers a "gzip file."

If the gzip header contains an optional filename or comment but they are not completely contained in the buffer sent to a write::GzDecoder, then a valid header is created, missing data from these optional sections. A subsequent write call will treat the remaining header as encoded data and attempt to decode it, making the file appear to be corrupt.

This change rewrites the header parsing code to handle partial headers correctly for both Read (where WouldBlock is handled specially) and Write (where UnexpectedEof is handled specially).

The parsing code moves from bufread.rs to mod.rs to emphasize that it is also used by write.rs. write.rs no longer uses any symbols from bufread.rs.

The only test in bufread.rs tested Buffer which is no longer needed, so it was removed. The support code in the bufread tests module was used by a test in mod.rs which tests a read::GzDecoder so the support code and the test move to read.rs.

Fixes #320