Brotli compressor and decompressor written in rust that optionally avoids the stdlib

Overview

rust-brotli

crates.io Build Status

What's new in 3.2

  • into_inner conversions for both Reader and Writer classes

What's new in 3.0

  • A fully compatible FFI for drop-in compatibiltiy with the https://github.com/google/brotli binaries
    • custom allocators fully supported
  • Multithreaded compression so multiple threads can operate in unison on a single file
  • Concatenatability mode to add the feature requested in https://github.com/google/brotli/issues/628
    • binary tool catbrotli can accomplish this if the first file was specified with -apendable and the second with -catable
  • validation mode where a file is double-checked to be able to be decompressed with the same settings; useful for benchmarking or fuzzing
  • Magic Number: where the brotli file can have a useful header with a few magic bytes, concatability info and a final output size for pre-allocating memory

What's new in 2.5

  • In 2.5 The callback also passes down an allocator to make new StaticCommands and PDFs and 256 bit floating point vectors.
  • In 2.4 The callback with the compression intermediate representation now passes a full metablock at a time. Also these items are mutable in case futher optimization is desired

What's new in 2.3

  • Flush now produces output instead of calling finish on the stream. This allows you to use the writer abstraction to get immediate output without having to resort to the CompressStream internal abstraction

Project Requirements

Direct no-stdlib port of the C brotli compressor to Rust

no dependency on the Rust stdlib: this library would be ideal for decompressing within a rust kernel among other things.

This is useful to see how C and Rust compare in an apples-to-apples comparison where the same algorithms and data structures and optimizations are employed.

Compression Usage

Rust brotli currently supports compression levels 0 - 11 They should be bitwise identical to the brotli C compression engine at compression levels 0-9 Recommended lg_window_size is between 20 and 22

With the io::Read abstraction

let mut input = brotli::CompressorReader::new(&mut io::stdin(), 4096 /* buffer size */,
                                              quality as u32, lg_window_size as u32);

then you can simply read input as you would any other io::Read class

With the io::Write abstraction

let mut writer = brotli::Compressor::new(&mut io::stdout(), 4096 /* buffer size */,
                                         quality as u32, lg_window_size as u32);

There are also methods to build Compressor Readers or Writers using the with_params static function

eg:

let params = BrotliEncoderParams::default();
// modify params to fit the application needs
let mut writer = brotli::Compressor::with_params(&mut io::stdout(), 4096 /* buffer size */,
                                         params);

or for the reader

let params = BrotliEncoderParams::default();
// modify params to fit the application needs
let mut writer = brotli::CompressorReader::with_params(&mut io::stdin(), 4096 /* buffer size */,
                                                       params);

With the Stream Copy abstraction

match brotli::BrotliCompress(&mut io::stdin(), &mut io::stdout(), &brotli_encoder_params) {
    Ok(_) => {},
    Err(e) => panic!("Error {:?}", e),
}

Decompression Usage

With the io::Read abstraction

let mut input = brotli::Decompressor::new(&mut io::stdin(), 4096 /* buffer size */);

then you can simply read input as you would any other io::Read class

With the io::Write abstraction

let mut writer = brotli::DecompressorWriter::new(&mut io::stdout(), 4096 /* buffer size */);

With the Stream Copy abstraction

match brotli::BrotliDecompress(&mut io::stdin(), &mut io::stdout()) {
    Ok(_) => {},
    Err(e) => panic!("Error {:?}", e),
}

With manual memory management

There are 3 steps to using brotli without stdlib

  1. setup the memory manager
  2. setup the BrotliState
  3. in a loop, call BrotliDecompressStream

in Detail

// at global scope declare a MemPool type -- in this case we'll choose the heap to
// avoid unsafe code, and avoid restrictions of the stack size

declare_stack_allocator_struct!(MemPool, heap);

// at local scope, make a heap allocated buffers to hold uint8's uint32's and huffman codes
let mut u8_buffer = define_allocator_memory_pool!(4096, u8, [0; 32 * 1024 * 1024], heap);
let mut u32_buffer = define_allocator_memory_pool!(4096, u32, [0; 1024 * 1024], heap);
let mut hc_buffer = define_allocator_memory_pool!(4096, HuffmanCode, [0; 4 * 1024 * 1024], heap);
let heap_u8_allocator = HeapPrealloc::<u8>::new_allocator(4096, &mut u8_buffer, bzero);
let heap_u32_allocator = HeapPrealloc::<u32>::new_allocator(4096, &mut u32_buffer, bzero);
let heap_hc_allocator = HeapPrealloc::<HuffmanCode>::new_allocator(4096, &mut hc_buffer, bzero);

// At this point no more syscalls are going to be needed since everything can come from the allocators.

// Feel free to activate SECCOMP jailing or other mechanisms to secure your application if you wish.

// Now it's possible to setup the decompressor state
let mut brotli_state = BrotliState::new(heap_u8_allocator, heap_u32_allocator, heap_hc_allocator);

// at this point the decompressor simply needs an input and output buffer and the ability to track
// the available data left in each buffer
loop {
    result = BrotliDecompressStream(&mut available_in, &mut input_offset, &input.slice(),
                                    &mut available_out, &mut output_offset, &mut output.slice_mut(),
                                    &mut written, &mut brotli_state);

    // just end the decompression if result is BrotliResult::ResultSuccess or BrotliResult::ResultFailure
}

This interface is the same interface that the C brotli decompressor uses

Also feel free to use custom allocators that invoke Box directly. This example illustrates a mechanism to avoid subsequent syscalls after the initial allocation

Using the C interface

rust-brotli is a drop-in replacement for the official http://github.com/google/brotli/ C implementation. That means you can use it from any place that supports that library. To build rust-brotli in this manner enter the c subdirectory and run make there

cd c && make

this should build c/target/release/libbrotli.so and should build the vanilla command line tool in C for compressing and decompressing any brotli file.

the libbrotli.so in c/target/release should be able to replace any other libbrotli.so file, but with all the advantages of using safe rust (except in the FFI bindings)

The code also allows a wider range of options, including forcing the prediction mode (eg UTF8 vs signed vs MSB vs LSB) and changing the weight of the literal cost from 540 to other values.

Additionally the CATABLE and APPENDABLE options are exposed and allow concatenation of files created in this manner.

Specifically CATABLE files can be concatenated in any order using the catbrotli tool and APPENDABLE files can be the first file in a sequence of catable files... eg you can combine appendable.br catable1.br catable2.br catable3.br

or simply catable0.br catable1.br catable2.br catable3.br

Comments
  • Full examples/doc would be useful

    Full examples/doc would be useful

    I'm currently attempting to use this library and I admit that I am at a loss.

    A full example of a command-line brotli encoder/decoder would be nice, for instance.

    opened by Yoric 5
  • Won't build with the latest Nightly

    Won't build with the latest Nightly

    I get a bunch of errors such as:

    ^^^^^^^^^^^^^ slice indices are of typeusizeor ranges ofusize``

    See for instance https://travis-ci.org/binast/binjs-ref/jobs/380663418#L725 .

    opened by Yoric 4
  • simd feature doesn't compile

    simd feature doesn't compile

    Problem

    The simd feature doesn't compile because of an error in packed_simd.

    Steps to reproduce

    Add dependency brotli with simd feature to any crate.

    Solution

    Switching to packed_simd_2 is probably the solution.

    Details

    Tested with Rust stable 1.54.0 and nightly 1.56.0 (2021-07-28), on Linux 5.10.52-1-lts.

    opened by ZettaScript 3
  • format rust code

    format rust code

    It was pretty difficult to read the code without it being formatted in the standard way. Contributors to this crate will be more forthcoming when things are formatted neatly.

    There's already a rustfmt config in the repo, I've just run cargo fmt, cargo fix and cargo clippy --fix.

    opened by robjtede 2
  • Can't build `c` drop-in replacement

    Can't build `c` drop-in replacement

    ➜  rust-brotli git:(master) ✗ cd c && make
    cargo build
       Compiling brotli-ffi v1.1.1 (/home/mkpankov/work/rust-brotli/c)
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
     --> src/lib.rs:8:17
      |
    8 | pub use brotli::ffi::compressor::*;
      |                 ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
     --> src/lib.rs:9:17
      |
    9 | pub use brotli::ffi::multicompress::*;
      |                 ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:10:17
       |
    10 | pub use brotli::ffi::decompressor::*;
       |                 ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:14:21
       |
    14 |     let _ = brotli::ffi::decompressor::CBrotliDecoderDecompress(0, null_mut(), null_mut(), null_mut());
       |                     ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:23:25
       |
    23 |         let _ = brotli::ffi::compressor::BrotliEncoderVersion();
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:24:25
       |
    24 |         let _ = brotli::ffi::decompressor::CBrotliDecoderCreateInstance(None, None, null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:25:25
       |
    25 | ...   let _ = brotli::ffi::decompressor::CBrotliDecoderSetParameter(null_mut(), brotli::ffi::decompressor::ffi::interface::BrotliDecoderP...
       |                       ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:25:91
       |
    25 | ...   let _ = brotli::ffi::decompressor::CBrotliDecoderSetParameter(null_mut(), brotli::ffi::decompressor::ffi::interface::BrotliDecoderP...
       |                                                                                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:26:25
       |
    26 | ...   let _ = brotli::ffi::decompressor::CBrotliDecoderDecompressStream(null_mut(), null_mut(), null_mut(), null_mut(), null_mut(), null_...
       |                       ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:28:25
       |
    28 |         let _ = brotli::ffi::decompressor::CBrotliDecoderMallocU8(null_mut(), 0);
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:29:25
       |
    29 |         let _ = brotli::ffi::decompressor::CBrotliDecoderMallocUsize(null_mut(), 0);
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:30:25
       |
    30 |         let _ = brotli::ffi::decompressor::CBrotliDecoderFreeU8(null_mut(), null_mut(), 0);
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:31:25
       |
    31 |         let _ = brotli::ffi::decompressor::CBrotliDecoderFreeUsize(null_mut(), null_mut(), 0);
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:32:25
       |
    32 |         let _ = brotli::ffi::decompressor::CBrotliDecoderDestroyInstance(null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:33:25
       |
    33 |         let _ = brotli::ffi::decompressor::CBrotliDecoderHasMoreOutput(null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:34:25
       |
    34 |         let _ = brotli::ffi::decompressor::CBrotliDecoderTakeOutput(null_mut(), null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:35:25
       |
    35 |         let _ = brotli::ffi::decompressor::CBrotliDecoderIsUsed(null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:36:25
       |
    36 |         let _ = brotli::ffi::decompressor::CBrotliDecoderIsFinished(null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:37:25
       |
    37 |         let _ = brotli::ffi::decompressor::CBrotliDecoderGetErrorCode(null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:38:25
       |
    38 |         let _ = brotli::ffi::decompressor::CBrotliDecoderGetErrorString(null_mut());
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:39:25
       |
    39 |         let _ = brotli::ffi::decompressor::CBrotliDecoderErrorString(
       |                         ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: could not find `ffi` in `brotli`
      --> src/lib.rs:40:21
       |
    40 |             brotli::ffi::decompressor::ffi::BrotliDecoderErrorCode::BROTLI_DECODER_ERROR_UNREACHABLE);
       |                     ^^^ could not find `ffi` in `brotli`
    
    error[E0433]: failed to resolve: use of undeclared type `BrotliEncoderMode`
      --> src/lib.rs:49:43
       |
    49 |         let _ = BrotliEncoderCompress(0,0,BrotliEncoderMode::BROTLI_MODE_GENERIC, 0, null_mut(), null_mut(), null_mut());
       |                                           ^^^^^^^^^^^^^^^^^ use of undeclared type `BrotliEncoderMode`
    
    error[E0433]: failed to resolve: use of undeclared type `BrotliEncoderOperation`
      --> src/lib.rs:50:57
       |
    50 | ...   let _ = BrotliEncoderCompressStream(null_mut(), BrotliEncoderOperation::BROTLI_OPERATION_FINISH, null_mut(), null_mut(), null_mut()...
       |                                                       ^^^^^^^^^^^^^^^^^^^^^^ use of undeclared type `BrotliEncoderOperation`
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderCreateInstance` in this scope
      --> src/lib.rs:41:17
       |
    41 |         let _ = BrotliEncoderCreateInstance(None, None, null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::encode::BrotliEncoderCreateInstance;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderSetParameter` in this scope
      --> src/lib.rs:42:17
       |
    42 |         let _ = BrotliEncoderSetParameter(null_mut(), brotli::enc::encode::BrotliEncoderParameter::BROTLI_PARAM_MODE, 0);
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::BrotliEncoderSetParameter;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderDestroyInstance` in this scope
      --> src/lib.rs:43:17
       |
    43 |         let _ = BrotliEncoderDestroyInstance(null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::encode::BrotliEncoderDestroyInstance;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderIsFinished` in this scope
      --> src/lib.rs:44:17
       |
    44 |         let _ = BrotliEncoderIsFinished(null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::encode::BrotliEncoderIsFinished;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderHasMoreOutput` in this scope
      --> src/lib.rs:45:17
       |
    45 |         let _ = BrotliEncoderHasMoreOutput(null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::encode::BrotliEncoderHasMoreOutput;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderTakeOutput` in this scope
      --> src/lib.rs:46:17
       |
    46 |         let _ = BrotliEncoderTakeOutput(null_mut(), null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::encode::BrotliEncoderTakeOutput;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderMaxCompressedSize` in this scope
      --> src/lib.rs:47:17
       |
    47 |         let _ = BrotliEncoderMaxCompressedSize(0);
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::BrotliEncoderMaxCompressedSize;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderSetCustomDictionary` in this scope
      --> src/lib.rs:48:17
       |
    48 |         let _ = BrotliEncoderSetCustomDictionary(null_mut(), 0, null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::encode::BrotliEncoderSetCustomDictionary;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderCompress` in this scope
       --> src/lib.rs:49:17
        |
    49  |           let _ = BrotliEncoderCompress(0,0,BrotliEncoderMode::BROTLI_MODE_GENERIC, 0, null_mut(), null_mut(), null_mut());
        |                   ^^^^^^^^^^^^^^^^^^^^^
        |
       ::: /home/mkpankov/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-3.3.2/src/enc/mod.rs:139:1
        |
    139 | / pub fn BrotliCompress<InputType, OutputType>(r: &mut InputType,
    140 | |                                              w: &mut OutputType,
    141 | |                                              params: &BrotliEncoderParams)
    142 | |                                                -> Result<usize, io::Error>
    143 | |   where InputType: Read,
    144 | |         OutputType: Write
        | |_________________________- similarly named function `BrotliCompress` defined here
        |
    help: a function with a similar name exists
        |
    49  |         let _ = BrotliCompress(0,0,BrotliEncoderMode::BROTLI_MODE_GENERIC, 0, null_mut(), null_mut(), null_mut());
        |                 ^^^^^^^^^^^^^^
    help: consider importing this function
        |
    7   | use enc::encode::BrotliEncoderCompress;
        |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderCompressStream` in this scope
        --> src/lib.rs:50:17
         |
    50   |   ...   let _ = BrotliEncoderCompressStream(null_mut(), BrotliEncoderOperation::BROTLI_OPERATION_FINISH, null_mut(), null_mut(), null_mut()...
         |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
         |
        ::: /home/mkpankov/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-decompressor-2.3.2/src/decode.rs:2643:1
         |
    2643 | / pub fn BrotliDecompressStream<AllocU8: alloc::Allocator<u8>,
    2644 | |                               AllocU32: alloc::Allocator<u32>,
    2645 | |                               AllocHC: alloc::Allocator<HuffmanCode>>
    2646 | |   (available_in: &mut usize,
    ...    |
    2653 | |    mut s: &mut BrotliState<AllocU8, AllocU32, AllocHC>)
    2654 | |    -> BrotliResult {
         | |__________________- similarly named function `BrotliDecompressStream` defined here
         |
    help: a function with a similar name exists
         |
    50   |         let _ = BrotliDecompressStream(null_mut(), BrotliEncoderOperation::BROTLI_OPERATION_FINISH, null_mut(), null_mut(), null_mut(), null_mut(), null_mut());
         |                 ^^^^^^^^^^^^^^^^^^^^^^
    help: consider importing this function
         |
    7    | use enc::encode::BrotliEncoderCompressStream;
         |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderMallocU8` in this scope
      --> src/lib.rs:51:17
       |
    51 |         let _ = BrotliEncoderMallocU8(null_mut(), 0);
       |                 ^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderFreeU8` in this scope
      --> src/lib.rs:52:17
       |
    52 |         let _ = BrotliEncoderFreeU8(null_mut(), null_mut(), 0);
       |                 ^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderMallocUsize` in this scope
      --> src/lib.rs:53:17
       |
    53 |         let _ = BrotliEncoderMallocUsize(null_mut(), 0);
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderFreeUsize` in this scope
      --> src/lib.rs:54:17
       |
    54 |         let _ = BrotliEncoderFreeUsize(null_mut(), null_mut(), 0);
       |                 ^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderMaxCompressedSizeMulti` in this scope
      --> src/lib.rs:55:17
       |
    55 |         let _ = BrotliEncoderMaxCompressedSizeMulti(0,0);
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
       |
    help: consider importing this function
       |
    7  | use enc::BrotliEncoderMaxCompressedSizeMulti;
       |
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderCompressMulti` in this scope
      --> src/lib.rs:56:17
       |
    56 |         let _ = BrotliEncoderCompressMulti(0,null_mut(), null_mut(), 0, null_mut(), null_mut(), null_mut(), 0, None, None, null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderCreateWorkPool` in this scope
      --> src/lib.rs:57:17
       |
    57 |         let _ = BrotliEncoderCreateWorkPool(0, None, None, null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderDestroyWorkPool` in this scope
      --> src/lib.rs:58:17
       |
    58 |         let _ = BrotliEncoderDestroyWorkPool(null_mut());
       |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    error[E0425]: cannot find function, tuple struct or tuple variant `BrotliEncoderCompressWorkPool` in this scope
      --> src/lib.rs:59:17
       |
    59 | ...   let _ = BrotliEncoderCompressWorkPool(null_mut(), 0, null_mut(), null_mut(), 0, null_mut(), null_mut(), null_mut(), 0 ,None, None, ...
       |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ not found in this scope
    
    Some errors have detailed explanations: E0425, E0433.
    For more information about an error, try `rustc --explain E0425`.
    error: could not compile `brotli-ffi` due to 43 previous errors
    make: *** [Makefile:31: target/debug/libbrotli.so] Error 101
    ➜  c git:(master) ✗ rustc --version
    rustc 1.55.0 (c8dfcfe04 2021-09-06)
    
    opened by mkpankov 2
  • BrotliEncode stalls forever

    BrotliEncode stalls forever

    The following stalls forever with brotli = { version = "3.3" , features = ["std"] }

    let mut src = vec![27; 10<<10];
    let mut src = Cursor::new(src);
    
    let mut dest = Vec::with_capacity(10<<9);
    let mut dest = Cursor::new(&mut dest);
    BrotliEncode(&mut src, &mut dest, &Default::default()).unwrap()
    
    opened by drahnr 2
  • Behavioral difference between google/brotli and dropbox/rust-brotli

    Behavioral difference between google/brotli and dropbox/rust-brotli

    I've been tinkering with replacing brotli with rust-brotli, and I've hit an interesting behavioral difference. The included C++ program works correctly when using the C brotli implementation, but segfaults with the Rust one:

    #include <fstream>
    #include <iostream>
    #include <streambuf>
    #include <string>
    #include <string_view>
    
    #include "brotli/encode.h"
    
    int main(int argc, char** argv) {
      if (argc < 2) {
        std::cerr << "pass a file to compress" << std::endl;
        return 1;
      }
      std::string path = argv[1];
      std::cout << "attempting to brotli-ize " << path << std::endl;
    
      std::ifstream input_file(path);
      std::string data((std::istreambuf_iterator<char>(input_file)),
                       std::istreambuf_iterator<char>());
      std::string_view dv(data);
    
      auto* state = BrotliEncoderCreateInstance(nullptr, nullptr, nullptr);
      if (state == nullptr) {
        std::cerr << "failed to make brotli encoder" << std::endl;
        return 1;
      }
      if (!BrotliEncoderSetParameter(state, BROTLI_PARAM_SIZE_HINT,
                                     dv.size())) {
        std::cerr << "failed to give brotli a size hint" << std::endl;
        return 1;
      }
      size_t in_left, out_left, written = 0;
      BROTLI_BOOL result = BROTLI_TRUE;
      while (dv.size() && result) {
        in_left = dv.size();
        out_left = 0;
        const uint8_t* next_in = reinterpret_cast<const uint8_t*>(dv.data());
        std::cout << "attempt to compress " << dv.size() << std::endl;
        result =
            BrotliEncoderCompressStream(state, BROTLI_OPERATION_PROCESS, &in_left,
                                        &next_in, &out_left, nullptr, nullptr);
        size_t buffer_size = 0;
        /*const uint8_t* buf = */ BrotliEncoderTakeOutput(state, &buffer_size);
        if (buffer_size) {
          written += buffer_size;
        }
        dv.remove_prefix(dv.size() - in_left);
      }
      while (result && !BrotliEncoderIsFinished(state)) {
        in_left = 0;
        out_left = 0;
        const uint8_t* next_in = nullptr;
        result = BrotliEncoderCompressStream(
            state, BROTLI_OPERATION_FINISH, &in_left, &next_in,
            &out_left, nullptr, nullptr);
        size_t buffer_size = 0;
        /*const uint8_t* buffer = */BrotliEncoderTakeOutput(state, &buffer_size);
        if (buffer_size > 0) {
          written += buffer_size;
        }
      }
      BrotliEncoderDestroyInstance(state);
      std::cout << "brotli'd down to " << written << " bytes" << std::endl;
      return 0;
    }
    

    I'm not familiar enough with the internals of rust-brotli to figure this out quickly, but I was hoping by filing a bug with a reduced test case might help someone figure this out. I think this might be related to #44, but I don't appear to need multiple chunks to trigger the segfault. As far as I can tell, the way this test program uses brotli is supported per the comments in encode.h, so I assume that the crasher in Rust is a bug of missing functionality and not merely a guard against an unexpected NULL.

    opened by durin42 2
  • Provide Rust only interface

    Provide Rust only interface

    Having the C FFI interface exported via no_mangle functions when brotli is used as an rlib dependency causes issues when linking into a project that also depends on the C brotli library (see https://github.com/Nemo157/async-compression/issues/83).

    I'm not sure what the best approach for this would be; as someone that has not worked on a library exposing a C ABI, it seems to me that this would work better as a separate crate simply wrapping a Rustic implementation, instead of being combined into a single crate. Another idea I had was putting the ffi module behind a cargo feature. But neither of these feel like the perfect solution to me.

    opened by Nemo157 2
  • Compression mishandles errors from underlying reader

    Compression mishandles errors from underlying reader

    I was using this library with a wrapped reader that can timeout to bound search time for a best compression algorithm. The reader returns an Error(ErrorKind::TimedOut) but brotli doesn't report it. I was looking at the code and it appears that https://github.com/dropbox/rust-brotli/blob/master/src/enc/mod.rs#L249 doesn't follow the contract of readers. The only error that is ignorable is an interrupt, but all errors are ignored and the stream happily closes even though it's only read like 2% of the data. Needless to say this makes it impossible to get the original data back when the BrotliCompress function returns Ok(_).

    opened by tylerhawkes 2
  • impl Default for BrotliEncoderParams and documentation of the parameters

    impl Default for BrotliEncoderParams and documentation of the parameters

    Hello.

    I'm using this library and I have noticed that the struct BrotliEncoderParams has many and not documented fields.

    It would nice having a new method or a Default implementation for this struct with sane defaults.

    For instance, quality 1 and lgwin 22 are the values I am using, and they provide a fast compression with the recommended value for the window size. There are many other attributes and I am slightly lost however.

    Updating the README should be also desirable, since the BrotliCompress method receives quality and lgwin as parameters instead BrotliEncoderParams in the README. I have noticed it because a compilation error, and I have needed navigating into the source code to find the correct BrotliCompress signature.

    Thanks and regards.

    opened by Martin1887 2
  • Overflow on 32bit target

    Overflow on 32bit target

    I've been working on some WebAssembly projects using the wasm32-unknown-unknown target and I sometimes get a panic / abort.

    To reproduce this issue, I made a small crate which one can use with cargo run and a target of i686-unknown-linux-gnu (this may fail to build on 64bit machines; I had to use a 32bit Ubuntu VM). Running that gives this error:

    thread 'main' panicked at 'attempt to shift right with overflow', /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/backward_references.rs:1131:34
    

    Cargo.toml

    [package]
    name = "test-brotli-32bit-target"
    version = "0.0.0"
    
    [dependencies]
    brotli = "1.1.0"
    

    main.rs

    extern crate brotli;
    
    use brotli::{BrotliCompress, BrotliDecompress};
    use brotli::enc::{BrotliEncoderInitParams};
    
    fn main() {
        let mut input: &[u8] = b"
          <!doctype html>
          <html>
            <head>
              <title>Testing!</title>
            </head>
            <body>
              <h1>Hello, world!</h1>
            </body>
          </html>
        ";
    
        let mut compressed_input = vec![];
        let params = BrotliEncoderInitParams();
        BrotliCompress(&mut input, &mut compressed_input, &params).unwrap();
    
        let mut result = vec![];
        match BrotliDecompress(&mut compressed_input.as_slice(), &mut result) {
            Ok(_) => println!("Success! {:?}", result),
            Err(e) => println!("Error! {:?}", e),
        }
    }
    
    Here's the full Rust backtrace
        
        Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
         Running `target/debug/test-brotli-32bit-target`
    thread 'main' panicked at 'attempt to shift right with overflow', /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/backward_references.rs:1131:34
    stack backtrace:
       0: 0x8012d18d - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h56cb4e8fab26ca56
                           at /checkout/src/libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
       1: 0x8012f603 - std::sys_common::backtrace::print::hdde3d983397ec5dc
                           at /checkout/src/libstd/sys_common/backtrace.rs:68
                           at /checkout/src/libstd/sys_common/backtrace.rs:57
       2: 0x80129e96 - std::panicking::default_hook::{{closure}}::hf7f4c21e0cce9605
                           at /checkout/src/libstd/panicking.rs:381
       3: 0x80129aa6 - std::panicking::default_hook::h0273df4f31ad7ce9
                           at /checkout/src/libstd/panicking.rs:397
       4: 0x8012a245 - std::panicking::rust_panic_with_hook::hb110eb893cf2f262
                           at /checkout/src/libstd/panicking.rs:577
       5: 0x8012a10a - std::panicking::begin_panic::hba4f7483f98fa441
                           at /checkout/src/libstd/panicking.rs:538
       6: 0x8012a026 - std::panicking::begin_panic_fmt::hf4a59e9c0b17558b
                           at /checkout/src/libstd/panicking.rs:522
       7: 0x80129f9b - rust_begin_unwind
                           at /checkout/src/libstd/panicking.rs:498
       8: 0x8017728e - core::panicking::panic_fmt::h34d249fd2e4d796e
                           at /checkout/src/libcore/panicking.rs:71
       9: 0x8017717c - core::panicking::panic::h912f4f8254e1a3ab
                           at /checkout/src/libcore/panicking.rs:51
      10: 0x800f923c - brotli::enc::backward_references::TestStaticDictionaryItem::h8e3ceceaf96520b2
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/backward_references.rs:1131
      11: 0x80042d3e - brotli::enc::backward_references::SearchInStaticDictionary::hbe772f9a65587fc3
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/backward_references.rs:1171
      12: 0x80035f59 -  as brotli::enc::backward_references::AnyHasher>::FindLongestMatch::h0b8eb4ddd2cce0b7
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/backward_references.rs:997
      13: 0x8003e19c - brotli::enc::backward_references::CreateBackwardReferences::he3119ac7c3437ed4
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/backward_references.rs:1348
      14: 0x8004458d - brotli::enc::backward_references::BrotliCreateBackwardReferences::h4ce86d42244db8fa
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/backward_references.rs:1535
      15: 0x80087f11 - brotli::enc::encode::EncodeData::h2155e11331a85a0b
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/encode.rs:2799
      16: 0x800907c1 - brotli::enc::encode::BrotliEncoderCompressStream::hc8b4e371187e8bca
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/encode.rs:3366
      17: 0x800b082d - brotli::enc::BrotliCompressCustomIo::hbb4c34f675cafcbe
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/mod.rs:240
      18: 0x800b12c3 - brotli::enc::BrotliCompressCustomAlloc::h2dfef9c4c5059e19
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/mod.rs:138
      19: 0x800b0184 - brotli::enc::BrotliCompress::he08961ea1a50f683
                           at /home/osboxes/.cargo/registry/src/github.com-1ecc6299db9ec823/brotli-1.1.0/src/enc/mod.rs:65
      20: 0x800e831f - test_brotli_32bit_target::main::hf8054a0ec07e1298
                           at src/main.rs:21
      21: 0x800e8212 - std::rt::lang_start::{{closure}}::he8b899603e478150
                           at /checkout/src/libstd/rt.rs:74
      22: 0x8012f97c - std::sys_common::backtrace::__rust_begin_short_backtrace::h72d10003122a0a61
                           at /checkout/src/libstd/rt.rs:59
                           at /checkout/src/libstd/sys_common/backtrace.rs:133
      23: 0x80129f52 - std::panicking::try::do_call::hb55332b2ef16d564
                           at /checkout/src/libstd/rt.rs:59
                           at /checkout/src/libstd/panicking.rs:480
      24: 0x8013d2d2 - __rust_maybe_catch_panic
                           at /checkout/src/libpanic_unwind/lib.rs:101
      25: 0x8012cf65 - std::rt::lang_start_internal::h2cf06419fef56d5f
                           at /checkout/src/libstd/panicking.rs:459
                           at /checkout/src/libstd/panic.rs:365
                           at /checkout/src/libstd/rt.rs:58
      26: 0x800e81e1 - std::rt::lang_start::h2e896fbd6e6fa04c
                           at /checkout/src/libstd/rt.rs:74
      27: 0x800e878a - main
      28: 0xb758f275 - __libc_start_main
      29: 0x80010c4d - 
        
      
    opened by dfrankland 2
  • Add a helper for compressing a slice into a vec

    Add a helper for compressing a slice into a vec

    A common use case is compressing a &[u8] into a Vec<u8> without doing any I/O. The current docs don't make it clear how to do this.

    A google search turns up this forum post with this code snippet:

    use std::io::Write;
    
    pub fn compress(input: &[u8]) -> Vec<u8> {
        let mut writer = brotli::CompressorWriter::new(
            Vec::new(),
            4096,
            11,
            22);
        writer.write_all(input).unwrap();
        writer.into_inner()
    }
    

    Which is quite verbose. A helper functions in the root of the crate to compress a &[u8] into a Vec<u8> and to decompress a Vec<u8> into a &[u8] would help a lot with this. (Also, if these functions don't do I/O then perhaps they could be be infallible.)

    opened by casey 0
  • Strip out all mentions of project gutenberg from the testdata

    Strip out all mentions of project gutenberg from the testdata

    This affects the file testdata/random_then_unicode file,

    This is done in order to not have the project fall under the Gutenberg license, as that forbids commercial redistribution of the software without an 20% royalty fee (clause 1.E.8. of the Gutenberg license).

    The problem with the Gutenberg license is that it not compatible with Debians requirement that the things it distributes should be free to reuse, even for commercial purposes. See https://lists.debian.org/debian-legal/2017/08/msg00001.html

    This PR removes the Gutenberg meta-data from the file, but keeps the actual ebook part.

    opened by alexanderkjall 1
  • into_inner on Decompressor discards data due to buffering

    into_inner on Decompressor discards data due to buffering

    Steps to reproduce:

    1. Decompress a Brotli stream from some input data.
    2. Call into_inner(), then read() the result to check if there is any remaining unread input data (for instance, extra junk at the end of the compressed stream).

    Expected result:

    If there is unread input data, it is readable.

    Actual result:

    If there is unread input data, it may or may not be available after calling into_inner(), depending on whether it was in the buffer or not.

    Proposed fix:

    If a fix for #83 is implemented, Decompressor will have the property that reading Ok(0) from it means the entire input data was consumed. In that situation, into_inner() can safely be called. Decompressor's into_inner() could be documented to only be safe after reading Ok(0) from Decompressor.

    This would be useful in the ureq HTTP client, where the inner reader might be a PoolReturnRead that returns a connection to a connection pool once a response body has been fully read. In that situation, we need to check for any unread bytes at the end of the response body, which could indicate a desynchronized HTTP connection that we need to drop.

    Test code to reproduce:

        #[test]
        fn test_no_vanishing_bytes() {
            // Output from this command:
            // (brotli -c <<<"hello"; echo goodbye) | xxd -p | sed 's/../\\x&/g'
            let compressed_with_extra = b"\x8f\x02\x80\x68\x65\x6c\x6c\x6f\x0a\x03\x67\x6f\x6f\x64\x62\x79\x65\x0a";
            let mut cursor = Cursor::new(compressed_with_extra);
            let mut reader = brotli_decompressor::Decompressor::new(cursor, 8000);
            std::io::read_to_string(&mut reader).unwrap();
            let unwrapped_cursor = reader.into_inner();
            assert_eq!(std::io::read_to_string(unwrapped_cursor).unwrap(), "goodbye".to_string());
        }
    
    opened by jsha 0
  • corrupt stream with extra bytes is not rejected

    corrupt stream with extra bytes is not rejected

    Steps to reproduce:

    Read a Brotli stream with extra junk at the end.

    Expected result:

    Receive an Err from brotli_decompressor::Decompressor::read.

    Actual result:

    No error.

    For comparison, the brotli command line tool rejects streams with extra junk at the end:

    $ (brotli -c <<<"hello"; echo goodbye) > corrupt.br
    $ brotli -d < corrupt.br
    hello
    corrupt input [con]
    $ echo $?
    1
    

    Test code to reproduce:

        #[test]
        fn test_no_leftovers() {
            // Output from this command:
            // (brotli -c <<<"hello"; echo goodbye) | xxd -p | sed 's/../\\x&/g'
            let compressed_with_extra = b"\x8f\x02\x80\x68\x65\x6c\x6c\x6f\x0a\x03\x67\x6f\x6f\x64\x62\x79\x65\x0a";
            let mut reader = brotli_decompressor::Decompressor::new(Cursor::new(compressed_with_extra), 8000);
            assert!(std::io::read_to_string(&mut reader).is_err());
        }
    
    opened by jsha 0
  • Generate a man page with the help of the man crate

    Generate a man page with the help of the man crate

    Generate a manual page and place it in the generated target directory.

    This is useful when packaging the brotli binary in linux distributions, the debian linter lintian complains when there is a missing manual page.

    The description of the different options are bare bones at best, it was what I found out by a quick read of the source code. Suggestions of improvements are very welcome :)

    opened by alexanderkjall 1
Releases(1.0.110)
A Brotli implementation in pure and safe Rust

Brotli-rs - Brotli decompression in pure, safe Rust Documentation Compression provides a <Read>-struct to wrap a Brotli-compressed stream. A consumer

Thomas Pickert 59 Oct 7, 2022
Brotlic (or BrotliC) is a thin wrapper around brotli.

Bindings to the brotli library featuring a low-overhead encoder and decoder, Writers and Readers for compression and decompression at customizable compression qualities and window sizes.

Aron Parker 18 Dec 9, 2022
An extremely fast alternative to zip which is written in rust.

Zap Compress and/or encrypt folders fast. Like, really fast. or as some say, blazingly fast. Installation To install Zap, run the following command fr

null 39 Dec 23, 2022
An extremely fast alternative to zip which is written in rust.

Zap Compress and/or encrypt folders fast. Like, really fast. or as some say, blazingly fast. Installation To install Zap, run the following command fr

null 37 Nov 9, 2022
Simple NoNG songs manager for GD, written in Rust.

nong-manager Simple NoNG songs manager for GD, written in Rust. Powered by Song File Hub (https://songfilehub.com/home) How to use Enter song ID that

Alexander Simonov 4 May 13, 2023
Basic (and naïve) LZW and Huffman compression algorithms in Rust.

Naive implementation of the LZW and Huffman compression algorithms. To run, install the Rust toolchain. Cargo may be used to compile the source. Examp

Luiz Felipe Gonçalves 9 May 22, 2023
A utility that can download JavaScript and TypeScript module graphs and store them locally in a special zip file.

eszip A utility that can download JavaScript and TypeScript module graphs and store them locally in a special zip file. To create a new archive: > esz

Deno Land 162 Dec 24, 2022
A simple rust library to read and write Zip archives, which is also my pet project for learning Rust

rust-zip A simple rust library to read and write Zip archives, which is also my pet project for learning Rust. At the moment you can list the files in

Kang Seonghoon 2 Jan 5, 2022
DEFLATE, gzip, and zlib bindings for Rust

flate2 A streaming compression/decompression library DEFLATE-based streams in Rust. This crate by default uses the miniz_oxide crate, a port of miniz.

The Rust Programming Language 619 Jan 8, 2023
Like pigz, but rust - a cross platform, fast, compression and decompression tool.

?? crabz Like pigz, but rust. A cross platform, fast, compression and decompression tool. Synopsis This is currently a proof of concept CLI tool using

Seth 232 Jan 2, 2023
A Rust application that compress files and folders

Quick Storer This is a Rust application that compress files and folders. Usage Download or build the binary and place it on your desktop, or any other

AL68 & co. 1 Feb 2, 2022
SIMD Floating point and integer compressed vector library

compressed_vec Floating point and integer compressed vector library, SIMD-enabled for fast processing/iteration over compressed representations. This

Evan Chan 56 Nov 24, 2022
Convenience library for reading and writing compressed files/streams

compress_io Convenience library for reading and writing compressed files/streams The aim of compress_io is to make it simple for an application to sup

Simon Heath 0 Dec 16, 2021
Obvious Unified Compression Helper is a CLI tool to help you compress and decompress files of several formats

Ouch! ouch stands for Obvious Unified Compression Helper and is a CLI tool to help you compress and decompress files of several formats. Features Usag

null 734 Dec 30, 2022
libbz2 (bzip2 compression) bindings for Rust

bzip2 Documentation A streaming compression/decompression library for rust with bindings to libbz2. # Cargo.toml [dependencies] bzip2 = "0.4" License

Alex Crichton 67 Dec 27, 2022
A Rust implementation of the Zopfli compression algorithm.

Zopfli in Rust This is a reimplementation of the Zopfli compression tool in Rust. I have totally ignored zopflipng. More info about why and how I did

Carol (Nichols || Goulding) 76 Oct 20, 2022
Snappy bindings for Rust

Snappy [ Originally forked from https://github.com/thestinger/rust-snappy ] Documentation Usage Add this to your Cargo.toml: [dependencies] snappy = "

Jeff Belgum 14 Jan 21, 2022
Tar file reading/writing for Rust

tar-rs Documentation A tar archive reading/writing library for Rust. # Cargo.toml [dependencies] tar = "0.4" Reading an archive extern crate tar; use

Alex Crichton 490 Dec 30, 2022
Zip implementation in Rust

zip-rs Documentation Info A zip library for rust which supports reading and writing of simple ZIP files. Supported compression formats: stored (i.e. n

null 549 Jan 4, 2023