Rust LZ4 bindins library

Related tags

Miscellaneous lz4-rs
Overview

lz4

Build Status Crates.io GitHub license Join the chat at https://gitter.im/bozaro/lz4-rs Rustdoc

This repository contains binding for lz4 compression library (https://github.com/Cyan4973/lz4).

LZ4 is a very fast lossless compression algorithm, providing compression speed at 400 MB/s per core, with near-linear scalability for multi-threaded applications. It also features an extremely fast decoder, with speed in multiple GB/s per core, typically reaching RAM speed limits on multi-core systems.

Usage

Put this in your Cargo.toml:

[dependencies]
lz4 = "1.23.1"

Sample code for compression/decompression:

extern crate lz4;

use std::env;
use std::fs::File;
use std::io::{self, Result};
use std::path::{Path, PathBuf};

use lz4::{Decoder, EncoderBuilder};

fn main() {
    println!("LZ4 version: {}", lz4::version());

    for path in env::args().skip(1).map(PathBuf::from) {
        if let Some("lz4") = path.extension().and_then(|e| e.to_str()) {
            decompress(&path, &path.with_extension("")).unwrap();
        } else {
            compress(&path, &path.with_extension("lz4")).unwrap();
        }
    }
}

fn compress(source: &Path, destination: &Path) -> Result<()> {
    println!("Compressing: {} -> {}", source.display(), destination.display());

    let mut input_file = File::open(source)?;
    let output_file = File::create(destination)?;
    let mut encoder = EncoderBuilder::new()
        .level(4)
        .build(output_file)?;
    io::copy(&mut input_file, &mut encoder)?;
    let (_output, result) = encoder.finish();
    result
}

fn decompress(source: &Path, destination: &Path) -> Result<()> {
    println!("Decompressing: {} -> {}", source.display(), destination.display());

    let input_file = File::open(source)?;
    let mut decoder = Decoder::new(input_file)?;
    let mut output_file = File::create(destination)?;
    io::copy(&mut decoder, &mut output_file)?;

    Ok(())
}
Comments
  • Looking for new maintainer

    Looking for new maintainer

    Dear users,

    I am sorry I didn't live up to the promises I made in some of the issues and pull requests. My intention was not to deceive you, but to actually continue working on this library and to let this be my small contribution to the Rust community. However, I don't want to make any more empty promises, I won't fulfill, so I am now looking for a new maintainer for this library.

    My plan forward would be to give contributor access to a volunteer, but I just noticed, I can't even do that, since I am not a maintainer of this Github project. If someone volunteers, we should migrate to another namespace.

    Regarding the crates.io access rights, I would suggest doing the updates myself for a transitioning period to ensure the new maintainer does not have any malicious intentions.

    Cheers, Jens

    opened by jheyens 6
  • Allow the user the get the current number of bytes written to the wrapped writer.

    Allow the user the get the current number of bytes written to the wrapped writer.

    In a project I'm working on, I need to know an estimation of how much bytes have already been written to decide when to call Encoder.finish(). I need that to be able to create evenly sized compressed blocks in a bigtable like implementation.

    opened by PaulGrandperrin 4
  • Build fails on armv7

    Build fails on armv7

    Can't get this crate to compile on my pi -- it does compile on the same toolchain for x86_64 though.

    Host info:

    Linux pi 4.14.50-v7+ #1122 SMP Tue Jun 19 12:26:26 BST 2018 armv7l GNU/Linux
    stable-armv7-unknown-linux-gnueabihf (default)
    rustc 1.29.1 (b801ae664 2018-09-20)
    

    Build log:

     Compiling lz4 v1.23.0                                                                                        
    error[E0308]: mismatched types
      --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:81:21
       |
    81 |                     src.as_ptr() as *const i8,
       |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
       |
       = note: expected type `*const u8`
                  found type `*const i8`
    
    error[E0308]: mismatched types
      --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:82:21
       |
    82 |                     dst_buf.as_mut_ptr() as *mut i8,
       |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
       |
       = note: expected type `*mut u8`
                  found type `*mut i8`
    
    error[E0308]: mismatched types
      --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:90:21
       |
    90 |                     src.as_ptr() as *const i8,
       |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
       |
       = note: expected type `*const u8`
                  found type `*const i8`
    
    error[E0308]: mismatched types
      --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:91:21
       |
    91 |                     dst_buf.as_mut_ptr() as *mut i8,
       |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
       |
       = note: expected type `*mut u8`
                  found type `*mut i8`
    
    error[E0308]: mismatched types
      --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:99:21
       |
    99 |                     src.as_ptr() as *const i8,
       |                     ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
       |
       = note: expected type `*const u8`
                  found type `*const i8`
    
    error[E0308]: mismatched types
       --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:100:21
        |
    100 |                     dst_buf.as_mut_ptr() as *mut i8,
        |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
        |
        = note: expected type `*mut u8`
                   found type `*mut i8`
    
    error[E0308]: mismatched types
       --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:164:13
        |
    164 |             src.as_ptr() as *const i8,
        |             ^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
        |
        = note: expected type `*const u8`
                   found type `*const i8`
    
    error[E0308]: mismatched types
       --> /home/jtst/.cargo/registry/src/github.com-1ecc6299db9ec823/lz4-1.23.0/src/block/mod.rs:165:13
        |
    165 |             decompressed.as_mut_ptr() as *mut i8,
        |             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected u8, found i8
        |
        = note: expected type `*mut u8`
                   found type `*mut i8`
    
    opened by codeandkey 3
  • Don't try to fill up the entire buffer on a read.

    Don't try to fill up the entire buffer on a read.

    An LZ4 frame is designed so that partial blocks can be "flushed" by the encoder at any time. A frame can be a long term stream sent over a network connection. The decoder needs to be sure that it only calls the underlying read once at a time, when necessary, and be able to return partially filled buffers. This allows the caller to respond to flushed data without the decoder blocking on a network read.

    opened by bvinc83 3
  • New release?

    New release?

    Heyo,

    I'm planning on releasing a tool that uses the lz4-rs block-compression module, which is unreleased on crates.io yet. Because crates.io doesn't allow git dependencies, I'm currently stuck. Is there any plans on making a new release to make those functions available to crates.io users ? Anything I can do to help ?

    Thanks for the great dependency :)

    opened by roblabla 2
  • make write_end public

    make write_end public

    Currently calling the encoder finish method from drop is tough because it takes mut self. However, it just calls write_end underneath, which is &mut self already. Making this public will allow easily calling it from drop if the user chooses.

    opened by DrRibosome 2
  • Split out separate sys package

    Split out separate sys package

    When you have an lz4 dependency in another linked C library you don't need the full wrapper, just a way to compile and link lz4. This commit splits this part out into a separate sys crate to create this possiblity.

    opened by thijsc 2
  • Add LZ4_compress_default and LZ4_decompress_safe functions

    Add LZ4_compress_default and LZ4_decompress_safe functions

    Although the library exposes mostly LZ4F functionality, it's useful (especially as the library is called lz4) to have access to (at least) two main function calls for LZ4 compression and decompression.

    Usage may look like the following (using LZ4_decompress_safe):

        // Assume src as being &Vec<u8>
        let src_ptr = src.as_ptr();
        let compressed_size = src.len() as i32;
        let mut dst: Box<[u8]> = vec![0; max_decompressed_size].into_boxed_slice();
    
        unsafe {
            let decompressed_size = lz4_sys::LZ4_decompress_safe(
                src_ptr,
                dst.as_mut_ptr(),
                compressed_size,
                max_decompressed_size);
        }
    

    Should provide a minimum for https://github.com/bozaro/lz4-rs/issues/25

    opened by nlopes 1
  • Causes projects to always rebuild

    Causes projects to always rebuild

    When lz4 is listed anywhere in the dependencies of a project, the project must always rebuild/relink; it is never clean. I would expect the second cargo build here to report Finished, without any Compiling.

    % cargo build
       Compiling lz4 v1.21.0
       Compiling lz4rebuild v0.1.0 (file:///home/faux/code/testcases/lz4rebuild)
        Finished dev [unoptimized + debuginfo] target(s) in 0.43 secs
    % cargo build
       Compiling lz4 v1.21.0
       Compiling lz4rebuild v0.1.0 (file:///home/faux/code/testcases/lz4rebuild)
        Finished dev [unoptimized + debuginfo] target(s) in 0.43 secs
    

    This is especially annoying when repeatedly re-running tests in a project that takes a while to link.

    I don't know what's causing the problem

    The above project is just a cargo init --bin with the dependency added. Using an explicit version makes no difference:

    % cat Cargo.toml 
    [package]
    name = "lz4rebuild"
    version = "0.1.0"
    authors = ["Chris West (Faux) <[email protected]>"]
    
    [dependencies]
    lz4 = "*"
    
    
    % cat src/main.rs 
    fn main() {
        println!("Hello, world!");
    }
    
    opened by FauxFaux 1
  • Make Encoder and Decoder Send

    Make Encoder and Decoder Send

    LZ4F{Compression,Decompression}Context are raw pointers which make Encoder and Decoder non-Send. However, given that those contexts are uniquely owned by the Encoder/Decoder, it is safe to implement Send on them, making Encoder/Decoder Send themselves.

    It would have been better to use std::ptr::Unique but it is still unstable.

    opened by maximelenoir 1
  • Don't leave Decoder in invalid state if read fails

    Don't leave Decoder in invalid state if read fails

    If a read fails, such as in the case of non-blocking IO, we don't want to leave the Decoder in an invalid state. Only change the state of the Decoder if a read succeeds, otherwise, allow for calling again.

    opened by bvinc83 1
  • How to resize output buffer for use in `compress_to_buffer`?

    How to resize output buffer for use in `compress_to_buffer`?

    E.g. snap's API has a max_compress_len that users use to resize the buffer so the contents fit. However, I can't find this here. I can't also find it in the tests.

    Any ideas?

    opened by jorgecarleitao 0
  • failed to compress and decompress empty bytes

    failed to compress and decompress empty bytes

    The following code fails, which is not desired behavior

    use lz4::block::{compress,decompress};
    let v = vec![0u8; 0];
    let comp_with_prefix = compress(&v, None, true).unwrap();
    assert_eq!(v, decompress(&comp_with_prefix, None).unwrap());
    
    opened by NOBLES5E 0
  • Decoder: allow to specify the buffer size

    Decoder: allow to specify the buffer size

    Hey,

    while tweaking the encoder's parameters for my Lolcate program, I experienced a performance increase when increasing its block size (I found a sweet spot at 256 kb).

    This made me willing to try and give the decoder's buffer size different values, hoping to get similar performance increases, given that the default value seems small.

    Is there any interest in doing so, or is the default value the best one in your own experience ?

    Cheers

    opened by ngirard 0
  • Add support for dictionary compression.

    Add support for dictionary compression.

    LZ4 supports using a dictionary when processing data, which can make a big difference on small messages/files. This change exposes the relevant functions in lz4-sys, and extends Encoder and Decoder to support using dictionaries.

    opened by thoren-d 0
  • Allow delegating all linking

    Allow delegating all linking

    This is helpful when an external build system (e.g. Buck) is taking care of linking liblz4, and we don't want the Rust compiler to emit any linking arguments.

    To do so, this adds a flag to just bail out of the build script and not build any native crate if LZ4_SYS_DELEGATE_LINKING is set in the environment.

    opened by krallin 0
Owner
Artem V. Navrotskiy
Artem V. Navrotskiy
Rust library that can be reset if you think it's slow

GoodbyeKT Rust library that can be reset if you think it's slow

null 39 Jun 16, 2022
Notion Offical API client library for rust

Notion API client library for rust.

Jake Swenson 65 Dec 26, 2022
Rust library for program synthesis of string transformations from input-output examples 🔮

Synox implements program synthesis of string transformations from input-output examples. Perhaps the most well-known use of string program synthesis in end-user programs is the Flash Fill feature in Excel. These string transformations are learned from input-output examples.

Anish Athalye 21 Apr 27, 2022
SE3 Rust library for Robotics

Algebraic Robots A small Rust Library for SE3 Supported: Twist Screw SE3 Group se3 algebra Adjoint SE3 Twist Chains Wrenches Future plans: Jacobians V

Pau Carré Cardona 4 Jul 18, 2021
Rust library for emulating RISC-V rv32imac

This library can execute instructions against any memory and register file that implements the required primitives in the traits lib_rv32::traits::{Memory, RegisterFile}. This is to encourage usage with whatever frontend you desire.

Trevor McKay 14 Dec 7, 2022
Yet another ROS2 client library written in Rust

RclRust Target CI Status Document Foxy (Ubuntu 20.04) Introduction This is yet another ROS2 client library written in Rust. I have implemented it inde

rclrust 42 Dec 1, 2022
A boiler plate code to create dynamic link library in rust.

?? rust-dll-bp This is a boiler plate code that will be generated as a dll binary. I personally cache this here for me but if you're intend to create

s3pt3mb3r 9 Nov 7, 2022
Rust telegram bot library for many runtimes

Telbot Telbot provides telegram bot types and api wrappers. Specifically, telbot now supports: telbot-types: basic telegram types / requests / respons

kiwiyou 17 Dec 3, 2022
Strongly typed Gura library for Rust

Serde Gura This crate is a Rust library for using the Serde serialization framework with data in Gura file format. This library does not re-implement

Gura Config Lang 12 Nov 14, 2022
An asynchronous Rust client library for the Hashicorp Vault API

vaultrs An asynchronous Rust client library for the Hashicorp Vault API The following features are currently supported: Auth AppRole JWT/OIDC Token Us

Joshua Gilman 59 Dec 29, 2022
Culture ship names in a rust library.

General Systems Vehicles Culture Ships In case you ever needed Iain M. Banks's Culture ship names as a service. Names sourced from the pleasingly exte

C J Silverio 1 Oct 18, 2021
Modrinth API is a simple library for using, you guessed it, the Modrinth API in Rust projects

Modrinth API is a simple library for using, you guessed it, the Modrinth API in Rust projects. It uses reqwest as its HTTP(S) client and deserialises responses to typed structs using serde.

null 21 Jan 1, 2023
Wrapper library for utilizing DigitalOcean API v2 in Rust

doapi-rs Wrapper library for utilizing DigitalOcean API v2 in Rust Disclaimer This library is in alpha - it may do anything up to, and including, eati

Kevin K. 30 Nov 5, 2022
Noto-sans-mono-bitmap (Rust library)

noto-sans-mono-bitmap (Rust library) Pre-rasterized bitmap font from "Noto Sans Mono", an open font from Google. Original font files taken from: https

Philipp Schuster 21 Dec 7, 2022
Serenity is a Rust library for the Discord API

serenity Serenity is a Rust library for the Discord API. View the examples on how to make and structure a bot. Serenity supports bot login via the use

serenity 3.3k Jan 2, 2023
Deser: an experimental serialization and deserialization library for Rust

deser: an experimental serialization and deserialization library for Rust Deser is an experimental serialization system for Rust. It wants to explore

Armin Ronacher 252 Dec 29, 2022
An (unofficial) Rust library for querying db-ip.com data

db_ip An (unofficial) library for querying db-ip.com CSV databases in safe Rust. This library is not affiliated with or endorsed by db-ip.com. Be advi

Finn Bear 4 Dec 2, 2022
Quinine is a Rust library that implements atomic, lock-free, but write-once versions of containers like `Box` or `Arc`

Quinine is a Rust library that implements atomic, lock-free, but write-once versions of containers like `Box` or `Arc`

Paul Khuong 4 Feb 19, 2022
Ocular seeks to be the preferred cosmos client library UX for Rust projects

Ocular seeks to be the preferred cosmos client library UX for Rust projects. It is strongly based on lens, a go client library for blockchains built with the Cosmos SDK.

Peggy JV, Inc 34 Dec 26, 2022