A Rust crate that provides a simple interface for LZMA compression and decompression.

Related tags

Utilities rust-lzma
Overview

rust-lzma Build Status Crates.io

Documentation

This crate provides a simple interface to liblzma. LZMA is more commonly known as XZ or 7zip, (as in, files with the .xz or .7z file extension). LZMA compression is fast and aggressive, compressing better than bzip2. liblzma implements the XZ variant, so it can read and write .xz files/streams.

Two interfaces are provided. LzmaReader/LzmaWriter are generic Readers and Writers that can be composed with other Read/Write interfaces. For example, wrap them around a File and you can write data to a file while compressing it on the fly, or stream in an xz file from disk.

compress/decompress are easy to use functions for simple use cases.

See the documentation for details on usage.

Example

Cargo.toml:

[dependencies]
rust-lzma = "0.5"

main.rs:

extern crate lzma;

use lzma::LzmaWriter;
use std::io::prelude::*;
use std::fs::File;

fn main() {
	let f = File::create("foo.xz").unwrap();
	let mut f = LzmaWriter::new_compressor(f, 6).unwrap();

	write!(f, "It's a small world!").unwrap();
	f.finish().unwrap();
}
Comments
  • Error on decompressing an LZMA file (as opposed to an XZ file)

    Error on decompressing an LZMA file (as opposed to an XZ file)

    The example code works fine for me; if I write the file using:

    let f = std::fs::File::create("/tmp/test.xz").unwrap();
    let mut f = lzma::LzmaWriter::new_compressor(f, 6).unwrap();
    write!(f, "Hello LZMA World!").unwrap();
    f.finish().unwrap();
    

    (I'm new to Rust, so forgive me if my code has a weird accent.)

    Then I can read it back in with:

    let file = std::fs::File::open("/tmp/test.xz").unwrap();
    let mut file = lzma::LzmaReader::new_decompressor(file).unwrap();
    let mut contents = String::new();
    file.read_to_string(&mut contents).expect("read failed");
    println!("{}", contents);
    

    However, if I create a file on the command line:

    $ echo 'Hello LZMA World!' | lzma > /tmp/test.lzma
    $ lzmadec /tmp/test.lzma 
    Hello LZMA World!
    

    Then the above reading code (with the file name changed, of course) fails:

    thread 'main' panicked at 'read failed: Custom { kind: Other, error: Format }', libcore/result.rs:1009:5
    

    Indeed there is a difference between these two files:

    $ file /tmp/test.xz
    /tmp/test.xz: XZ compressed data
    $ file /tmp/test.lzma 
    /tmp/test.lzma: LZMA compressed data, streamed
    

    Conversely, lzmadec can't read the .xz file:

    $ lzmadec /tmp/test.xz
    lzmadec: /tmp/test.xz: File format not recognized
    

    So is the library mis-named? Can it actually read LZMA, as opposed to XZ? The compression algorithm is the same, but the container format is different.

    It's worth noting that both my compiled Rust binary and /usr/bin/lzma are linked with /usr/lib/liblzma.so.5, so the functionality should be in there somewhere.

    opened by ttencate 5
  • LzmaWriter.write should not return zero unless there's an error

    LzmaWriter.write should not return zero unless there's an error

    I've noticed that the LzmaWriter write can return zero bytes consumed, but then if called again and again, it will eventually accept more bytes.

    This means we cannot use write_all, since it will stop with error ErrorKind::WriteZero as soon as the write above returns zero.

    The workaround is ugly,

    let todo = buffer.len();
    let mut index = 0;
    while {
        let bytes = decompressor.write(&buffer[index..]).unwrap();
        index += bytes;
    
        index < todo
    } {}
    

    when it could have been just:

    decompressor.write_all(&buffer).unwrap();
    
    opened by jpap 5
  • Impl Drop for Lzmawriter

    Impl Drop for Lzmawriter

    I have implemented drop for the lzmawriter so that it is not mandatory to run finish().

    I have unfortunately run into situations where it was no longer possible for me to execute finish(). For this purpose I have implemented what is executed in finish() in Drop. With finish(), however, the return value must then be adjusted (see pull request).

    With Drop you don't get any feedback in case of an error - but you wouldn't if finish() wasn't executed. I see no disadvantage here at the moment.

    opened by ph0llux 3
  • docs.rs documentation is not working

    docs.rs documentation is not working

    For some reason, the current version of the documentation on docs.rs shows "rust-lzma-0.5.1 doesn't have any documentation."

    https://docs.rs/crate/rust-lzma/0.5.1

    opened by Taywee 3
  • [Critical] Make `write` implementaion compatible with std::io::Write

    [Critical] Make `write` implementaion compatible with std::io::Write

    If the write-method reuturns zero bytes written it is considered an error. Since it is possible, that liblzma needs to write out some internal buffers first before consuming more input, write could return 0 in some cases. Since this seems to be normal behaviour of liblzma, this patch makes sure, that all internal buffers are written out and at least some of the input is consumed.

    Fixes #13

    opened by clamydo 3
  • How to compile with Windows?

    How to compile with Windows?

    Error: `thread 'main' panicked at 'Could not find liblzma using pkg-config',

    I had to download a Ubuntu lzma package to make it work there. I downloaded a Windows bin - how do I make this compile?

    opened by David-OConnor 2
  • Return the moved Writer after finishing

    Return the moved Writer after finishing

    When creating a LzmaWriter object, the enclosed writer is moved. In order to get it back, .finish() returns it. This required the removal of the Drop implementation. But since .finish() must be called, all cleanups can be done there.

    Or am I mistaken?

    This fixes #14

    opened by clamydo 2
  • Not working (

    Not working ("Unsupported compression options") on armhf (e. g. Raspberry Pi)

    I've tried using rust-lzma on my Raspberry Pi (Debian Jessie armhf), but unfortunately, it doesn't work, it always fails when writing with "Unsupported compression options". My example:

    extern crate lzma;
    
    use lzma::LzmaWriter;
    use std::io::prelude::*;
    use std::fs::File;
    
    fn main() {
        let f = File::create("foo.xz").unwrap();
        let mut f = LzmaWriter::new_compressor(f, 6).unwrap();
    
        match write!(f, "It's a small world!") {
            Ok(_) => println!("ok"),
            Err(err) => panic!("error writing: {}", err)
        }
        f.finish().unwrap();
    }
    

    When run, I get:

    thread '<main>' panicked at 'error writing: Unsupported compression options', examples/compress.rs:13
    stack backtrace:
       1: 0x54afc747 - sys::backtrace::tracing::imp::write::hf670d8b6386d22deJcv
       2: 0x54afed67 - panicking::default_handler::_$u7b$$u7b$closure$u7d$$u7d$::closure.44482
       3: 0x54afe8f3 - panicking::default_handler::h8779de026b877d91YTz
       4: 0x54af5dc3 - sys_common::unwind::begin_unwind_inner::h3b511d4750e909fd20t
       5: 0x54af602b - sys_common::unwind::begin_unwind_fmt::hfb02698582574ecb8Zt
       6: 0x54aeeab3 - main::h476f27c94c33d42diaa
                    at /home/user/src/rust-lzma/<std macros>:8
       7: 0x54afe4bb - sys_common::unwind::try::try_fn::h5358980013828492550
       8: 0x54afbd23 - __rust_try
       9: 0x54afbd23 - __rust_try
    ...
    

    Also, all tests fail on the same error:

    thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Error { repr: Custom(Custom { kind: Other, error: Options }) }', ../src/libcore/result.rs:746
    ...
    

    I've tried using 0, 6, and 9 as compression preset, no difference. I've looked into your code, but I couldn't find any clue what could be wrong. It works on Debian Jessie x86_64 though. Thanks

    opened by superlupo12 2
  • Improve drop implementation to prevent accidental corruption

    Improve drop implementation to prevent accidental corruption

    Basically, this PR just forces LzmaWriter to call finish before ending the stream. This prevents any accidental corruptions and makes it much easier to write any structs that might contain it, and makes it behave a bit more like compressors in the flate2 and bzip2 libraries.

    opened by efyang 2
  • LzmaWriter.finish() should return the inner std::io::Write

    LzmaWriter.finish() should return the inner std::io::Write

    I can appreciate that perhaps the LzmaWriter was designed to write to files on disk, where the compressed data isn't immediately re-read from disk in the same process. However if we want to compress to memory, where the stream we write to is backed by memory, this becomes a pain.

    We currently have to hack around the stream being moved into the LzmaWriter:

    let mut backing: Vec<u8> = Vec::with_capacity(backing_size);
    {
        let mut compressor = lzma::LzmaWriter::new_compressor(&mut backing, 6).unwrap();
        // ... perform writes ...
        compressor.finish().unwrap();
    }
    // ... now use compressed data in backing.
    

    If the finish() method of the LzmaWriter returned the inner io::Write, it would reduce to:

    let mut compressor = lzma::LzmaWriter::new_compressor(Vec::with_capacity(backing_size), 6).unwrap();
    // ... perform writes ...
    let backing = compressor.finish().unwrap();
    // ... now use compressed data in backing.
    

    The flate2 library does this and makes for some clean code.

    opened by jpap 0
  • finish should consume self

    finish should consume self

    Using LzmaWriter after calling finish is not a currently supported feature, so have finish consume self instead of &self.

    Credits to krdln for the feedback.

    NOTE: liblzma supports compressing/decompressing multiple xz streams. In the future the LzmaWriter API could be enhanced with a finish_and_continue method which finishes the current stream and then starts handling the next stream. Or something like that.

    opened by fpgaminer 0
  • finish ergonomics

    finish ergonomics

    Right now, finish needs to be called on an LzmaWriter to get it to finish compressing/decompressing. This differs from the way a normal Write-like object works, so it's easy to forget and cause weird bugs.

    Investigate the suggestion of using Drop to either call finish or panic if finish hasn't been called. BufWriter implements Drop like this: https://static.rust-lang.org/doc/master/src/std/io/buffered.rs.html#501-509

    Credits to krdln for the feedback.

    opened by fpgaminer 3
  • Code duplication

    Code duplication

    LZMAReader and LZMAWriter duplicate a lot of code between each other. It would be nice to eliminate that, as well as the duplication of documentation. It's necessary to keep distinct Reader and Writer types, so they can't just be condensed into a single Struct.

    opened by fpgaminer 0
  • Handling of errors is weird

    Handling of errors is weird

    Some parts of the library return Result with an LZMAError, which is good, but the parts of the library that implement the Read and Write traits have to return io::Result. They take any LZMAError errors that arise and squish them into an io::Error. This makes handling those errors outside of the library challenging.

    Need to find a better solution.

    opened by fpgaminer 0
Owner
null
archive-rs provides a generic way of dealing with multiple archive and compression formats in Rust

archive-rs A Rust crate that aims to provide a generic way of dealing with multiple archive and compression formats by providing a generic abstraction

S.J.R. van Schaik 2 Nov 21, 2021
A cross-platform serial port library in Rust. Provides a blocking I/O interface and port enumeration including USB device information.

Note: This is a fork of the original serialport-rs project on GitLab. Please note there have been some changes to both the supported targets and which

Serialport 128 Jan 4, 2023
This plugin provides an interface for storing unencrypted values on the application cache folder.

Tauri Plugin Store This plugin provides an interface for storing unencrypted values on the application cache folder. Architecture This repo shape migh

Tauri 128 Jan 1, 2023
Rust crate which provides direct access to files within a Debian archive

debarchive This Rust crate provides direct access to files within a Debian archive. This crate is used by our debrep utility to generate the Packages

Pop!_OS 11 Dec 18, 2021
Rust crate that provides a convenient macro to quickly plot variables.

Debug Plotter This crate provides a convenient macro to quickly plot variables. Documentation For more information on how to use this crate, please ta

Fabian Bösiger 82 Dec 31, 2022
hy-rs, pronounced high rise, provides a unified and portable to the hypervisor APIs provided by various platforms.

Introduction The hy-rs crate, pronounced as high rise, provides a unified and portable interface to the hypervisor APIs provided by various platforms.

S.J.R. van Schaik 12 Nov 1, 2022
untyped-arena provides an Arena allocator implementation that is safe and untyped with minimal complexity

untyped-arena untyped-arena provides an Arena allocator implementation that is safe and untyped with minimal complexity Usage let arena = Arena::new()

Max Bruce 1 Jan 9, 2022
A translator interface between VoiceMeeter/VBAN and the X-Touch

VBAN -> XCtrl translator This serves as a network based translator for interfacing the Behringer X-Touch with VoiceMeeter Potato. It is designed to ru

Ollie Nye 4 Sep 3, 2022
This library provides a stable polyfill for Rust's Strict Provenance experiment.

This library provides a stable polyfill for Rust's Strict Provenance experiment.

Aria Beingessner 57 Nov 15, 2022
A rust interval arithmetic library which provides flags that detect domain errors.

intervals-good A Rust interval arithmetic library which provides flags that detect domain errors, supports more functions than any other interval arit

Oliver Flatt 3 Jul 27, 2022
Provides utility functions to perform a graceful shutdown on an tokio-rs based service

tokio-graceful-shutdown IMPORTANT: This crate is in an early stage and not ready for production. This crate provides utility functions to perform a gr

null 61 Jan 8, 2023
Stack buffer provides alternatives to Buf{Reader,Writer} allocated on the stack instead of the heap.

StackBuf{Reader,Writer} Stack buffer provides alternatives to BufReader and BufWriter allocated on the stack instead of the heap. Its implementation i

Alex Saveau 14 Nov 20, 2022
Stdto provides a set of functional traits for conversion between various data representations.

Stdto stdto provides a set of functional traits for conversion between various data representations. | Examples | Docs | Latest Note | stdto = "0.13.0

Doha Lee 5 Dec 21, 2022
Scaffold that provides a basic setup for a Yew app with DaisyUI for styling.

Yew and Tailwind CSS Scaffold This scaffold provides a basic setup for a Yew app with Tailwind CSS for styling. It also includes DaisyUI, a Tailwind C

Alessio Marchi 6 Mar 3, 2023
UnlimCloud provides unlimited cloud storage for your files, utilizing Telegram as the storage solution

UnlimCloud provides unlimited cloud storage for your files, utilizing Telegram as the storage solution. Simply log in using your Telegram ID, and you are good to go.

inulute 5 Nov 27, 2023
Command line interface to manage clipboard

cb Command line interface to manage clipboard How to install Pre-Compiled you can download a pre-compiled executable, then you should copy that execut

Navid 74 Dec 18, 2022
Interface definitions for the Compute@Edge platform in witx.

?? compute-at-edge-abi This repository contains witx definitions for the Compute@Edge platform ABI. About witx The witx file format is an experimental

Fastly 14 Apr 5, 2022
Transfer data with the LCU in the command-line interface! 🖥

llux llux (short for LCU Lux) is an open-source CLI tool to easily transfer data with the LCU without any script or code setup, hehe! Preview Download

Blossomi Shymae 5 Jul 30, 2023
This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to the arrow format.

Arrow2-derive - derive for Arrow2 This crate allows writing a struct in Rust and have it derive a struct of arrays layed out in memory according to th

Jorge Leitao 29 Dec 27, 2022