Solves #5

• [ ] Compress file x.y to x.y.bz2 and decompress file x.y.bz2 to x.y
• [ ] Check if the output files already exist and fail with an error message
• [ ] Handle I/O failures with error messages

Check that all files in the workspace are formatted properly using rustfmt --check and fail the CI otherwise. As an extra the following could be improved w.r.t. to formatting

• add a vscode workspace file which recommends the rust analyzer plugin
• set "format on save" as default setting

Solves https://github.com/torfmaster/ribzip2/issues/4

• Calculate RLE data in the main thread
• Send calculated data to generate_block_data
• Move rle::rle from block_encoder to stream

Improve ergonomy of CLI

• [ ] use bzip2 logic of compression/uncompression by decompression a file named x.y.bz2 to x.y and compressing x.y to x.y.bz2
• [ ] check whether output file exists and fail with an error message
• [ ] handle I/O failures with meaningful error messages

Solves https://github.com/torfmaster/ribzip2/issues/1

• Add num_cpus crate.
• Mark --threads type as Option<usize>.
• Check whether --threads option has been set from command line. If not, request num_cpus::get().

Currently, the number of CPU defaults to 1 if it is not provided. As compression profits significantly from multi-threading it would make sense to use the num_cpus crate to use a higher default for the number of CPUs.

Following up this discussion https://github.com/torfmaster/ribzip2/pull/17/files#r835048506

Namely, that part: compute the checksum outside generate_block_data based on the amount of data read during RLE.

I am not an expert - but I expect the lib is libbzip2 and not libzip2 - a quick google seems to suggest I am right :) Feel free to merge - or to discard!

Currently, all steps of compression are distributed on threads. However, RLE consumes less then 1% of the time and as the output size of RLE is not deterministic and blows up the compressed block by 20% there is a bound of 720.000 bytes. This affects the efficiency of compression which allows up to 900.000 bytes. Thus, it is valid to do RLE in one thread and only then distribute the work on worker threads.

An attempt to solve #18. It seems to be a feasible goal. Further steps

• [ ] improve docs and make the Read implementation the preferred way
• [ ] also apply the concepts also for decompression. Open question: how to deal with domain specific errors? Logging? Create extra recovering/analysis related methods for decompression?
• [ ] attempt Write base trait implementations as well to get full compatibility

Also fyi @caglaryucekaya.

The bzip2 crate is a wrapper around the original C bzip2 implementation and has a Read/Write interface. The goal of this issue is to write such an interface as well for libribzip2 such that it could serve as a drop-in replacement for this library.

In a first step I would not make the block size configurable (as in the original implementation). The reason for the original design decision is that the complexity of the algorithm used is log n n (coming from fat pivot radix sort). We are using only linear time algorithms so run time should not be an issue (I would skip discussions about memory for now). You could either

• ignore the Compression struct in bzip2 completely
• re-interpret it as carrying information about the Huffman Code optimization taking place

At will.

This issue is mentored and I am trying to give you best support possible (on top of my every day job as developer).

The library usage is currently poorly documented. Moreover, the only existing tests testing compression / decompression end-to-end. Let's solve these problems at once!

• write round-trip unit tests for libribzip2 doing compression/decompression/compression. You can for example use the contents of cli/samples/pepper.txt which tests some of the features. You can also produce some static data larger than the block size of 900k to also test that use case
• write tests for decompression using original bzip2 compressed data. You can store a bz2 file in the source code using https://doc.rust-lang.org/std/macro.include_bytes.html
• write doc simple tests for the public library functions to explain the usage. Also run the doc tests in CI

Provide error types for compression and decompression. There should be

• a type for compression basically wrapping IO errors
• a type for decompression basically wrapping IO errors or a decompression domain specific error

As decompression errors are usually not recoverable the decompression domain specific error could be a flat enum containing the following cases (to be completed)

• selected non-existing Huffman table
• data exceeds block size of 900k (e.g. RLE or ZLE produces unexpectedly long blocks)
• crc errors
• orig_pointer out of bounds

and many more. The errors should come with speaking display implementations. Ideally the CLI should display them instead of just panicking.

ribzip2 uses a very naive representation of Huffman codes and also writes them in a naive way: it uses dynamically allocated arrays of enums. Also at other places the habit of representing bits as arrays of enums has large costs, in total at least 5% are spent during clone operations of these arrays. There are basically these places where this can be eliminated by a better internal representation, e.g. a 32 bit integer (bzip2 Huffman codes are length-limited to 17 bits writing and 20 bits reading anyway).

• [ ] replace representation of huffman codes by 32 bits integers to avoid cloning of arrays
• [ ] replace bitwriter internal representation by bytes or integers
• [ ] use bit array (represented by integers, for examples) instead of arrays of enums
• [ ] store block data more efficiently instead of just using arrays of Bit enums

Allow automatic publication of crate to crates.io on merge. One proposal would be using https://github.com/kettleby/semantic-release-rust and semantic commit format. One important requirement would be introducing a commit message lint for pull requests.