Fastest Snappy compression library in Node.js

Overview

snappy

https://github.com/Brooooooklyn/snappy/actions Install size

!!! For [email protected] and below, please go to node-snappy.

More background about the 6-7 changes, please read this, Thanks @kesla .

πŸš€ Help me to become a full-time open-source developer by sponsoring me on Github

Fastest Snappy compression library in Node.js, powered by napi-rs and rust-snappy.

For small size data, snappyjs is faster, and it support browser. But it doesn't have async API, which is important for Node.js program.

Install this package

yarn add snappy

Support matrix

node12 node14 node16
Windows x64 βœ“ βœ“ βœ“
Windows x32 βœ“ βœ“ βœ“
Windows arm64 βœ“ βœ“ βœ“
macOS x64 βœ“ βœ“ βœ“
macOS arm64 βœ“ βœ“ βœ“
Linux x64 gnu βœ“ βœ“ βœ“
Linux x64 musl βœ“ βœ“ βœ“
Linux arm gnu βœ“ βœ“ βœ“
Linux arm64 gnu βœ“ βœ“ βœ“
Linux arm64 musl βœ“ βœ“ βœ“
Android arm64 βœ“ βœ“ βœ“
FreeBSD x64 βœ“ βœ“ βœ“

API

export function compressSync(input: Buffer | string | ArrayBuffer | Uint8Array): Buffer
export function compress(input: Buffer | string | ArrayBuffer | Uint8Array): Promise<Buffer>
export function uncompressSync(compressed: Buffer): Buffer
export function uncompress(compressed: Buffer): Promise<Buffer>

Performance

Hardware

Model Name: MacBook Pro
Model Identifier: MacBookPro15,1
Processor Name: 6-Core Intel Core i7
Processor Speed: 2.6 GHz
Number of Processors: 1
Total Number of Cores: 6
L2 Cache (per Core): 256 KB
L3 Cache: 12 MB
Hyper-Threading Technology: Enabled
Memory: 16 GB

Result

Running "Compress" suite...
Progress: 25%

  snappy:
    1 426 ops/s, Β±2.26%

  gzip:
    152 ops/s, Β±1.54%

  deflate:
    155 ops/s, Β±2.14%

  brotli:
    3 ops/s, Β±3.43%       | slowest, 99.79% slower

Finished 4 cases!
  Fastest: snappy
  Slowest: brotli

Running "Decompress" suite...
Progress: 25%

  snappy:
    2 771 ops/s, Β±1.13%

  gzip:
    854 ops/s, Β±6.99%

  deflate:
    877 ops/s, Β±3.19%

  brotli:
    638 ops/s, Β±2.31%     | slowest, 76.98% slower

Finished 4 cases!
  Fastest: snappy
  Slowest: brotli
Comments
  • Memory leak in 7.2.0

    Memory leak in 7.2.0

    Upgrading from 7.1.2 to 7.2.0 caused a memory leak in my application. We use snappy heavily to compress/decompress (synchronously) massive amount of data.

    I can enable/disable leak by changing snappy version and nothing else. Will slowly eat memory until app crashes in 7.2.0, while will continue forever in 7.1.2

    bug 
    opened by frelars 15
  • Take over official snappy package on npm

    Take over official snappy package on npm

    Hi!

    Are you interested in taking over the snappy-package on npm?

    snappy is the first serious npm module I ever published, so I gotta be honest that it'll be a bit difficult to let it go, but since this project is snappier than mine, I think it would make sense.

    My suggestion would be that I add you as a maintainer to the snappy package and that you publish a new version of it (as version 7.0.0 so that no existing installations of snappy are affected). I would than also deprecate my repo on github, sending them your way instead.

    Let me know what you think about my suggestion, I think it would be a cool thing to do.

    opened by kesla 10
  • Optional dependencies are not installed

    Optional dependencies are not installed

    Hi! The package @napi-rs/snappy-darwin-x64 is not derived after the installation of the latest snappy. What may be the reason? Can not load bindings also appears until the latter is installed.

    $ npm -v
    7.11.2
    
    opened by xamgore 5
  • Incompatible with electron v21

    Incompatible with electron v21

    Test code (executed as: npx electron index.js):

    const {compress} = require("snappy");
    const {randomBytes} = require("crypto");
    const {writeFile} = require("fs/promises");
    
    (async () => {
        await writeFile(
            `./snappy-archive-${Date.now()}.bin`,
            await compress(
                randomBytes(/*1 KB:*/1024 * 100),
            ),
        );
        process.exit(0);
    })();
    

    SIGSEGV error:

    [28878:1001/195712.022506:ERROR:node_bindings.cc(146)] Fatal error in V8: v8_ArrayBuffer_NewBackingStore When the V8 Sandbox is enabled, ArrayBuffer backing stores must be allocated inside the sandbox address space. Please use an appropriate ArrayBuffer::Allocator to allocate these buffers.
    [28915:0100/000000.927082:ERROR:broker_posix.cc(43)] Invalid node channel message
    
    

    Possibly related issue: https://github.com/nodejs/abi-stable-node/issues/441.

    opened by vladimiry 3
  • Search ldd in the $PATH

    Search ldd in the $PATH

    I get the following crash on NixOS:

    Error: ENOENT: no such file or directory, open '/usr/bin/ldd'
    

    On NixOS, ldd is in an exotic location:

    > which ldd
    /run/current-system/sw/bin/ldd
    
    opened by getkey 3
  • benchmarks

    benchmarks

    Hello, I see in the release 7.1.1 snappy has an increase in performance of 2-3 times. But I did some local tests and I don't note the difference in performance between version 7.0.5 and7.1.1. I note that you have changed the computer for the benchmarks, is it possible you have compared results of different hardware?

    opened by antoniomuso 2
  • Error: Cannot find module 'snappy-linux-x64-gnu'

    Error: Cannot find module 'snappy-linux-x64-gnu'

    Hello,

    I'm having an error with version 7.1.0 of snappy.

    Steps to reproduce:

    mkdir test && cd test
    npm init -y
    npm i [email protected]
    node -e "require('snappy')"
    

    Error:

    Error: Cannot find module 'snappy-linux-x64-gnu'
    Require stack:
    - /tmp/node_modules/snappy/index.js
    - /tmp/[eval]
        at Function.Module._resolveFilename (internal/modules/cjs/loader.js:902:15)
        at Function.Module._load (internal/modules/cjs/loader.js:746:27)
        at Module.require (internal/modules/cjs/loader.js:974:19)
        at require (internal/modules/cjs/helpers.js:93:18)
        at Object.<anonymous> (/tmp/node_modules/snappy/index.js:135:31)
        at Module._compile (internal/modules/cjs/loader.js:1085:14)
        at Object.Module._extensions..js (internal/modules/cjs/loader.js:1114:10)
        at Module.load (internal/modules/cjs/loader.js:950:32)
        at Function.Module._load (internal/modules/cjs/loader.js:790:12)
        at Module.require (internal/modules/cjs/loader.js:974:19) {
      code: 'MODULE_NOT_FOUND',
      requireStack: [ '/tmp/node_modules/snappy/index.js', '/tmp/[eval]' ]
    }
    

    Environment:

    • node 14.18.2
    • npm 6.14.15
    • Linux 5.15.10-arch1-1 x86_64 GNU/Linux

    Is this a bug or I'm missing something? Thanks

    bug 
    opened by vitorveiga 2
  • chore(deps): update dependency legacy-snappy to v7

    chore(deps): update dependency legacy-snappy to v7

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Change | Age | Adoption | Passing | Confidence | |---|---|---|---|---|---| | legacy-snappy | 6 -> 7 | age | adoption | passing | confidence |


    Release Notes

    Brooooooklyn/snappy

    v7.1.1

    Compare Source

    Bug Fixes
    • override the package name in generated index.js (42346fd)

    v7.1.0

    Compare Source

    Features

    7.0.5 (2021-11-06)

    Bug Fixes

    7.0.4 (2021-10-28)

    Bug Fixes
    • avoid copy input buffer (00cdd39)

    7.0.3 (2021-08-27)

    Bug Fixes
    • remove Ref usage to avoid memory leak (c89bb2e)

    7.0.2 (2021-08-11)

    Bug Fixes
    • missing asBuffer option in uncompress/uncompressSync (ac573f8)

    7.0.1 (2021-08-04)

    Bug Fixes
    • native binding package name (6dc09af)

    v7.0.5

    Compare Source

    Bug Fixes

    v7.0.4

    Compare Source

    Bug Fixes
    • avoid copy input buffer (00cdd39)

    v7.0.3

    Compare Source

    Bug Fixes
    • remove Ref usage to avoid memory leak (c89bb2e)

    v7.0.2

    Compare Source

    Bug Fixes
    • missing asBuffer option in uncompress/uncompressSync (ac573f8)

    v7.0.1

    Compare Source

    Bug Fixes
    • native binding package name (6dc09af)

    v7.0.0

    Compare Source

    Change package name to snappy #​16 .

    1.0.2 (2021-07-22)

    Bug Fixes
    • linux aarch64 musl build (1a9a475)

    1.0.1 (2021-06-10)

    Performance Improvements
    • mimalloc as global allocator (3fbab59)

    Configuration

    πŸ“… Schedule: At any time (no schedule defined).

    🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, click this checkbox.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

    opened by renovate[bot] 1
  • Memory leak

    Memory leak

    I deployed this in production yesterday for usage with MongoDB driver. I was using version 6.x before. When I deployed, memory usage continually kept going up. I am almost certain that this package caused it.

    Let me know what information I can give to track it down.

    bug 
    opened by adityapatadia 1
  • chore(deps): roll back dependency @napi-rs/snappy-darwin-arm64 to ^1.0.2 - autoclosed

    chore(deps): roll back dependency @napi-rs/snappy-darwin-arm64 to ^1.0.2 - autoclosed

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | @napi-rs/snappy-darwin-arm64 | optionalDependencies | rollback | ^7.0.0 -> ^1.0.2 |


    Configuration

    πŸ“… Schedule: At any time (no schedule defined).

    🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

    opened by renovate[bot] 1
  • chore(deps): roll back dependency @napi-rs/snappy-android-arm64 to ^1.0.2 - autoclosed

    chore(deps): roll back dependency @napi-rs/snappy-android-arm64 to ^1.0.2 - autoclosed

    WhiteSource Renovate

    This PR contains the following updates:

    | Package | Type | Update | Change | |---|---|---|---| | @napi-rs/snappy-android-arm64 | optionalDependencies | rollback | ^7.0.0 -> ^1.0.2 |


    Configuration

    πŸ“… Schedule: At any time (no schedule defined).

    🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.

    β™» Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

    πŸ”• Ignore: Close this PR and you won't be reminded about this update again.


    • [ ] If you want to rebase/retry this PR, check this box.

    This PR has been generated by WhiteSource Renovate. View repository job log here.

    opened by renovate[bot] 1
Releases(v7.2.2)
Owner
LongYinan
TypeScript at day, Rustacean at night. Coding for fun.
LongYinan
Snappy bindings for Rust

Snappy [ Originally forked from https://github.com/thestinger/rust-snappy ] Documentation Usage Add this to your Cargo.toml: [dependencies] snappy = "

Jeff Belgum 14 Jan 21, 2022
libbz2 (bzip2 compression) bindings for Rust

bzip2 Documentation A streaming compression/decompression library for rust with bindings to libbz2. # Cargo.toml [dependencies] bzip2 = "0.4" License

Alex Crichton 67 Dec 27, 2022
A Rust implementation of the Zopfli compression algorithm.

Zopfli in Rust This is a reimplementation of the Zopfli compression tool in Rust. I have totally ignored zopflipng. More info about why and how I did

Carol (Nichols || Goulding) 76 Oct 20, 2022
(WIP) Taking the pain away from file (de)compression

Ouch! ouch loosely stands for Obvious Unified Compression files Helper and aims to be an easy and intuitive way of compressing and decompressing files

VinΓ­cius Miguel 734 Dec 30, 2022
gzp - Multi-threaded Compression

gzp - Multi-threaded Compression

Seth 123 Dec 28, 2022
Like pigz, but rust - a cross platform, fast, compression and decompression tool.

?? crabz Like pigz, but rust. A cross platform, fast, compression and decompression tool. Synopsis This is currently a proof of concept CLI tool using

Seth 232 Jan 2, 2023
Michael's Compression Algorithm

mca This repository contains a compression algorithm written by me (Michael Grigoryan). The algorithm is only able to compress and decompress text fil

Michael Grigoryan 1 Dec 19, 2022
A reimplementation of the Zopfli compression tool in Rust.

Zopfli in Rust This is a reimplementation of the Zopfli compression tool in Rust. Carol Nichols started the Rust implementation as an experiment in in

null 11 Dec 26, 2022
lzlib (lzip compression) bindings for Rust

lzip Documentation A streaming compression/decompression library for rust with bindings to lzlib. # Cargo.toml [dependencies] lzip = "0.1" License Lic

Firas Khalil Khana 8 Sep 20, 2022
Obvious Unified Compression Helper is a CLI tool to help you compress and decompress files of several formats

Ouch! ouch stands for Obvious Unified Compression Helper and is a CLI tool to help you compress and decompress files of several formats. Features Usag

null 734 Dec 30, 2022
Basic (and naΓ―ve) LZW and Huffman compression algorithms in Rust.

Naive implementation of the LZW and Huffman compression algorithms. To run, install the Rust toolchain. Cargo may be used to compile the source. Examp

Luiz Felipe Gonçalves 9 May 22, 2023
SIMD Floating point and integer compressed vector library

compressed_vec Floating point and integer compressed vector library, SIMD-enabled for fast processing/iteration over compressed representations. This

Evan Chan 56 Nov 24, 2022
Convenience library for reading and writing compressed files/streams

compress_io Convenience library for reading and writing compressed files/streams The aim of compress_io is to make it simple for an application to sup

Simon Heath 0 Dec 16, 2021
A simple rust library to read and write Zip archives, which is also my pet project for learning Rust

rust-zip A simple rust library to read and write Zip archives, which is also my pet project for learning Rust. At the moment you can list the files in

Kang Seonghoon 2 Jan 5, 2022
A library to create zip files on a non-seekable writer

A library to create zip files on a non-seekable writer

nyantec GmbH 2 Mar 17, 2022
Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Zopfli Compression Algorithm is a compression library programmed in C to perform very good, but slow, deflate or zlib compression.

Google 3.2k Jan 6, 2023
Snappy is a compression/decompression library.

Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger.

Google 5.5k Jan 6, 2023
Fastest lz4 compression library in Node.js, powered by napi-rs and lz4-flex.

Lz4 Fastest lz4 compression library in Node.js, powered by napi-rs and lz4-flex. Install this package yarn add lz4-napi API export function compress:

Antonio Musolino 34 Nov 22, 2022
Snappy bindings for Rust

Snappy [ Originally forked from https://github.com/thestinger/rust-snappy ] Documentation Usage Add this to your Cargo.toml: [dependencies] snappy = "

Jeff Belgum 14 Jan 21, 2022
A lightweight and snappy crate to remove emojis from a string.

A lightweight and snappy crate to remove emojis from a string.

Tejas Ravishankar 8 Jul 19, 2022