@bytedance/tar-wasm

A faster* experimental wasm-based tar implementation for browsers.

*50-160x faster, see benchmarks below.

Usage

Install

npm install @bytedance/tar-wasm

Example

import { TarBuilder } from "@bytedance/tar-wasm";

// Create a new tar builder
const tarBuilder = new TarBuilder();

// Optionally enable gzip compression
// tarBuilder.set_gzip(true);
// However, I recommend using the Compression Streams API instead when possible:
// https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API

for (const file of files) {
  // Add files to the tar
  tarBuilder.add_file(file.name, file.content);
}

// Finish building the tar and get the result
const tar: UInt8Array = tarBuilder.finish();

// Optionally convert the tar to a Blob or whatever
const tarBlob = new Blob([tar], { type: "application/x-tar" });

Performance Notes

This library is optimized for performance via WebAssembly. However, WebAssembly runs in the main thread by default and can be computationally expensive by nature depending on the size of the tar.

We recommend using Web Worker or Worker Threads to run the tar building in a separate thread. You may want to refer to Greenlet for a lightweight implementation.

Benchmarks

According to my benchmark, it can be 50-160x faster than existing JavaScript implementations on NPM, depending on the browser and the file size.

Results on my machine (Intel(R) Xeon(R) Platinum 8269CY CPU @ 2.50GHz):

Running benchmark...
Generated 1001 random files to test in 191.13816452026367ms
@bytedance/tar-wasm: 24.62919044494629ms (baseline)
tar-js: 3943.49857711792ms (160.11482740095886x slower than @bytedance/tar-wasm)
tarts: 1251.595199584961ms (50.817553357373065x slower than @bytedance/tar-wasm)

See ./benchmark folder for more details.

TODO

Include a Promise API working from a separate worker thread.
Add support for reading and writing tar files.
Support of Node.js environment
Tests
Benchmarks

Comments

Support Node.js

Now the published package is built with --target bundler, but doesn't work with Node.js. The Node.js benchmark works with a separately built package --target nodejs, but it is not published. Need to find a way to support both targets at the same time if possible.
enhancement

opened by eligao 0

Releases(v0.1.7)

v0.1.7(Oct 27, 2022)
What's Changed

Correct a typo in benchmark/README.md by @felixonmars in https://github.com/bytedance/tar-wasm/pull/4

feat: isomorphic package for both node.js and web bundler by @eligao in https://github.com/bytedance/tar-wasm/pull/5

Added an example next.js benchmark

New Contributors

@felixonmars made their first contribution in https://github.com/bytedance/tar-wasm/pull/4

@eligao made their first contribution in https://github.com/bytedance/tar-wasm/pull/5

Full Changelog: https://github.com/bytedance/tar-wasm/compare/v0.1.6...v0.1.7
Source code(tar.gz)
Source code(zip)
v0.1.6(Aug 18, 2022)

Source code(tar.gz)
Source code(zip)