A utility that can download JavaScript and TypeScript module graphs and store them locally in a special zip file.

Related tags

Compression eszip
Overview

eszip

A utility that can download JavaScript and TypeScript module graphs and store them locally in a special zip file.

To create a new archive:

> eszip get https://raw.githubusercontent.com/satyarohith/kament/main/mod.ts
Download https://raw.githubusercontent.com/satyarohith/kament/main/mod.ts
...
Wrote es.zip

To print the list of modules in an eszip file:

> eszip list es.zip
https://denopkg.com/chiefbiiko/sha512/mod.ts
https://deno.land/x/[email protected]/algorithm.ts
https://deno.land/x/[email protected]/src/rsa/rsa_key.ts
https://deno.land/x/[email protected]/src/rsa/rsa_js.ts
...

To read a module from the archive:

> eszip read es.zip https://denopkg.com/chiefbiiko/sha512/mod.ts
import { encode, decode } from "./deps.ts";

/** Byte length of a SHA512 hash. */
export const BYTES: number = 64;

/** A class representation of the SHA2-512 algorithm. */
export class SHA512 {
  readonly hashSize: number = BYTES;
...
Comments
  • feat: eszip v2 wasm library

    feat: eszip v2 wasm library

    API

    export interface Parser {
      parse(reader: ReadableStreamByobReader): Promise<string[]>;
      parseBytes(bytes: Uint8Array): Promise<string[]>;
      
      load(): Promise<void>;
    
      getModuleSource(specifier: string): Promise<string>;
      getModuleSourceMap(specifier: string): Promise<string>;
    };
    
    export function build(roots: string[], loader: () => LoadResult): Uint8Array;
    

    Supported parser sinks

    a BYOB readable stream

    import { Parser } from "./lib/eszip_wasm.generated.js";
    
    const res = await fetch("http://localhost:8000/redirect.eszip2");
    const reader = res.body!.getReader({ mode: "byob" });
    const parser = new Parser();
    const specifiers = await parser.parse(reader);
    
    await parser.load();
    for (const specifier of specifiers) {
       await parser.getModuleSource(specifier);
    }
    

    Uint8Array

    import { Parser } from "./lib/eszip_wasm.generated.js";
    
    const res = await Deno.readFile("src/testdata/redirect.eszip2");
    const parser = new Parser();
    const specifiers = await parser.parseBytes(res);
    
    await parser.load();
    for (const specifier of specifiers) {
       await parser.getModuleSource(specifier);
    }
    

    Building eszips

    const eszip = await build([
      "https://example.com/mod.ts",
      "https://example.com/a.ts",
    ], (specifier: string) => {
      if (specifier === "https://example.com/a.ts") {
        return {
          specifier,
          headers: {
            "content-type": "text/typescript",
          },
          content: "export const a = 1;",
        }
      };
    
      return {
        specifier: "https://example.com/mod.ts",
        headers: {
          "content-type": "application/typescript",
        },
        content: `import "https://example.com/a.ts";`,
      };
    });
    
    assert(eszip instanceof Uint8Array);
    
    opened by littledivy 4
  • Q: How to load modules from eszip files?

    Q: How to load modules from eszip files?

    Similar to #15, I'm also looking for more information about actual usage of this library, especially on how to load modules from generated eszip files. My understanding is that the JS code inside lib/ only provides the primitives to a) turn ES modules into the eszip container format and b) parse generated eszip files. What I found a bit confusing is that b) seems to be limited to reading a container's content without being able to access modules themselves.

    As far as I know, the only way to actually use/load modules from a eszip file is to write a custom loader using Rust, which is what deno compile does. Please correct me if I'm wrong.

    The main reason why I'm asking: Netlify seems to use eszip to bundle code in different stages, which makes me think that they must be using some custom Deno loader code behind the scenes. Without it, there seems to be no way to make use of the generated stage files other than writing my own loader in Rust, right? 🤔

    opened by mlafeldt 3
  • feat: streaming parser for eszip v2

    feat: streaming parser for eszip v2

    Implements the streaming decoder (a tokio_util codec) and a tiny encoder for the tests (to be replaced after module graphs integration in #40)

    cc @lucacasonato

    opened by littledivy 2
  • feat: store source maps out of band

    feat: store source maps out of band

    Don't store source maps inline. It bloats the transpiled file size by 100% or more and it prevents clever optimizations in Deploy.


    To support streaming parsing in Deploy, ideally source files are at the start of the archive and source maps at the end.

    opened by bnoordhuis 2
  • feat(examples): restore examples

    feat(examples): restore examples

    This PR restores the examples that were removed in #63. To get it working with the recent versions of upstream crates such as deno_graph and tokio, some refactors are applied, namely:

    • to adjust the change of parameters in deno_graph
    • to replace into_serde with serde_wasm_bindgen::from_value as suggested in wasm_bindgen doc
    • to replace tokio with futures in large parts of the code, because tokio has dropped support for the fs feature for wasm target since v1.21.0 (see https://github.com/tokio-rs/tokio/releases/tag/tokio-1.21.0). Switched to futures because it provides futures::io::AllowStdIo which is a good adapter that converts std stuff to AsyncRead, allowing us to use filesystem things for wasm build.

    Now we can try examples as the README describes :)

    ❯ cargo run --example eszip_viewer foo.eszip
        Finished dev [unoptimized + debuginfo] target(s) in 0.11s
         Running `target/debug/examples/eszip_viewer foo.eszip`
    Specifier: file:///Users/yusuktan/Repo/github.com/magurotuna/eszip/foo.ts
    Kind: JavaScript
    ---
    addEventListener("fetch", (event)=>{
        event.respondWith(new Response("Hello, world!"));
    });
    
    ---
    {"version":3,"sources":["file:///Users/yusuktan/Repo/github.com/magurotuna/eszip/foo.ts"],"sourcesContent":["addEventListener(\"fetch\", (event) => {\n  event.respondWith(new Response(\"Hello, world!\"));\n});\n"],"names":[],"mappings":"AAAA,iBAAiB,SAAS,CAAC,QAAU;IACnC,MAAM,WAAW,CAAC,IAAI,SAAS;AACjC"}
    ============
    

    Closes #64

    opened by magurotuna 1
  • bug: redirects leaves dangling/invalid specifiers

    bug: redirects leaves dangling/invalid specifiers

    The root cause of https://github.com/denoland/deno/issues/13704

    mod.ts -> a.js -> https://deno.land/std/testing/asserts.ts

    https://deno.land/std/testing/asserts.ts redirects to https://deno.land/[email protected]/testing/asserts.ts but its dependencies do not redirect in the graph: https://deno.land/[email protected]/testing/asserts.ts -> https://deno.land/std/fmt/colors.ts.

    This breaks loading the module since std/fmt/colors.ts (unversioned) does not exist on the graph.

    It should be https://deno.land/[email protected]/fmt/colors.ts instead.

    repro

    // mod.ts
    import "./a.js";
    
    // a.js
    import {
      assertNotEquals,
      assertStrictEquals,
    } from 'https://deno.land/std/testing/asserts.ts';
    

    Graph:

    ~> deno run -A lib/eszip.ts list src/testdata/issue13704.eszip
    
    Loading eszip v2
    file:///Users/divy/gh/eszip_pain/mod.ts
    file:///Users/divy/gh/eszip_pain/a.js
    https://deno.land/[email protected]/testing/asserts.ts
    https://deno.land/[email protected]/fmt/colors.ts
    https://deno.land/std/testing/asserts.ts
    https://deno.land/[email protected]/testing/_diff.ts
    
    bug 
    opened by littledivy 1
  • feat: first iteration of an eszip CLI

    feat: first iteration of an eszip CLI

    A first pass at a small "swiss army knife" for ESZIPs with the following subcommands: -l/ls/list: lists specifiers (without extracting) -x/extract: extracts to a directory of the user's choice -r/run: extracts to a tmp dir, runs deno with an import map then cleans up the tmp dir

    Future improvements

    • Use deno vendor logic for more robust extraction
    • Limit permissions of run
    • Print out module graph/tree vs list
    • Warn of dangling/missing specifiers
    • Rewrite in Rust ? (easier to reuse vendoring, graph tooling, etc...)
    opened by AaronO 1
  • Consider using a better `Accept` header when fetching modules

    Consider using a better `Accept` header when fetching modules

    Currently the following header is sent when fetching modules:

    Accept: */*
    

    Which makes it very difficult for a server to do content negotiation. Either a header of:

    Accept: application/javascript, application/typescript, text/javascript, text/typescript, text/jsx, text/tsx, application/json;q=0.9, text/plain;q=0.8, application/octet-stream;q=0.8
    

    Or:

    Accept: application/javascript, application/typescript, text/javascript, text/typescript, text/jsx, text/tsx, application/json;q=0.9, text/plain;q=0.8, application/octet-stream;q=0.8, */*;q=0.7
    

    Would allow better negotiation of content for a web server.

    Ref: https://github.com/denoland/deno/issues/13287

    opened by kitsonk 1
  • `eszip extract` fails when a specifier in the eszip has no path or is a directory of another specifier

    `eszip extract` fails when a specifier in the eszip has no path or is a directory of another specifier

    The code here:

    https://github.com/denoland/eszip/blob/c9e9144ec617f2db50c011b42107969382c4f0fb/lib/eszip.ts#L89

    ...is very basic and looks at the pathname only to figure out how to extract.

    This will fail in two scenarios:

    1. Where there is no path, like https://fresh.deno.dev
    2. A path is a directory of another specifier. For exmaple: something/ then something/other.ts
    bug 
    opened by dsherret 0
  • feat(js): add `EszipError` type

    feat(js): add `EszipError` type

    This commit adds a new error type that is returned from the JS API. It has fields to describe the location (specifier + line + col) of an error, in addition to storing the error message.

    opened by lucacasonato 2
  • eszip build fails with: Error: Blocked by null entry for

    eszip build fails with: Error: Blocked by null entry for ""https://deno.land/x/""

    Hello!

    When I attempt to build an eszip with the following importMap and a very simple module. I receive the error below.

    mod.ts

    import 'std/fs/mod.ts';
    import 'https://deno.land/x/fresh/server.ts'
    

    importMap.json

    {
        "imports": {
            "std/": "https://deno.land/std/",
            "https://deno.land/x/": "./vendor/deno.land/x/"
        }
    }
    
    error: Error: Blocked by null entry for ""https://deno.land/x/""
          const ret = new Error(getStringFromWasm0(arg0, arg1));
                      ^
        at __wbg_new_651776e932b7e9c7 (file:///workspaces/eszip/lib/eszip_wasm.generated.js:313:19)
        at <anonymous> (wasm://wasm/00a90a36:1:78732)
        at <anonymous> (wasm://wasm/00a90a36:1:1463894)
        at <anonymous> (wasm://wasm/00a90a36:1:1957066)
        at __wbg_adapter_18 (file:///workspaces/eszip/lib/eszip_wasm.generated.js:144:6)
        at real (file:///workspaces/eszip/lib/eszip_wasm.generated.js:128:14)
    
    opened by deckchairlabs 0
Releases(v0.31.0)
  • v0.30.0(Nov 10, 2022)

  • v0.29.0(Oct 24, 2022)

  • v0.28.0(Sep 8, 2022)

  • v0.26.0(Aug 29, 2022)

  • v0.25.0(Aug 24, 2022)

    What's Changed

    • 0.25.0 by @lucacasonato in https://github.com/denoland/eszip/pull/86

    Full Changelog: https://github.com/denoland/eszip/compare/v0.24.0...v0.25.0

    Source code(tar.gz)
    Source code(zip)
  • v0.24.0(Aug 24, 2022)

    What's Changed

    • 0.24.0 by @lucacasonato in https://github.com/denoland/eszip/pull/85

    Full Changelog: https://github.com/denoland/eszip/compare/0.24.0...v0.24.0

    Source code(tar.gz)
    Source code(zip)
  • v0.23.0(Aug 9, 2022)

    What's Changed

    • chore: update deno_graph by @kitsonk in https://github.com/denoland/eszip/pull/82

    Full Changelog: https://github.com/denoland/eszip/compare/v0.22.0...v0.23.0

    Source code(tar.gz)
    Source code(zip)
  • v0.22.0(Jul 11, 2022)

    What's Changed

    • upgrade swc https://github.com/denoland/eszip/pull/80

    Full Changelog: https://github.com/denoland/eszip/compare/v0.21.0...v0.22.0

    Source code(tar.gz)
    Source code(zip)
  • v0.21.0(Jun 21, 2022)

    What's Changed

    • feat: upgrade deno ast 0.16 https://github.com/denoland/eszip/pull/78

    Full Changelog: https://github.com/denoland/eszip/compare/v0.20.0...v0.21.0

    Source code(tar.gz)
    Source code(zip)
  • v0.19.0(Apr 8, 2022)

    What's Changed

    • chmod +x lib/eszip.ts https://github.com/denoland/eszip/pull/68
    • tool: Add eszip build subcommand https://github.com/denoland/eszip/pull/69
    • examples: chainloading https://github.com/denoland/eszip/pull/67
    • upgrade deno_ast to 0.14 https://github.com/denoland/eszip/pull/71

    Full Changelog: https://github.com/denoland/eszip/compare/v0.18.0...v0.19.0

    Source code(tar.gz)
    Source code(zip)
  • v0.18.0(Mar 21, 2022)

    What's Changed

    • feat: first iteration of an eszip CLI https://github.com/denoland/eszip/pull/62
    • feat: upgrade swc_ecmascript to 0.137.0 https://github.com/denoland/eszip/pull/63

    Full Changelog: https://github.com/denoland/eszip/compare/v0.17.0...v0.18.0

    Source code(tar.gz)
    Source code(zip)
  • v0.17.0(Feb 24, 2022)

  • v0.16.0(Feb 14, 2022)

    What's Changed

    • feat: support external specifiers by @littledivy in https://github.com/denoland/eszip/pull/54

    Full Changelog: https://github.com/denoland/eszip/compare/v0.15.3...v0.16.0

    Source code(tar.gz)
    Source code(zip)
  • v0.15.3(Feb 14, 2022)

    What's Changed

    • fix: dynamic imports error the graph by @lucacasonato in https://github.com/denoland/eszip/pull/55
    • build: wasm-opt -Oz release builds by @littledivy in https://github.com/denoland/eszip/pull/50
    • chore: limit deno_ast features by @bartlomieju in https://github.com/denoland/eszip/pull/53

    Full Changelog: https://github.com/denoland/eszip/compare/0.15.2...v0.15.3

    Source code(tar.gz)
    Source code(zip)
  • v0.15.1(Jan 28, 2022)

    What's Changed

    • fix: prefer transpiled entry in eszip v1 by @lucacasonato in https://github.com/denoland/eszip/pull/47

    Full Changelog: https://github.com/denoland/eszip/compare/v0.15.0...v0.15.1

    Source code(tar.gz)
    Source code(zip)
  • v0.15.0(Jan 27, 2022)

    What's Changed

    • feat: eszip v2 by @lucacasonato and @littledivy (#40)

    Full Changelog: https://github.com/denoland/eszip/compare/v0.14.1...v0.15.0

    Source code(tar.gz)
    Source code(zip)
  • v0.14.1(Nov 3, 2021)

    What's Changed

    • fix: enable use_define_for_class_fields by @dsherret in https://github.com/denoland/eszip/pull/38
    • 0.14.1 by @lucacasonato in https://github.com/denoland/eszip/pull/39

    Full Changelog: https://github.com/denoland/eszip/compare/v0.14.0...v0.14.1

    Source code(tar.gz)
    Source code(zip)
  • v0.14.0(Nov 1, 2021)

    What's Changed

    • chore: update README by @bnoordhuis in https://github.com/denoland/eszip/pull/32
    • chore: fix clippy warnings by @bnoordhuis in https://github.com/denoland/eszip/pull/33
    • chore: add syntax error regression tests by @bnoordhuis in https://github.com/denoland/eszip/pull/35
    • feat: upgrade deno_ast to 0.5 by @dsherret in https://github.com/denoland/eszip/pull/36
    • 0.14.0 by @lucacasonato in https://github.com/denoland/eszip/pull/37

    New Contributors

    • @bnoordhuis made their first contribution in https://github.com/denoland/eszip/pull/32
    • @dsherret made their first contribution in https://github.com/denoland/eszip/pull/36

    Full Changelog: https://github.com/denoland/eszip/compare/v0.13.0...v0.14.0

    Source code(tar.gz)
    Source code(zip)
  • v0.13.0(Sep 27, 2021)

Zip implementation in Rust

zip-rs Documentation Info A zip library for rust which supports reading and writing of simple ZIP files. Supported compression formats: stored (i.e. n

null 549 Jan 4, 2023
A library to create zip files on a non-seekable writer

A library to create zip files on a non-seekable writer

nyantec GmbH 2 Mar 17, 2022
An extremely fast alternative to zip which is written in rust.

Zap Compress and/or encrypt folders fast. Like, really fast. or as some say, blazingly fast. Installation To install Zap, run the following command fr

null 39 Dec 23, 2022
An extremely fast alternative to zip which is written in rust.

Zap Compress and/or encrypt folders fast. Like, really fast. or as some say, blazingly fast. Installation To install Zap, run the following command fr

null 37 Nov 9, 2022
Tar file reading/writing for Rust

tar-rs Documentation A tar archive reading/writing library for Rust. # Cargo.toml [dependencies] tar = "0.4" Reading an archive extern crate tar; use

Alex Crichton 490 Dec 30, 2022
(WIP) Taking the pain away from file (de)compression

Ouch! ouch loosely stands for Obvious Unified Compression files Helper and aims to be an easy and intuitive way of compressing and decompressing files

Vinícius Miguel 734 Dec 30, 2022
📦 Unpack deep archive files recursively over a file tree or a folder

deep-unpack Unpack deep archive files recursively over a file tree or a folder. Usage [dependencies] deep-unpack = { version = "0.1.2" } Usage fn main

null 3 Dec 4, 2022
Basic (and naïve) LZW and Huffman compression algorithms in Rust.

Naive implementation of the LZW and Huffman compression algorithms. To run, install the Rust toolchain. Cargo may be used to compile the source. Examp

Luiz Felipe Gonçalves 9 May 22, 2023
A Brotli implementation in pure and safe Rust

Brotli-rs - Brotli decompression in pure, safe Rust Documentation Compression provides a <Read>-struct to wrap a Brotli-compressed stream. A consumer

Thomas Pickert 59 Oct 7, 2022
Brotli compressor and decompressor written in rust that optionally avoids the stdlib

rust-brotli What's new in 3.2 into_inner conversions for both Reader and Writer classes What's new in 3.0 A fully compatible FFI for drop-in compatibi

Dropbox 659 Dec 29, 2022
SIMD Floating point and integer compressed vector library

compressed_vec Floating point and integer compressed vector library, SIMD-enabled for fast processing/iteration over compressed representations. This

Evan Chan 56 Nov 24, 2022
DEFLATE, gzip, and zlib bindings for Rust

flate2 A streaming compression/decompression library DEFLATE-based streams in Rust. This crate by default uses the miniz_oxide crate, a port of miniz.

The Rust Programming Language 619 Jan 8, 2023
Lossless compressor and decompressor for numerical data using quantiles

This rust library compresses and decompresses sequences of numerical data very well. It currently supports the following data types: i32, i64, u32, u64, f32, f64. Smaller data types like i16 can be efficiently compressed by casting to i32. Timestamp support may come soon in the future.

Martin 163 Dec 14, 2022
Like pigz, but rust - a cross platform, fast, compression and decompression tool.

?? crabz Like pigz, but rust. A cross platform, fast, compression and decompression tool. Synopsis This is currently a proof of concept CLI tool using

Seth 232 Jan 2, 2023
Convenience library for reading and writing compressed files/streams

compress_io Convenience library for reading and writing compressed files/streams The aim of compress_io is to make it simple for an application to sup

Simon Heath 0 Dec 16, 2021
A Rust application that compress files and folders

Quick Storer This is a Rust application that compress files and folders. Usage Download or build the binary and place it on your desktop, or any other

AL68 & co. 1 Feb 2, 2022
Obvious Unified Compression Helper is a CLI tool to help you compress and decompress files of several formats

Ouch! ouch stands for Obvious Unified Compression Helper and is a CLI tool to help you compress and decompress files of several formats. Features Usag

null 734 Dec 30, 2022
Resolve JavaScript/TypeScript module with Rust

ES Resolve JavaScript/TypeScript module resolution in Rust Installation cargo add es_resolve Get Started use std::path::{Path, PathBuf}; use es_resolv

wang chenyu 2 Oct 12, 2022
Download a file using multiple threads in parallel for faster download speeds.

multidl Download a file using multiple threads in parallel for faster download speeds. Uses 0 external dependencies. Usage Usage: multidl [--help] ADD

Divyanshu Agrawal 2 Sep 12, 2021
Download pdbs from symbol servers and cache locally, parse symbol paths from env vars

symsrv This crate lets you download and cache pdb files from symbol servers, according to the rules from the _NT_SYMBOL_PATH environment variable. It

Markus Stange 6 Sep 15, 2022