API

export interface Parser {
  parse(reader: ReadableStreamByobReader): Promise<string[]>;
  parseBytes(bytes: Uint8Array): Promise<string[]>;
  
  load(): Promise<void>;

  getModuleSource(specifier: string): Promise<string>;
  getModuleSourceMap(specifier: string): Promise<string>;
};

export function build(roots: string[], loader: () => LoadResult): Uint8Array;

Supported parser sinks

a BYOB readable stream

import { Parser } from "./lib/eszip_wasm.generated.js";

const res = await fetch("http://localhost:8000/redirect.eszip2");
const reader = res.body!.getReader({ mode: "byob" });
const parser = new Parser();
const specifiers = await parser.parse(reader);

await parser.load();
for (const specifier of specifiers) {
   await parser.getModuleSource(specifier);
}

Uint8Array

import { Parser } from "./lib/eszip_wasm.generated.js";

const res = await Deno.readFile("src/testdata/redirect.eszip2");
const parser = new Parser();
const specifiers = await parser.parseBytes(res);

await parser.load();
for (const specifier of specifiers) {
   await parser.getModuleSource(specifier);
}

Building eszips

const eszip = await build([
  "https://example.com/mod.ts",
  "https://example.com/a.ts",
], (specifier: string) => {
  if (specifier === "https://example.com/a.ts") {
    return {
      specifier,
      headers: {
        "content-type": "text/typescript",
      },
      content: "export const a = 1;",
    }
  };

  return {
    specifier: "https://example.com/mod.ts",
    headers: {
      "content-type": "application/typescript",
    },
    content: `import "https://example.com/a.ts";`,
  };
});

assert(eszip instanceof Uint8Array);

Similar to #15, I'm also looking for more information about actual usage of this library, especially on how to load modules from generated eszip files. My understanding is that the JS code inside lib/ only provides the primitives to a) turn ES modules into the eszip container format and b) parse generated eszip files. What I found a bit confusing is that b) seems to be limited to reading a container's content without being able to access modules themselves.

As far as I know, the only way to actually use/load modules from a eszip file is to write a custom loader using Rust, which is what deno compile does. Please correct me if I'm wrong.

The main reason why I'm asking: Netlify seems to use eszip to bundle code in different stages, which makes me think that they must be using some custom Deno loader code behind the scenes. Without it, there seems to be no way to make use of the generated stage files other than writing my own loader in Rust, right? 🤔

Implements the streaming decoder (a tokio_util codec) and a tiny encoder for the tests (to be replaced after module graphs integration in #40)

cc @lucacasonato

Don't store source maps inline. It bloats the transpiled file size by 100% or more and it prevents clever optimizations in Deploy.

To support streaming parsing in Deploy, ideally source files are at the start of the archive and source maps at the end.

This PR restores the examples that were removed in #63. To get it working with the recent versions of upstream crates such as deno_graph and tokio, some refactors are applied, namely:

• to adjust the change of parameters in deno_graph
• to replace into_serde with serde_wasm_bindgen::from_value as suggested in wasm_bindgen doc
• to replace tokio with futures in large parts of the code, because tokio has dropped support for the fs feature for wasm target since v1.21.0 (see https://github.com/tokio-rs/tokio/releases/tag/tokio-1.21.0). Switched to futures because it provides futures::io::AllowStdIo which is a good adapter that converts std stuff to AsyncRead, allowing us to use filesystem things for wasm build.

Now we can try examples as the README describes :)

❯ cargo run --example eszip_viewer foo.eszip
    Finished dev [unoptimized + debuginfo] target(s) in 0.11s
     Running `target/debug/examples/eszip_viewer foo.eszip`
Specifier: file:///Users/yusuktan/Repo/github.com/magurotuna/eszip/foo.ts
Kind: JavaScript
---
addEventListener("fetch", (event)=>{
    event.respondWith(new Response("Hello, world!"));
});

---
{"version":3,"sources":["file:///Users/yusuktan/Repo/github.com/magurotuna/eszip/foo.ts"],"sourcesContent":["addEventListener(\"fetch\", (event) => {\n  event.respondWith(new Response(\"Hello, world!\"));\n});\n"],"names":[],"mappings":"AAAA,iBAAiB,SAAS,CAAC,QAAU;IACnC,MAAM,WAAW,CAAC,IAAI,SAAS;AACjC"}
============

Closes #64

The root cause of https://github.com/denoland/deno/issues/13704

mod.ts -> a.js -> https://deno.land/std/testing/asserts.ts

https://deno.land/std/testing/asserts.ts redirects to https://deno.land/[email protected]/testing/asserts.ts but its dependencies do not redirect in the graph: https://deno.land/[email protected]/testing/asserts.ts -> https://deno.land/std/fmt/colors.ts.

This breaks loading the module since std/fmt/colors.ts (unversioned) does not exist on the graph.

It should be https://deno.land/[email protected]/fmt/colors.ts instead.

repro

// mod.ts
import "./a.js";

// a.js
import {
  assertNotEquals,
  assertStrictEquals,
} from 'https://deno.land/std/testing/asserts.ts';

Graph:

~> deno run -A lib/eszip.ts list src/testdata/issue13704.eszip

Loading eszip v2
file:///Users/divy/gh/eszip_pain/mod.ts
file:///Users/divy/gh/eszip_pain/a.js
https://deno.land/[email protected]/testing/asserts.ts
https://deno.land/[email protected]/fmt/colors.ts
https://deno.land/std/testing/asserts.ts
https://deno.land/[email protected]/testing/_diff.ts

A first pass at a small "swiss army knife" for ESZIPs with the following subcommands: -l/ls/list: lists specifiers (without extracting) -x/extract: extracts to a directory of the user's choice -r/run: extracts to a tmp dir, runs deno with an import map then cleans up the tmp dir

Future improvements

• Use deno vendor logic for more robust extraction
• Limit permissions of run
• Print out module graph/tree vs list
• Warn of dangling/missing specifiers
• Rewrite in Rust ? (easier to reuse vendoring, graph tooling, etc...)

Currently the following header is sent when fetching modules:

Accept: */*

Which makes it very difficult for a server to do content negotiation. Either a header of:

Accept: application/javascript, application/typescript, text/javascript, text/typescript, text/jsx, text/tsx, application/json;q=0.9, text/plain;q=0.8, application/octet-stream;q=0.8

Or:

Accept: application/javascript, application/typescript, text/javascript, text/typescript, text/jsx, text/tsx, application/json;q=0.9, text/plain;q=0.8, application/octet-stream;q=0.8, */*;q=0.7

Would allow better negotiation of content for a web server.

Ref: https://github.com/denoland/deno/issues/13287

The code here:

https://github.com/denoland/eszip/blob/c9e9144ec617f2db50c011b42107969382c4f0fb/lib/eszip.ts#L89

...is very basic and looks at the pathname only to figure out how to extract.

This will fail in two scenarios:

1. Where there is no path, like https://fresh.deno.dev
2. A path is a directory of another specifier. For exmaple: something/ then something/other.ts

This commit adds a new error type that is returned from the JS API. It has fields to describe the location (specifier + line + col) of an error, in addition to storing the error message.

Hello!

When I attempt to build an eszip with the following importMap and a very simple module. I receive the error below.

mod.ts

import 'std/fs/mod.ts';
import 'https://deno.land/x/fresh/server.ts'

importMap.json

{
    "imports": {
        "std/": "https://deno.land/std/",
        "https://deno.land/x/": "./vendor/deno.land/x/"
    }
}

error: Error: Blocked by null entry for ""https://deno.land/x/""
      const ret = new Error(getStringFromWasm0(arg0, arg1));
                  ^
    at __wbg_new_651776e932b7e9c7 (file:///workspaces/eszip/lib/eszip_wasm.generated.js:313:19)
    at <anonymous> (wasm://wasm/00a90a36:1:78732)
    at <anonymous> (wasm://wasm/00a90a36:1:1463894)
    at <anonymous> (wasm://wasm/00a90a36:1:1957066)
    at __wbg_adapter_18 (file:///workspaces/eszip/lib/eszip_wasm.generated.js:144:6)
    at real (file:///workspaces/eszip/lib/eszip_wasm.generated.js:128:14)