Write Extism plugins in JavaScript (Experimental)

Related tags

Cryptography js-pdk
Overview

Extism JavaScript PDK

Note: This is very experimental. If you are interested in helping or following development, join the #js-pdk room in our discord channel.

Overview

This PDK uses QuickJS and wizer to run javascript as an Extism Plug-in.

This is essentially a fork of Javy by Shopify. We may wish to collaborate and upstream some things to them. For the time being I built this up from scratch using some of their crates, namely quickjs-wasm-rs.

How it works

This works a little differently than other PDKs. You cannot compile JS to Wasm because it doesn't have an appropriate type system to do this. Something like Assemblyscript is better suited for this. Instead, we have compiled QuickJS to Wasm. The extism-js command we have provided here is a little compiler / wrapper that does a series of things for you:

  1. It loads an "engine" Wasm program containing the QuickJS runtime
  2. It initializes a QuickJS context
  3. It loads your js source code into memory
  4. It parses the js source code for exports and generates 1-to-1 proxy export functions in Wasm
  5. It freezes and emits the machine state as a new Wasm file at this post-initialized point in time

This new Wasm file can be used just like any other Extism plugin.

Install the compiler

We now have released binaries. Check the releases page for the latest.

Note: Windows is not currently a supported platform, only mac and linux

Install Script

curl -O https://raw.githubusercontent.com/extism/js-pdk/main/install.sh
sh install.sh

Then run command with no args to see the help:

extism-js
error: The following required arguments were not provided:
    <input>

USAGE:
    extism-js <input> -o <output>

For more information try --help

Note: If you are using mac, you may need to tell your security system this unsigned binary is fine. If you think this is dangerous, or can't get it to work, see the "compile from source" section below.

Try it on a script file. Name this `script.js:

Note: You must use CJS Module syntax when not using a bundler.

// script.js

const VOWELS = [
    'a', 'e', 'i', 'o', 'u',
]

function count_vowels() {
    let input = Host.inputString()
    let count = 0
    for (let i = 0; i < input.length; i++) {
        if (VOWELS.includes(input[i].toLowerCase())) {
            count += 1
        }
    }
    Host.outputString(JSON.stringify({count}))
    return 0
}

module.exports = {count_vowels}
extism-js script.js -o count_vowels.wasm
extism call count_vowels.wasm count_vowels --input="Hello World!" --wasi
# => {"count":3}                          

Using with a bundler

The compiler cli and core engine can now run bundled code. You will want to use a bundler if you want to want to or include modules from NPM, or write the plugin in Typescript, for example.

There are 2 primary constraints to using a bundler:

  1. Your compiled output must be CJS format, not ESM
  2. You must target es2020 or lower

Using with esbuild

The easiest way to set this up would be to use esbuild. The following is a quickstart guide to setting up a project:

# Make a new JS project
mkdir extism-plugin
cd extism-plugin
npm init -y
npm install esbuild --save-dev
mkdir src
mkdir dist

Add esbuild.js:

const esbuild = require('esbuild');

esbuild
    .build({
        entryPoints: ['src/index.js'],
        outdir: 'dist',
        bundle: true,
        sourcemap: true,
        minify: false, // might want to use true for production build
        format: 'cjs', // needs to be CJS for now
        target: ['es2020'] // don't go over es2020 because quickjs doesn't support it
    })

Add a build script to your package.json:

{
  "name": "extism-plugin",
  // ...
  "scripts": {
    // ...
    "build": "node esbuild.js && extism-js dist/index.js -o dist/plugin.wasm"
  },
  // ...
}

Let's import a module from NPM:

npm install --save fastest-levenshtein

Now make some code in src/index.js. You can use import to load node_modules:

Note: This module uses the ESM Module syntax. The bundler will transform all the code to CJS for us

import {distance, closest} from 'fastest-levenshtein'

// this function is private to the module
function privateFunc() { return 'world' }

// use any export syntax to export a function be callable by the extism host
export function get_closest() {
  let input = Host.inputString()
  let result = closest(input, ['slow', 'faster', 'fastest'])
  Host.outputString(result + ' ' + privateFunc())
  return 0
}
# Run the build script and the plugin will be compiled to dist/plugin.wasm
npm run build
# You can now call from the extism cli or a host SDK
extism call dist/plugin.wasm get_closest --input="fest" --wasi
faster World

Compiling the compiler from source

You need the wasi sdk which can be fetched with the makefile:

make download-wasi-sdk

Then run make to compile the core crate (the engine) and the cli:

make
./target/release/extism-js script.js -o out.wasm
extism call out.wasm count_vowels --wasi --input="Hello World Test!"
# => "{\"count\":4}"

Why not use Javy?

Javy, and many other high level language Wasm tools, assume use of the command pattern. This is when the Wasm module only exports a main function and communicates with the host through stdin and stdout. With Extism, we have more of a shared library interface. The module exposes multiple entry points through exported functions. Furthermore, Javy has many Javy and Shopify specific things it's doing that we will not need. However, the core idea is the same, and we can possibly contribute by adding support to Javy for non-command-pattern modules. Then separating the Extism PDK specific stuff into another repo.

What needs to be done?

Implemented so far:

  • Host.inputBytes
  • Host.inputString
  • Host.outputBytes
  • Host.outputString
  • Var.get
  • Var.set
  • console.log
  • console.error
  • throw Error

The above are implemented but need some more validation and resilience built into them. debating whether I should implement the bulk of the code in js or rust. Working on implementing the other pdk methods.

I've got the exports to work, but it's a fragile and complicated solution. Will write it up soon, and maybe it can be replaced with something simpler.

You might also like...
Simple library to host lv2 plugins. Is not meant to support any kind of GUI.

lv2-host-minimal Simple library to host lv2 plugins. Is not meant to support any kind of GUI. Host fx plugins (audio in, audio out) Set parameters Hos

🐱‍👤 Cross-language static library for accessing the Lua state in Garry's Mod server plugins

gmserverplugin This is a utility library for making Server Plugins that access the Lua state in Garry's Mod. Currently, accessing the Lua state from a

VST 2.4 API implementation in rust. Create plugins or hosts.

rust-vst2 A library to help facilitate creating VST plugins in rust. This library is a work in progress and as such does not yet implement all opcodes

Plugins and helpful methods for using sepax2d with Bevy for 2d overlap detection and collision resolution.

bevy_sepax2d Plugins and helpful methods for using sepax2d with Bevy for 2d overlap detection and collision resolution. Compatible Versions bevy bevy_

A Rust framework to develop and use plugins within your project, without worrying about the low-level details.

VPlugin: A plugin framework for Rust. Website | Issues | Documentation VPlugin is a Rust framework to develop and use plugins on applications and libr

Create WASM plugins easily in Rust.

Scotch Library for creating WASM plugins with Rust. Scotch allows you to pass complex types to/from functions in WASM plugins. It achieves that by enc

A secure JavaScript and TypeScript runtime
A secure JavaScript and TypeScript runtime

Deno Deno is a simple, modern and secure runtime for JavaScript and TypeScript that uses V8 and is built in Rust. Features Secure by default. No file,

Facilitating high-level interactions between Wasm modules and JavaScript

wasm-bindgen Facilitating high-level interactions between Wasm modules and JavaScript. Guide | API Docs | Contributing | Chat Built with 🦀 🕸 by The

Generate PDF files with JavaScript and WASM (WebAssembly)

WASM-PDF Generates PDF files directly in the browser with JavaScript and WASM (WebAssembly). Idea here is to push all the work involved in creating a

JsonPath engine written in Rust. Webassembly and Javascript support too

jsonpath_lib Rust 버전 JsonPath 구현으로 Webassembly와 Javascript에서도 유사한 API 인터페이스를 제공 한다. It is JsonPath JsonPath engine written in Rust. it provide a simil

The JavaScript runtime that aims for productivity and ease

Byte Byte is a easy and productive runtime for Javascript . It makes making complex programs simple and easy-to-scale with its large and fast Rust API

Crabzilla provides a simple interface for running JavaScript modules alongside Rust code.

Crabzilla Crabzilla provides a simple interface for running JavaScript modules alongside Rust code. Example use crabzilla::*; use std::io::stdin; #[i

A JavaScript Runtime built with Mozilla's SpiderMonkey Engine and Rust

Spiderfire Spiderfire is a javascript runtime built with Mozilla's SpiderMonkey engine and Rust. Spiderfire aims to disrupt the server-side javascript

A utility that can download JavaScript and TypeScript module graphs and store them locally in a special zip file.

eszip A utility that can download JavaScript and TypeScript module graphs and store them locally in a special zip file. To create a new archive: esz

An n-tuple pendulum simulator in Rust + WebAssembly (and JavaScript)

An n-tuple pendulum simulator in Rust + WebAssembly (and JavaScript) Remaking this n-tuple pendulum simulator moving the math to Rust 🦀 and WebAssemb

Guarding 是一个用于 Java、JavaScript、Rust、Golang 等语言的架构守护工具。借助于易于理解的 DSL,来编写守护规则。Guarding is a guardians for code, architecture, layered.

Guarding Guarding is a guardians for code, architecture, layered. Using git hooks and DSL for design guard rules. Usage install cargo install guarding

A node API for the dprint TypeScript and JavaScript code formatter

dprint-node A node API for the dprint TypeScript and JavaScript code formatter. It's written in Rust for blazing fast speed. Usage Pass a file path an

SixtyFPS is a toolkit to efficiently develop fluid graphical user interfaces for any display: embedded devices and desktop applications. We support multiple programming languages, such as Rust, C++ or JavaScript.
SixtyFPS is a toolkit to efficiently develop fluid graphical user interfaces for any display: embedded devices and desktop applications. We support multiple programming languages, such as Rust, C++ or JavaScript.

SixtyFPS is a toolkit to efficiently develop fluid graphical user interfaces for any display: embedded devices and desktop applications. We support multiple programming languages, such as Rust, C++ or JavaScript.

Build terminal dashboards using ascii/ansi art and javascript
Build terminal dashboards using ascii/ansi art and javascript

blessed-contrib Build dashboards (or any other application) using ascii/ansi art and javascript. Friendly to terminals, ssh and developers.

Comments
  • unknown os and unknown arch

    unknown os and unknown arch

    👋 Hello. I'm using Multipass for my development workspace (on a Mac M1), then

    • the value of $OSTYPE is linux-gnu but I need to run the installer like this: bash install.sh (instead of sh install.sh otherwise I will get this message: unknown os
    • then I get: unknown arch, the workaround was to change arm64*) ARCH="aarch64" ;; by arm64*|aarch64*) ARCH="aarch64" ;;
    opened by k33g 1
  • feat: Support bundled JS applications

    feat: Support bundled JS applications

    Instead of parsing and statically looking for module exports in the compile step, this evaluates the code to find the exports. Thus we can support bundled JS code.

    How to use

    In order to get this to work, you just need a working bundler and you need to output in CJS format, not ESM format. I've tested in esbuild but webpack and other should work. You should be able to write your plugin in typescript too, i haven't tested though!

    Here is an example of a js project from scratch with esbuild:

    # Make a new JS project
    mkdir extism-plugin
    cd extism-plugin
    npm init -y
    npm install --save-dev
    

    Add esbuild.js:

    const esbuild = require('esbuild');
    
    esbuild
        .build({
            entryPoints: ['src/index.js'],
            outdir: 'dist',
            bundle: true,
            sourcemap: true,
            minify: false,
            format: 'cjs', // needs to be CJS for now
            target: ['es2020'] // don't go over es2020 because quickjs doesn't support it
        })
    

    make some directories

    mkdir src
    mkdir dist
    

    Add a build script to your package.json:

    {
      "name": "extism-plugin",
      "version": "1.0.0",
      "description": "",
      "main": "index.js",
      "scripts": {
        "test": "echo \"Error: no test specified\" && exit 1",
        "build": "node esbuild.js && extism-js dist/index.js -o dist/plugin.wasm"
      },
      "keywords": [],
      "author": "",
      "license": "ISC",
      "devDependencies": {
        "esbuild": "^0.17.2"
      }
    }
    

    Let's import a module from NPM:

    npm install --save fastest-levenshtein
    

    Now make some code in src/index.js. You can use import to load node_modules:

    import {distance, closest} from 'fastest-levenshtein'
    
    // this function is private to the module
    function privateFunc() { return 'world' }
    
    // use any export syntax to export a function be callable by the extism host
    export function get_closest() {
      let input = Host.inputString()
      let result = closest(input, ['slow', 'faster', 'fastest'])
      Host.outputString(result + ' ' + privateFunc())
      return 0
    }
    
    # Run the build script and the plugin will be compiled to dist/plugin.wasm
    npm run build
    # You can now call from the extism cli or a host SDK
    extism call dist/plugin.wasm get_closest --input="fest" --wasi
    faster World                     
    

    Next Steps

    I need to go back and possibly refactor the core after this. Right now it seems to still work but I'm concerned the global functions will get their names mangled. So instead core should look inside module.exports and find the function reference there rather than expecting the global name to be preserved.

    opened by bhelx 0
Releases(v0.3.1)
Owner
Extism
Make all software programmable. Extend from within.
Extism
A Secure Capability-Based Runtime for JavaScript Based on Deno

Secure Runtime secure-runtime, as the name implies, is a secure runtime for JavaScript, designed for the multi-tenant serverless environment. It is an

Gigamono 7 Oct 7, 2022
A modern runtime for javascript.

Just NOTICE: README LINKS AND SITE ARE WIP. LINKS MAY NOT WORK Just is a simple, and modern runtime for JavaScript that uses V8 and is built in Rust.

Exact Labs 6 Dec 15, 2022
A re-write of polkadot staking miner using subxt to avoid hard dependency to each runtime version

Staking Miner v2 WARNING this library is under active development DO NOT USE IN PRODUCTION. The library is a re-write of polkadot staking miner using

Parity Technologies 19 Dec 28, 2022
VSCode extension to quickly write and customize well tested Solana snippets.

Solana Snippets The Solana Snippets VSCode Extension allows you to quickly insert Solana snippets into your code. This snippets are well tested in a r

patriciobcs 7 Dec 15, 2022
EXPERIMENTAL: Bitcoin Core Prometheus exporter based on User-Space, Statically Defined Tracing and eBPF.

bitcoind-observer An experimental Prometheus metric exporter for Bitcoin Core based on Userspace, Statically Defined Tracing and eBPF. This demo is ba

0xB10C 24 Nov 8, 2022
Experimental binary transparency for pacman with sigstore and rekor

pacman-bintrans This is an experimental implementation of binary transparency for pacman, the Arch Linux package manager. This project was originally

null 80 Dec 23, 2022
An experimental rust zksnarks compiler with embeeded bellman-bn128 prover

Za! An experimental port of the circom zk-SNARK compiler in Rust with embedded bellman-bn128 prover. I created it as a PoC port of the existing JavaSc

adria0.eth 39 Aug 26, 2022
Rust implementation of the Matter protocol. Status: Experimental

matter-rs: The Rust Implementation of Matter Build Building the library: $ cd matter $ cargo build Building the example: $ cd matter $ RUST_LOG="matt

Connectivity Standards Alliance 12 Jan 5, 2023
A low-ish level tool for easily writing and hosting WASM based plugins.

A low-ish level tool for easily writing and hosting WASM based plugins. The goal of wasm_plugin is to make communicating across the host-plugin bounda

Alec Deason 62 Sep 20, 2022
A graphical user interface toolkit for audio plugins.

HexoTK - A graphic user interface toolkit for audio plugins State of Development Super early! Building cargo run --example demo TODO / Features Every

Weird Constructor 14 Oct 20, 2022