Extend anything with WebAssembly.

Related tags

Command-line ruby python c go rust haskell node elixir sdk cpp runtime ocaml webassembly wasm plugin-system hacktoberfest
Overview

Welcome!

Please note: this project still under active development. It's usable, but expect some rough edges while work is underway. If you're interested in working on or building with Extism, please join our Discord and let us know - we are happy to help get you started.

Discord

Extism

The universal plug-in system. Run WebAssembly extensions inside your app. Use idiomatic Host SDKs for Go, Ruby, Python, Node, Rust, C, C++, OCaml, Haskell, PHP & more (others coming soon).

Plug-in development kits (PDK) for plug-in authors supported in Rust, AssemblyScript, Go, C/C++.

Add a flexible, secure, and bLaZiNg FaSt plug-in system to your project. Server, desktop, mobile, web, database -- you name it. Enable users to write and execute safe extensions to your software in 3 easy steps:

1. Import

Import an Extism Host SDK into your code as a library dependency.

2. Integrate

Identify the place(s) in your code where some arbitrary logic should run (the plug-in!), returning your code some results.

3. Execute

Load WebAssembly modules at any time in your app's lifetime and Extism will execute them in a secure sandbox, fully isolated from your program's memory.

Usage

Head to the project website for more information and docs. Also, consider reading an overview of Extism and its goals & approach.

Contribution

Thank you for considering a contribution to Extism, we are happy to help you make a PR or find something to work on!

The easiest way to start would be to join the Discord or open an issue on the extism/proposals issue tracker, which can eventually become an Extism Improvement Proposal (EIP).

Who's behind this?

Extism is an open-source product from the team at:

Reach out and tell us what you're building! We'd love to help.

Comments
  • feat: Implement .NET Host SDK

    feat: Implement .NET Host SDK

    I am working on a .NET host for extism. My plan is to do the following:

    • [x] Implement a proof of concept to make sure things are possible
    • [x] Write docs for the C# API so that the users get a nice IDE experience
    • [x] Create a github action to publish the NuGet packages
      • [x] Edit ci.yml to include .NET Sdk
      • [x] Create release-dotnet.yml to release Extism.Sdk nuget package
      • [x] Maybe Create release-dotnet-native.yml to release Extism.runtime.win nuget package
    • [x] Test on Linux (Help needed)
    • [x] Test on Mac (Help needed)
    • [x] Expose all of the Extism functions
    • [x] Write automated tests
    • [x] ~Edit README show that the there is a .NET SDK~. Probably we should not do this until we have a docs page.
    • [x] ~Use the Extism.runtime.win-x64 package in the sample project~

    Out of scope for this PR:

    • Json Serialization/Desererialization support
    opened by mhmd-azeez 68
  • building for macOS-x86_64 but attempting to link with file built for macOS-arm64 GoLang 1.19.3 version

    building for macOS-x86_64 but attempting to link with file built for macOS-arm64 GoLang 1.19.3 version

    I'm trying to execute the sample Go code from Extism from https://extism.org/docs/integrate-into-your-codebase/go-host-sdk. After the error, I tried to debug the problem, so I ran this pretty minimal code.

    import (
	"github.com/extism/extism"
)

func callWasm() {
	ctx := extism.NewContext()
	defer ctx.Free() // this will free the context and all associated plugins
}

    I'm getting the following error.

    GoLang Version: 1.19.3

    Mac Chip: Apple M1 Pro

    # github.com/extism/extism
ld: warning: ignoring file /usr/local/lib/libextism.dylib, building for macOS-x86_64 but attempting to link with file built for macOS-arm64
Undefined symbols for architecture x86_64:
  "_extism_context_free", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_context_free in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_context_free)
  "_extism_context_new", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_context_new in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_context_new)
  "_extism_context_reset", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_context_reset in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_context_reset)
  "_extism_error", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_error in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_error)
  "_extism_log_file", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_log_file in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_log_file)
  "_extism_plugin_call", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_call in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_call)
  "_extism_plugin_config", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_config in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_config)
  "_extism_plugin_free", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_free in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_free)
  "_extism_plugin_function_exists", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_function_exists in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_function_exists)
  "_extism_plugin_new", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_new in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_new)
  "_extism_plugin_output_data", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_output_data in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_output_data)
  "_extism_plugin_output_length", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_output_length in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_output_length)
  "_extism_plugin_update", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_plugin_update in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_plugin_update)
  "_extism_version", referenced from:
      __cgo_48636a3be7c2_Cfunc_extism_version in _x002.o
     (maybe you meant: __cgo_48636a3be7c2_Cfunc_extism_version)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)

    Note: I'm able to run the sample wasm file with the Extism command line tool in my machine, only the GoLang code is not working

    opened by ananthdurai 13
  • Refactor java-sdk

    Refactor java-sdk

    • Refactor Plugin
    • Make Plugin and Context AutoClosable to automatically free resources in TWR blocks
    • Add missing javadoc
    • Upgrade to Junit 5
    • Reorganize packages
    • Use more meaningful names
    • Introduce ExtismException
    • Make JSON serialization pluggable
    • Lower required java version to Java 17

    Improves #117

    opened by thomasdarimont 9
  • Add ExtismContext to SDK + better errors for failed register/update

    Add ExtismContext to SDK + better errors for failed register/update

    • Adds ExtismContext instead of global PLUGINS registry
      • Adds extism_context_new, extism_context_free and extism_context_reset
      • Requires updating nearly every SDK function to add context parameter
    • Renames some SDK functions to follow better naming conventions
      • extism_plugin_register -> extism_plugin_new
      • extism_output_get -> extism_plugin_output_data
      • extism_output_length -> extism_plugin_output_length
      • extism_call -> extism_plugin_call
    • Updates extism_error to return the context error when -1 issued for the plug-in ID
    • Adds extism_plugin_free to remove an existing plugin
    • Updates SDKs to include these functions
      • It looks like only Rust, OCaml, Python, PHP, C++ have the ability to write destructors, while Haskell, Ruby, Go, C and Node will require users to call the free method - do we want to keep it like this? I think it's nice to take advantage of the automatic cleanup when possible, but also see the value in having the SDKs match.
    • 100% coverage for all SDKs (except C++, since it can use C functions directly for some operations)
    opened by zshipko 8
  • feat: initial php sdk

    feat: initial php sdk

    WIP

    @zshipko I could use an initial review here, as I get a segfault at the call to FFI::memcpy and am a bit stuck.

    Also, unsure about php library structure, so will need to investigate that. Used this to generate a php library and removed a bunch of stuff.

    opened by nilslice 8
  • feat: Improve usability of manifest types

    feat: Improve usability of manifest types

    • Adds some helper functions for creating a manifest, mostly helpful for the Rust SDK
    • Renames ManifestWasm to Wasm
    • Renames ManifestMemory to MemoryOptions
    • ManifestWasm and ManifestMemory have been marked as deprecated
    • Adds an alias from MemoryOptions::max_pages to MemoryOptions::max in serde specification
    enhancement 
    opened by zshipko 7
  • WASI-based filesystem access

    WASI-based filesystem access

    All the use-cases I might want to use extism for (a shell, a package manager) involve fairly heavy filesystem access, and I think it would make a lot of sense to provide filesystem access to plugins (with configurable scope limitations). I saw on the site how you were looking for feedback on this front so you can consider this an official feature request for the functionality. Extism looks awesome and I'd love to make use of it!

    opened by zkat 6
  • Failed to run simple rust plugin in playground

    Failed to run simple rust plugin in playground

    Following error was found while trying to run a pluin following the tutorial "write a plugin in rust". I also tried to upload code.wasm shipped with source code and it works fine instead.

    image

    • env rustc 1.66.0 (69f9c33d7 2022-12-12)

    • my cargo.toml:

    [lib]
crate_type = ["cdylib"]

[dependencies]
extism-pdk = "0.1.1"
serde = { version = "1.0", features = ["derive"] }

    wasm_web_plugin.zip

    opened by piaoger 5
  • feat: add var/config functions, merge ExtismPlugin and ExtismPluginCall

    feat: add var/config functions, merge ExtismPlugin and ExtismPluginCall

    This also only instantiates the module once and combines ExtismPlugin with ExtismPluginCall, I can pull out just the new function implementations if that's an issue.

    opened by zshipko 5
  • Fix large allocations

    Fix large allocations

    This PR fixes an issue where large allocations are currently causing some arithmetic to wrap which caused a panic.

    This was discovered by calling http_get on youtube.com from the http example in the Rust PDK

    opened by zshipko 4
  • feat: access input/output buffers directly

    feat: access input/output buffers directly

    This PR updates extism_output_get to return an actual pointer to the output value (const uint8_t* extism_output_get(PluginIndex plugin) instead of void extism_output_get(PluginIndex plugin, uint8_t *buffer, uint64_t length)), this pointer will only be valid until the next call, but it makes it possible to access the output data without copying.

    The only issue I can see is this complicates the lifetime of the output value. In most cases a copy will still be made or the output buffer will only be used until the next call, otherwise it's possible to lose access to an existing output once the same plugin is called again. But this is easily fixed by making a copy in the host program. Overall I think this change will provide a lot of benefit, do you think it's worth it?

    EDIT: Now the input buffer is also not copied and the same issue applies - the input buffer must not change during call - I think this is a reasonable constraint though.

    opened by zshipko 4
  • Support jdk11

    Support jdk11

    Hi,

    We want to use Extism on our project Otoroshi but we need to run it on jdk11

    This pull request makes everything run smoothly on jdk11

    If you have any suggestion about this pull request, i'm open to it

    Thanks for your time

    opened by Zwiterrion 1
  • FR: Iterable Object for Functions

    FR: Iterable Object for Functions

    Hi,

    It would be amazing to have an iterable object for the functions available to call on the wasm module. The online playground does this to populate the dropdown but I believe it requires secondary files. If possible, loading from the .wasm file would be amazing for schema checking and matching functions at runtime. Checking if a specific function exists on the wasm object is currently implemented, however creating an iterable opens this up further.

    Keep up the great work, Bowbee

    opened by Bowbee 0
  • feat(rust-sdk): `Context` collection

    feat(rust-sdk): `Context` collection

    Example implementation of the idea proposed in #202.

    The goal here is to treat Context as it is treated internally; a collection of plugins. This will allow users to store and call plugins whenever they want, without them being freed and without having to cache them again.

    A lot of the code doesn't make use of the libraries conventions, I threw it together as an example. Let me know what you think and if we should go ahead with implementing this.

    opened by ok-nick 15
  • Rich `Context` API and decoupling `Plugin`s

    Rich `Context` API and decoupling `Plugin`s

    extism::Context has two, very minimal functions for managing a pool of plugins. Underneath the hood, in extism_runtime::Context, there are numerous useful functions that could be exposed so that the user doesn't need to reimplement it themselves.

    I found that I often need to manage a pool of Plugins. To currently do this, you must create your own collection, when extism_runtime::Context already does this for you.

    On that note, I think it might be a little nicer to decouple a Plugin and a Context. You can create a plugin on your own, then when you want to use it, you upload it to a Context. I think the reason this wasn't done originally is so that you don't have to pass a Context to Plugin::call, which is why I also think that there should be a distinction between a Plugin and a PluginSchema. Essentially, a PluginSchema is a way to upload a WASM module to a Context with a given Manifest and other additional information. After upload, it will return a reference to the Plugin to which you can then call functions on. When you need it for another time, you pull the Plugin from the Context, using the proper API, then operate on it. This turns a Context into a custom collection for managing Plugins.

    In addition to the above, it would also be useful to expose PluginIndexs and possibly extism_runtime::Context::next_id. I found that I want to generate ids for plugins to identify them, however, these plugins are not directly managed by extism. Maybe the "unique id" system should be decoupled into a separate struct that is used by a Context and could be passed by the user (otherwise defaulted).

    Edit: I see the point of why Contexts aren't a proper collection is because a Plugin can be freed after being dropped. That's nice and all, but often times I need to store it and call it later. Why force the user to create their own collection when it's done internally? extism can still expose a wrapper struct around a Plugin that frees on drop, but I just don't think it's that useful when users can free them by calling a separate function.

    opened by ok-nick 4
  • feat: Add C API for host functions + support for C++, Python, Go, Node, OCaml

    feat: Add C API for host functions + support for C++, Python, Go, Node, OCaml

    • New types:

      • ExtismValType - Enum of WebAssembly types
      • ExtismValUnion - A union of the possible WebAssembly types
      • ExtismVal - A struct with ExtismValType and ExtismValUnion
      • ExtismFunction - The host function wrapper type
      • ExtismFunctionType - The type of the host function callback
      • ExtismCurrentPlugin - Provides access to the currently running plugin from inside a host function

    • New functions:

      • extism_function_new - Create a new ExtismFunction
      • extism_function_free - Free an ExtismFunction
      • extism_current_plugin_memory, extism_current_plugin_memory_alloc, extism_current_plugin_memory_free, extism_current_plugin_memory_length - Manage plugin memory from inside a host functions

    • Updated functions

      • extism_plugin_new and extsim_plugin_update - now accept two extra parameters for ExtismFunction* array and length of that array

    Notes

    • Host functions take a user-data argument, which is owned by the resulting ExtismFunction and will be cleaned up when extism_function_free is called (if a cleanup function was passed in with the user data)
    • Host functions in every SDK require working with ExtismVal arguments directly, this is pretty low-level for what is kind of a high-level feature. We could work on adding some types to the SDKs that make working with pointers to plugin data more accessible, maybe something similar to how the Rust PDK handes input/output data.
    • In each language the host functions more-or-less share a signature: (CurrentPlugin plugin, Val inputs[], Val outputs[], userData)
      • C, C++, OCaml and Go take a single userData argument but Python and Node take a "rest" argument which allows passing any number of user-data values
    • Go requires the host function to be exported: https://github.com/extism/extism/blob/f9eb5ed8395218f08627622517e801491efb2a53/go/main.go#L13-L26
    • Zig and Ruby should be relatively simple to add host functions to next but I haven't really looked into Elixir, .NET or Java yet.
    • Also closes #20
    opened by zshipko 4
Releases(v0.1.0)
Owner
Extism
Make all software programmable. Extend from within.
Extism
GitHub https://extism.org
Donate To Charity (A.K.A. Public Goods) Without Lossing Anything

Donate To Charity (A.K.A. Public Goods) Without Lossing Anything

Gajesh Naik 10 Mar 3, 2022
[WIP] Store bookmarks to anything

Handyman - store bookmarks to anything Handyman Acronym's Noticeably Dumb, Yet Makes A Name The motivation Internet browsers have bookmarks. File mana

Mateusz Koteja 2 Nov 8, 2022
Ask ChatGPT for a shell script, code, or anything, directly from your terminal 🤖🧠👨‍💻

ShellGPT Ask ChatGPT for a shell script, code, or anything, directly from your terminal ?? ?? ??‍?? Demo Install The binary is named gpt when installe

null 4 May 15, 2023
🐚+🦞 Ultra-portable Rust game engine suited for offline 2D games powered by WebAssembly

pagurus ?? + ?? Ultra-portable Rust game engine suited for offline 2D games powered by WebAssembly. Examples Snake Traditional snake game: examples/sn

Takeru Ohta 20 Mar 7, 2023
A tool that makes writing WebAssembly Text files easier.

Silly WAT Linker SWL is a tool that makes writing WebAssembly Text files easier. It is future-proof and simple because it doesn’t actually understand

Surma 27 Feb 17, 2023
High-performance Javascript color gradient library powered by Rust + WebAssembly

colorgrad-js High-performance Javascript color gradient library powered by Rust + WebAssembly. No dependencies. Faster than d3-scale, chroma-js, culor

Nor Khasyatillah 168 Apr 25, 2023
Gauzilla: a 3D Gaussian Splatting renderer written in Rust for WebAssembly with lock-free multithreading

Gauzilla A 3D Gaussian Splatting (3DGS) renderer written in Rust for platform-agnostic WebAssembly (WASM) with lock-free multithreading. Uses WebGL an

Yoshi Sato 90 Jan 2, 2024
Rust crate to extend io::Read & io::Write types with progress callbacks

progress-streams Rust crate to provide progress callbacks for types which implement io::Read or io::Write. Examples Reader extern crate progress_strea

Pop!_OS 19 Dec 3, 2022
Small crate to extend `reqwest` to be able to send with digest auth flow.

diqwest This crate extends reqwest to be able to send requests with digest auth flow. It is currently implemented for async usage only. When you send

Mathias Oertel 14 Aug 29, 2022
zink! is a library for developing ink! smart contracts with useful Rust macros that extend functionality and reduce boilerplate code.

zink! Smart Contract Macros This is a helper library for developing ink! smart contracts. It contains useful Rust macros that extend functionality and

Scio Labs 3 Nov 3, 2023
A Rust LLaMA project to load, serve and extend LLM models

OpenLLaMA Overview A Rust LLaMA project to load, serve and extend LLM models. Key Objectives Support both GGML and HF(HuggingFace) models Support a st

Compute IO 4 Apr 9, 2024
A webring of people who make cool stuff. technology, music, art, writing, anything goes!

a webring of people who make cool stuff. technology, music, art, writing, anything goes!

Kognise 44 Dec 6, 2022
Donate To Charity (A.K.A. Public Goods) Without Lossing Anything

Donate To Charity (A.K.A. Public Goods) Without Lossing Anything

Gajesh Naik 10 Mar 3, 2022
A canvas on which you can draw anything with ease before drawing the pixels on your small hardware display.

embedded-canvas    canvas - a piece of cloth backed or framed as a surface for a painting NOTE: This crate is still in development and may have breaki

Lechev.space 13 Aug 31, 2022
Watch for changes on a webpage and do anything with it!

Sukurappa Watch for changes on a webpage and do anything with it! Install With cargo: cargo install sukurappa Or use the install-script and add $HOME/

Jean-Philippe Bidegain 2 Sep 4, 2022
[WIP] Store bookmarks to anything

Handyman - store bookmarks to anything Handyman Acronym's Noticeably Dumb, Yet Makes A Name The motivation Internet browsers have bookmarks. File mana

Mateusz Koteja 2 Nov 8, 2022
Ask the Terminal Anything (ATA): ChatGPT in the terminal

ata: Ask the Terminal Anything ChatGPT in the terminal TIP: Run a terminal with this tool in your background and show/hide it with a keypress. This ca

Rik Huijzer 147 Mar 8, 2023
Ask ChatGPT for a shell script, code, or anything, directly from your terminal 🤖🧠👨‍💻

ShellGPT Ask ChatGPT for a shell script, code, or anything, directly from your terminal ?? ?? ??‍?? Demo Install The binary is named gpt when installe

null 4 May 15, 2023
Open source p2p share for devs to share anything with teammates across machines securely.

Secure share Share anything with teammates across machines via CLI. Share is a tool for secure peer-to-peer connections, enabling direct communication

Onboardbase 10 Aug 4, 2023
An asynchronous runtime compatible with WebAssembly and non-WebAssembly targets.

Promise x Tokio = Prokio An asynchronous runtime compatible with WebAssembly and non-WebAssembly targets. Rationale When designing components and libr

Yew Stack 29 Feb 6, 2023