Continuous runtime observablity SDKs to monitor WebAssembly code.

Last update: Jun 8, 2023

Overview

Observe

Observe is an observability SDK for WebAssembly. At the moment we support wasmtime hosts and we output opentelemetry data to stdout. We plan to support more adapters and wasm runtimes in the near future. If there is a configuration you are interested in reach out to [email protected].

Overview

There are two components to this process:

Instrumenting the Wasm code
Including a runtime/host SDK

Instrumenting Wasm Module

This package expects the wasm code to be instrumented using our instrumenting compiler. The only way to instrument your wasm right now is through the instrumentation service. The easiest way to do this is to send up your wasm with curl and get an instrumented wasm module back:

curl -F [email protected] https://compiler-preview.dylibso.com/instrument -X POST -H 'Authorization: Bearer <your-api-key>' > code.instr.wasm

🔑 You can get an API key by contacting [email protected].

Note: The Instrumentation Service (https://compiler-preview.dylibso.com/instrument) only re-compiles a .wasm binary and returns the updated code. We do not log or store any information about your submitted code. The compilation also adds no telemetry or other information besides the strictly-necessary auto-instrumentation to the .wasm instructions. If you would prefer to run this service yourself, please contact [email protected] to discuss the available options.

Including a runtime SDK

First install the cargo dependency for the SDK:

[dependencies]
dylibso-observe-sdk = { git = "https://github.com/dylibso/observe-sdk.git" }

Note: A runnable example can be found here.

use dylibso_observe_sdk::adapter::otelstdout::OtelStdoutAdapter;
use tokio::task;

#[tokio::main]
pub async fn main() -> anyhow::Result<()> {
    let args: Vec<_> = std::env::args().skip(1).collect();
    let data = std::fs::read(&args[0])?;
    let function_name = "_start";
    let config = wasmtime::Config::new();

    // Create instance
    let engine = wasmtime::Engine::new(&config)?;
    let module = wasmtime::Module::new(&engine, data)?;
    
    // Create Observe SDK adapter for OpenTelemetry output written to STDOUT
    let adapter = OtelStdoutAdapter::new();

    // Setup WASI
    let wasi_ctx = wasmtime_wasi::WasiCtxBuilder::new()
        .inherit_env()?
        .inherit_stdio()
        .args(&args.clone())?
        .build();
    let mut store = wasmtime::Store::new(&engine, wasi_ctx);
    let mut linker = wasmtime::Linker::new(&engine);
    wasmtime_wasi::add_to_linker(&mut linker, |wasi| wasi)?;

    // Provide the observability functions to the `Linker` to be made available
    // to the instrumented guest code. These are safe to add and are a no-op
    // if guest code is uninstrumented.
    let trace_ctx = adapter.start(&mut linker).await?;
    
    // get the function and run it, the events pop into the queue
    // as the function is running
    let instance = linker.instantiate(&mut store, &module)?;
    let f = instance
        .get_func(&mut store, function_name)
        .expect("function exists");
    f.call(&mut store, &[], &mut []).unwrap();
    task::yield_now().await;

    // shut down the trace context to flush all remaining spans and cleanup
    trace_ctx.shutdown().await;

    Ok(())
}

Development

Building

To build the current wasmtime-based SDK, run:

$ cargo build

Testing

$ make test

Compile the Test Modules

These are already checked in, but you can compile and instrument them with. Please check in any changes in the test/ directory.

make instrument WASM_INSTR_API_KEY=<your-api-key>

Running Zipkin

One of the test adapters will output to Zipkin, defaulting to one running on localhost.

docker run -d -p 9411:9411 openzipkin/zipkin

Comments

chore: simplify otel adapter api, minor cleanup

This adds a couple types in the otelstdout adapter crate to simplify the API a bit... it hides the creation of all the id's and collector things, provides a "trace context" that is used on a per-module invocation and is where the trace id is set.

Few additional edits and cleanups to the README and Cargo.toml, and renames the add_to_linker to start so it contrasts shutdown a little bit more clearly.

opened by nilslice 1
fix: A homerolled otel json formatter

Because of the issues we've had with the Opentelemetry provided crate this is an attempt at extracting just the JSON formatting. It's still early days but this creates the correct json and makes it much easier to set custom trace_id's and attributes on the spans.

opened by wwkeyboard 1
chore: move rust code into parent dir
In preparation for additional language/runtime support, moving the existing SDK code into a language specific directory will make it easier to minimize any impact to dependent users.

To check this, run:

make test

GitHub Actions

cargo build

Adding a Cargo workspace in the root here should keep all of these working just fine.
opened by nilslice 0
feat: Integration test runner that tests the examples
This test runner uses executes the examples and checks their output. This lets us be clearer about what specifically failed in the test output.

Downsides:

Highly coupled to the examples and the test module, if any of those change it'll break the test

Parsing the output can get annoying

Upsides:

Fully tests the adapters and modules
opened by wwkeyboard 0
feat: Support Datadog
Some stuff left to do:

Flesh out the Config object, should we use a builder pattern?

We can put http level tags and stuff about the request in Config#default_tags, but that feels wrong. maybe there should be a way to pass in metadata before or when calling the func but that feels like too big of a change.

Ideally we could pass in some more metadata after the func is called. e.g., where would you pass in an http status code?
opened by bhelx 0
Loosen crate dependency version requirements

wasmtime, serde, etc could all probably use less strict versions to increase compatibility across variety of end-user codebases. We should look at the "oldest" versions of these that support our needs and make those then minimum requirements. serde being set to "1" is probably safe.

opened by nilslice 0
Benchmarking

We need repeatable benchmarks around the latency this introduces into executing the module. The benchmarks should focus on initially on anything that's run in line with the module as that's the most important place not to introduce latency. After we are confident in that code path we can concentrate on the collector and the adapters.

opened by wwkeyboard 0
Better testing

The testing situation is currently grim, mostly done by hand. A first start is to unit test the individual modules. We also need to simplistic and well understood modules that we can run and verify that the captured events match what we expected from the module. This will involve building at least one of those modules, and making a list of others we'll need. Then well need a way to test that the modules are events are being generated correctly, however we should not let changes to the adapters break these tests.

opened by wwkeyboard 0

Owner

Dylibso

We make @extism and other software to help developers build with WebAssembly & take it to production.

GitHub

🖥 Simple Arduino Serial Monitor

Born out of the desire to connect to an Arduino without having to run the whole Arduino suite.

2 Mar 19, 2022

Rust program to monitor Windows 10 Registry keys for changes or modifications.

RegMon This Rust program monitors changes to multiple registry keys in Windows 10 and writes the changes to a text file. It also periodically sends a

3 Jan 16, 2023

Github mirror of codeberg repo. Monitor live bandwidth usage/ network speed on PC. Native version also available for Android, separately.

Netspeed Monitor Netspeed is a cross-platform desktop application that shows the live upload speed, download speed and day's usage as an overlay. Feat