Middlewares and tools to integrate axum + tracing + opentelemetry

David Bernard

Last update: Jan 4, 2023

Related tags

Utilities axum-tracing-opentelemetry

Overview

axum-tracing-opentelemetry

Middlewares and tools to integrate axum + tracing + opentelemetry.

Read OpenTelemetry header from incoming request
Start a new trace if no trace found in the incoming request
Trace is attached into tracing'span

For examples, you can look at:

the examples folder
davidB/sandbox_axum_observability: Sandbox to experiment axum and observability. This example shows also propagation of the trace between tracing span and service (via reqwest).

//...
use axum_tracing_opentelemetry::{
  opentelemetry_tracing_layer,
  // optional tools to init tracer (require features)
  CollectorKind, init_tracer
};

fn init_tracing() {

    let otel_tracer = init_tracer(CollectorKind::Otlp).expect("setup of Tracer");
    let otel_layer = tracing_opentelemetry::layer().with_tracer(otel_tracer);

    let subscriber = tracing_subscriber::registry()
        //...
        .with(otel_layer);
    tracing::subscriber::set_global_default(subscriber).unwrap();
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    init_tracing();
    let app = app();
    // run it
    let addr = &"0.0.0.0:3000".parse::<SocketAddr>()?;
    tracing::warn!("listening on {}", addr);
    axum::Server::bind(&addr)
        .serve(app.into_make_service())
        .with_graceful_shutdown(shutdown_signal())
        .await?;
    Ok(())
}

fn app() -> Router {
    // build our application with a route
    Router::new()
        .route("/health", get(health))
        // opentelemetry_tracing_layer setup `TraceLayer`, that is provided by tower-http so you have to add that as a dependency.
        .layer(opentelemetry_tracing_layer())
}

async fn shutdown_signal() {
    //...
    opentelemetry::global::shutdown_tracer_provider();
}

History

0.1.0

Code originally created at part of axum-extra Add OpenTelemetry middleware by davidpdrsn · Pull Request #769 · tokio-rs/axum
Code copied and modified as part of davidB/sandbox_axum_observability: Sandbox to experiment axum and observability
Published as a standalone crate with OK from original author

Comments

http tags seems missing

the span create code is:

https://github.com/davidB/axum-tracing-opentelemetry/blob/67f64a6cd474ea9e189bca1649d7389c2c653d99/src/middleware.rs#L146-L163

but I do not see any http.xxx in final span.

and otel.kind = %opentelemetry::trace::SpanKind::Server, but in the result, I see it is still internal

I add a simple dbg!(&trace_id, &span, &user_agent); to get the trace_id so I can confirm that the span is created from here.

bug

opened by ttys3 10
Release for axum 0.6.0 release candidate? (MatchedPath not working)

It seems that when used with axum 0.6 release candidates, the MatchedPath isn't found (leading to not properly extracting http.route). Would it be possible to have an RC-release for 0.6?

You might want to wait for the actual release, in which case, this issue may help other people save on debugging time :)
enhancement good first issue

opened by fiadliel 5
Going from layer -> OTLP collector -> Jaeger, causes a warning to be displayed in Jaeger
When opening a root trace generated by this library, the following warning is displayed in Jaeger:

invalid parent span IDs=c1b973f219df2f05; skipping clock skew adjustment

This is a standalone server for now, so nothing is sending trace ids (yet), it just receives raw requests from a browser.
opened by MidasLamb 4
Update / add new examples

Hi!

I'm trying out this whole Prometheus/Tempo/OtelCollector setup, but it seems like the only example is a bit outdated (?).

Can you please update it? Or maybe add a new one?

opened by TmLev 4
update opentelemetry dependencies to the latest version

I also had to use a str instead of SpanKind because its Display impl was removed. I checked on tracing-opentelemetry and they did the same.

I suppose we need to go to 0.5 to release it?

opened by dbofmmbt 3

Add convenience function to set trace-Id/span-Id

In tools/mod.rs line 86 there is the convenience function find_current_trace_id() defined. This is nice to get the Trace-Id from from context (for the Span-Id is basically the same).

It would be nice to have a counterpart to set the Trace-Id and Span-Id from a string.

set_trace_id_in_context(trace_id: String) { ... }

My use-case is the following: I send a message with kafka where i add the Trace-Id and Span-Id in b3-format in the message:

pub fn get_b3_trace_id() -> Option<String> {
    use opentelemetry::trace::TraceContextExt;
    use tracing_opentelemetry::OpenTelemetrySpanExt;
    let context = tracing::Span::current().context();
    let span = context.span();
    let span_context = span.span_context();

    let span_id = span_context
        .is_valid()
        .then(|| span_context.span_id().to_string());

    let trace_id = span_context
        .is_valid()
        .then(|| span_context.trace_id().to_string());

    if trace_id.is_none() || span_id.is_none() {
        None
    } else {
        // https://github.com/openzipkin/b3-propagation
        Some(format!("{}-{}-1", trace_id.unwrap(), span_id.unwrap()))
    }
}

On receiver side i have problems though to set the context correctly so all instrumented functions (with their spans) are tracked as children under the received Trace-Id. I've tried different approaches but all implementations have the problem that only the directly instrumented code block is mapped to the correct Trace-Id (i only see a span under the trace for the block where I call the .instrument(span).

I use code similar to this:

let span = span!(Level::TRACE, "");
let context = get_context_from_b3(tracing_propagator.clone(), trace_id.to_string());
tracing_opentelemetry::OpenTelemetrySpanExt::set_parent(&span, context);
{
    ...
   // Problem: functions that are called here, that are annotated with #[instrument(name = "xyz"]
   // have a different Trace Id and are not correlated to the span that instruments the current block
}.instrument(span);


pub fn get_context_from_b3(propagator: Arc<Propagator>, b3_trace_id: String) -> Context {
    let mut extractor: HashMap<String, String> = HashMap::new();
    extractor.insert(B3_SINGLE_HEADER.to_string(), b3_trace_id);

    let context = propagator
        .extract_single_header(&extractor)
        .expect("Couldn't extract trace header");

    use opentelemetry::trace::TraceContextExt;

    Context::new().with_remote_span_context(context)
}

I would appreciate if you can provide a convenience function that simplifies setting the trace-id / (parent-)span-id or if you could explain how to solve it. Thanks in advance!

opened by cschaible 2

`http.client_ip` fallback isn't appropriate by the otel spec

Here, the connection information is used as a fallback in case the X-Forwarded-For header isn't set:

https://github.com/davidB/axum-tracing-opentelemetry/blob/f0672632df09d05a8520604a901dd423de20042d/src/middleware.rs#L129-L135

The http.client_ip should hold the client IP, but we may be connected to a proxy instead.

From the otel spec:

The IP address of the original client behind all proxies, if known (e.g. from X-Forwarded-For)

I think it would be a good idea to remove the fallback and conditionally insert this attribute to the span.

opened by dbofmmbt 2

Middleware to extend the headers

Hey there @davidB , great job with the project. I wonder if would make sense to add a feature to extend the headers in the current service, I know it already reads from the income request.

use axum_tracing_opentelemetry::propagate_headers_middleware;

        .layer(middleware::from_fn(propagate_headers_middleware))
        .layer(opentelemetry_tracing_layer())

Following B3 pattern https://docs.rs/opentelemetry/0.9.1/i686-pc-windows-msvc/opentelemetry/sdk/propagation/struct.B3Propagator.html

   Client Tracer                                                  Server Tracer     
┌───────────────────────┐                                       ┌───────────────────────┐
│                       │                                       │                       │
│   TraceContext        │          Http Request Headers         │   TraceContext        │
│ ┌───────────────────┐ │         ┌───────────────────┐         │ ┌───────────────────┐ │
│ │ TraceId           │ │         │ X-B3-TraceId      │         │ │ TraceId           │ │
│ │                   │ │         │                   │         │ │                   │ │
│ │ ParentSpanId      │ │ Inject  │ X-B3-ParentSpanId │ Extract │ │ ParentSpanId      │ │
│ │                   ├─┼────────>│                   ├─────────┼>│                   │ │
│ │ SpanId            │ │         │ X-B3-SpanId       │         │ │ SpanId            │ │
│ │                   │ │         │                   │         │ │                   │ │
│ │ Sampling decision │ │         │ X-B3-Sampled      │         │ │ Sampling decision │ │
│ └───────────────────┘ │         └───────────────────┘         │ └───────────────────┘ │
│                       │                                       │                       │
└───────────────────────┘                                       └───────────────────────┘

https://github.com/openzipkin/b3-propagation

Demo:

Let me know if that make sense, I can definitely help with it.

opened by raphamorim 2

⬆️ Update axum requirement from 0.5 to 0.6
Updates the requirements on axum to permit the latest version.

Release notes

Sourced from axum's releases.

axum - v0.6.0

Routing

fixed: Nested routers are now allowed to have fallbacks (#1521):

let api_router = Router::new() .route("/users", get(|| { ... })) .fallback(api_fallback);
let app = Router::new() // this would panic in 0.5 but in 0.6 it just works // // requests starting with /api but not handled by api_router // will go to /api_fallback .nest("/api", api_router);

The outer router's fallback will still apply if a nested router doesn't have its own fallback:

// this time without a fallback let api_router = Router::new().route("/users", get(|| { ... }));
let app = Router::new() .nest("/api", api_router) // api_fallback will inherit this fallback .fallback(app_fallback);

breaking: The request /foo/ no longer matches /foo/*rest. If you want to match /foo/ you have to add a route specifically for that (#1086)

For example:

use axum::{Router, routing::get, extract::Path}; let app = Router::new() // this will match /foo/bar/baz .route("/foo/*rest", get(handler)) // this will match /foo/ .route("/foo/", get(handler)) // if you want /foo to match you must also add an explicit route for it .route("/foo", get(handler));
async fn handler( // use an Option because /foo/ and /foo don't have any path params params: Option<Path<String>>,

... (truncated)

Commits

1b6780c axum 0.6 and friends 🎉 (#1570)

0b26411 Change Router::with_state and impl Service for Router\<()> (#1552)

fde38f6 docs: Add axum-csrf-sync-pattern to ECOSYSTEM.md (#1567)

102d588 add showcase freedit (#1559)

6771729 Improve State and Router docs (#1543)

7d0bb28 Minor doc fixes (#1562)

ddee1c1 docs: add how to use Arc<AppState> with the cookies examples (#1560)

c7e696b provide links to axum-sessions and axum-login (#1553)

3ab3e5f Use http re-export in docs (#1554)

8d2fb36 Remove unused imports from doctest (#1551)

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

dependencies rust
opened by dependabot[bot] 1
Compile with axum 0.6.0-rc.2

Currently extracting the route path silently fails with axum 0.6 RC. With this change, the path is available.

I'm not sure of the timeframe for the 0.6 release; I would like to hope it would be available soon, but https://github.com/tokio-rs/axum/issues/1416 makes me think it may take a bit more time.

opened by fiadliel 1
The span_kind in the generated SpanData is Internal

I'm not sure what causing the issue, but the span_kind in the generated SpanData is always Internal. I see the MakeSpan sets it to "server", but tracing_opentelemetry still generates Internal.

Do you have any clue why this could happen ? (The issue is that opentelemetry_application_insights generates different insight log based on this property: request, dependency, etc)

opened by gzp-crey 1
Add gRPC layer

Hello, I'm sure there is a bunch of stuff you will want me to do to get this ready for merge. I'm happy to do so. I just wanted to open this and ping for review. Thanks for putting this together. It is already working quite nicely with Tonic & Axum (thanks to the shared tower foundation).

opened by thedodd 3

Example doesn't work

When running the example with a local OTLP collector the following error is produced after making a request to /:

OpenTelemetry trace error occurred. Exporter otlp encountered the following error(s): no http client, you must select one from features or provide your own implementation

opened by MidasLamb 5

Make tracer initialization configurable
The initialization of the tracer isn't really customizable at the moment.

The function init_tracer as shown in the sample application only takes a CollectorKind parameter to specify the tracer type. This though isn't really satisfying as the tracer parameters cannot be customized:

fn init_tracer(kind: CollectorKind)

I've seen that there's the more specific function

fn init_tracer_jaeger(resource: Resource)

which is called inside the init_tracer function, but the Resource type isn't sufficient to configure things like the Sampler or the collector endpoint (has to be set on the pipeline).

My solution so far is to copy the content of the init_tracer_jaeger function into my own code and configure the tracer there. It would be better though, if relevant parameters could be configured / overwritten by the user without copying the code.
opened by cschaible 4