Cookbook / Production

Distributed tracing with OpenTelemetry

A span per request, trace context passed to other services, and export to any OTLP collector.

When one request fans out to a database, an LLM and a vendor API, logs alone cannot tell you where the time went. axeon-otel opens a server span for every request, continues the caller's trace, and exports spans over OTLP/HTTP to Jaeger, Tempo, Honeycomb or any collector.

Terminal

cargo add axeon-otel tracing tracing-subscriber

src/main.rs

use axeon::Server;
use axeon_otel::{Telemetry, TraceRequests};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;

#[tracing::instrument]
async fn load_products(category: String) -> Vec<&'static str> {
    // Spans from `tracing` nest under the request's span.
    tokio::time::sleep(std::time::Duration::from_millis(30)).await;
    vec!["lamp", "desk"]
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    // The endpoint defaults to $OTEL_EXPORTER_OTLP_ENDPOINT, else http://localhost:4318.
    let telemetry = Telemetry::builder("catalog-api")
        .endpoint("http://localhost:4318")
        .sample_ratio(0.2) // keep 20% of new traces
        .attribute("deployment.environment", "production")
        .build()?;

    // Log lines and exported spans from one subscriber.
    tracing_subscriber::registry()
        .with(tracing_subscriber::fmt::layer())
        .with(telemetry.layer())
        .init();

    let mut app = Server::new();
    app.middleware(TraceRequests::new()); // first, so it covers everything else
    app.get("/v1/products/:id", |_req| async {
        let products = load_products("lighting".into()).await;
        format!("{:?}", products)
    });

    let result = app.listen("127.0.0.1:3000");

    // After the server stops: export what is still waiting.
    telemetry.shutdown();
    result
}

Terminal

# A local collector with a UI, e.g. Jaeger:
docker run --rm -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one
curl http://127.0.0.1:3000/v1/products/7
open http://localhost:16686      # find the "catalog-api" service

What you get#

One SERVER span per request, named by the route pattern (GET /v1/products/:id, not the path), so traces group sensibly. It carries the standard HTTP attributes and is marked as an error for 5xx responses.
The caller's traceparent is continued, and the request's own traceparent is updated to point at the new span. Pass the request to the outbound client with .context(&req) and its calls become children of the request, so one operation is visible across services.
#[tracing::instrument] on your functions shows up in the trace.

Notes#

The sampler follows the caller: a request whose caller sampled its trace is always kept, and one whose caller did not is never kept, so traces stay whole. New traces use sample_ratio.
A caller can therefore turn tracing on for its own requests. On a public endpoint, strip traceparent at your edge proxy for untrusted clients if volume matters.
Only the path is recorded, never the query string, so tokens in URLs do not leak.
For https:// collectors enable the crate's tls feature. Hosted backends usually want an API key: .header("x-honeycomb-team", key).