Back to all posts

One Observability Timeline for Traces and Errors

How Temps merges requests, OpenTelemetry traces, errors and revenue events into one time-ordered timeline, with errors linked to the traces that produced them.

David Viejo
David Viejo · Updated · Published

If you're debugging a production incident across separate request logs, trace viewers, error dashboards, and revenue metrics, the fastest way to correlate everything is a single merged timeline — not four browser tabs. The Unified Observe page, shipped in the Temps v0.1.0 beta line, merges all four signal types into one time-ordered stream so you can follow an incident from HTTP requests through OTel spans and error events to missing revenue in one view.


How Do You Get a Unified Observability Timeline Without Multiple Tools?

Most teams answer this question by subscribing to Datadog (or paying for Sentry + Plausible + a separate request logger). Temps takes a different approach: because it already stores requests, traces, errors, and revenue events natively, it can merge them in a single backend query and display them in one view — no exporter glue, no separate SaaS accounts.

How it works, in brief:

  1. Four signal types, one query — the temps-observability crate runs one query per enabled kind in parallel (requests, traces, errors and revenue events), k-way merges them by ts DESC, and trims to a single page. Every row is a discriminated ObservabilityEvent union that carries all the data the side panel needs — no follow-up fetch in the common case.
  2. Cross-source correlation via trace_id — migration m20260502_000001_add_observe_correlation added proxy_logs.trace_id, proxy_logs.error_group_id, revenue_events.trace_id, and error_events.trace_id_indexed (denormalized from JSONB for index speed). Error events are already linked to their traces. Request rows will link the same way once the proxy's request-log writer stores the parsed traceparent trace ID, which it doesn't yet, and nothing fills revenue_events.trace_id yet either.
  3. Free to self-host, Temps Cloud coming soon — Temps is dual-licensed MIT / Apache 2.0, and the Observe page ships with every instance on the v0.1.0 beta line.

Temps Unified Observe vs. Alternatives

Temps Unified ObserveDatadog APMSentry + Plausible + Logger
Request logsBuilt in (Pingora proxy)Agent-basedSeparate tool
OTel tracesBuilt in (OTLP ingest)Built inSentry only (no request logs)
Error trackingBuilt in (Sentry-compatible DSN)Built inSentry
Revenue eventsBuilt inNot includedNot included
Merged timelineYes — single viewNo (separate APM / Log Analytics tabs)No
Cross-signal correlationtrace_id on errors and traces; request and revenue rows not linked yetYes (requires full agent)Manual
Self-hostableYes, freeNoSentry OSS only
CostFree self-host; Cloud from $29/mo (coming soon)See datadog.com/pricingThree separate subscriptions

The cockpit: sparklines as filter toggles

The top of the Observe page is a row of four sparklines, one per event kind:

  • Requests (proxy logs) — sky gutter
  • Traces (OTel spans) — violet gutter
  • Errors (error events) — rose gutter
  • Revenue (revenue events) — emerald gutter

Each sparkline plots the rows currently loaded on the page (the latest 100 merged events) in 24 time buckets, and a kind you've switched off reads "off". Click one to toggle that kind on or off. The colors carry through to the timeline below — every row's left gutter matches its sparkline.

Two reasons this works better than a checkbox bar:

  1. You see relative volume at a glance. If errors spiked inside the loaded rows, the rose sparkline shows it. You don't have to filter to errors first to find the spike.
  2. Filtering is one click, not three. The sparkline is the filter. There's no separate "kinds" panel to open.

The stream: console-style monospace

Below the cockpit is the timeline. Monospace font, HH:mm:ss timestamps, color-coded gutters per kind, one-line summaries. It looks like tail -f for your business signal.

Why monospace? Because the dominant pattern in observability tooling is dense rows of structured data, and proportional fonts waste horizontal space. Monospace gives you predictable column widths, which makes scanning fast.

Why one-line summaries? Because the side panel is one click away. Putting full payloads inline turns the timeline into a wall of JSON. Putting them behind a click keeps the timeline scannable.

Click a row and the side panel opens. The panel renders entirely from the row payload — no follow-up fetch in the common case. That's a deliberate choice. Network round-trips are the difference between "feels instant" and "feels laggy," and the merge service ships everything the panel needs in the initial response.


What the merge service actually does

A new crate called temps-observability backs the page. It runs one query per enabled kind, in parallel:

  • requests, read through the same storage backend as the Requests page (TimescaleDB by default, ClickHouse when TEMPS_CLICKHOUSE_* is configured)
  • error_events for errors (Postgres)
  • revenue_events for revenue (Postgres)
  • traces, read through the same OTel storage backend as the Traces page (root spans only, unless you search by name)

Then it k-way merges them by ts DESC and trims to a single page. Each row carries a type discriminator and all the data the panel needs.

Heavy fields get truncated server-side:

  • Stack frames → first 5
  • Span attributes → first 20 keys, alphabetized
  • Headers → whitelist of ~10 keys

Each truncation sets a *_truncated flag, and the panel tells you where the rest lives: a truncated stack trace links to its error group ("Show full stack trace"), truncated span attributes point you to the trace, and truncated headers get a "Truncated" badge. API clients can fetch the untruncated row from the events /full endpoint. The default is fast; the deep-dive is one click away.


Cross-source correlation: the trace_id story

The Observe view's most useful feature isn't the merge — it's jumping from an error to the trace that produced it, and (once request rows carry trace IDs) from a request to its trace. That requires trace_id columns everywhere.

The v0.1.0 betas add them via migration m20260502_000001_add_observe_correlation:

  • proxy_logs.trace_id and proxy_logs.error_group_id (with indexes)
  • revenue_events.deployment_id, environment_id, and trace_id (nothing writes trace_id on revenue rows yet)
  • error_events.trace_id_indexed (denormalized from data.trace.trace_id JSONB for index speed)

All nullable. Old rows just render without correlation links — no backfill required.

Two ingest paths are meant to populate the new columns:

  1. W3C traceparent extraction in the proxy. The proxy has a parser for inbound traceparent headers that validates the 32-hex trace_id (length, hex chars, rejects all-zero). The batched request-log writer doesn't store that value in proxy_logs.trace_id yet, so request rows don't carry trace links today (details).
  2. OTel trace_id promotion in error tracking. Sentry-compatible ingest probes data.sentry.contexts.trace.trace_id, top-level data.contexts.trace.trace_id, and data.trace.trace_id, and promotes the first valid 32-hex value to error_events.trace_id_indexed.

Both parsers have unit tests for the happy path, missing/malformed inputs, case-folding, and the all-zero invalid trace_id reserved by the W3C spec.


Three deliberate choices worth defending

Traces default OFF. OTel spans are a high-volume hypertable. If we forced them on by default, anyone with non-trivial OTel ingest would see a timeline dominated by trace rows and miss the business signal. You opt in via the cockpit card. We'd rather you find errors than drown in spans.

Runtime logs are not in this view. Container stdout/stderr volume would absolutely dominate everything else. They live on the dedicated Logs page, where the viewer is designed for high-volume streaming with batching and pause/resume.

Filter state lives in URL search params. The enabled kinds, time range (15 minutes to 30 days), search text, environment and bot filter are all in the URL, so you can send a teammate a link to "errors + revenue, last 24 hours, search checkout" and they see the same filtered view. Permalinks for incidents. Filter changes replace the current URL rather than adding history entries, and the timeline refreshes every 15 seconds.


What this is not

Honest scope: the Observe page is for apps deployed on Temps, by teams who want requests, traces, errors and revenue in one tool.

It's not a replacement for:

  • Datadog APM — no continuous profiler, no flame graphs at code level
  • Datadog log analytics — searchable by level + timeframe + free text, no DSL
  • Splunk / Elastic — same caveat, plus no anomaly detection or ML
  • PagerDuty / Opsgenie — no on-call rotation, no alert routing
  • k8s metrics platforms — Temps assumes Docker, not Kubernetes

If you're a 1,000-engineer org running thousands of services across multiple clouds, you probably already have Datadog and a dedicated observability team. Don't switch.

If you're a 1–50 person team and currently piecing together Sentry + Plausible + Pingdom + a request-logger you wrote yourself, Temps — the Observe page plus its built-in analytics and uptime monitoring — is the kind of tool that replaces the duct tape with a single coherent surface.


How to try it

The Observe page is available for every project on the v0.1.0 beta line. Upgrade your instance to a v0.1.0 beta (for example temps upgrade --version v0.1.0-beta.56, since bare temps upgrade only looks for stable releases) and open Observe → Activity in the project sidebar (/projects/:slug/observe). The Observe docs cover the filters and side panel, and the observability overview shows how traces, logs and metrics arrive.

To get correlated trace IDs on your error events, include trace context in the payloads your SDK sends; most OTel-instrumented clients also send a W3C traceparent header on requests, which request rows will use once the proxy stores it. The OpenTelemetry tracing guide covers instrumenting your services.


FAQ

How do you get a unified observability view without multiple SaaS tools?

Run Temps — it stores requests (via its Pingora proxy), OTel traces (OTLP ingest endpoint), errors (Sentry-compatible DSN), and revenue events natively. The built-in Observe page merges all four into a single timeline using the temps-observability crate. No exporters to configure, no per-signal SaaS subscriptions required.

Does the Observe page work without OTel?

Yes. Requests show up as soon as the proxy serves traffic, errors once an SDK reports to the project's DSN, and revenue events once you connect Stripe (revenue tracking is labelled Experimental) — none of that needs OTel. Traces only appear if you've configured OTel ingest. Error rows link to their trace whenever the error payload carries a trace_id.

Can I add custom event kinds?

Not in the v0.1.0 betas. The four kinds are hardcoded into the merge service. Future releases may add a generic event-kind extension API.

What's the performance overhead on the database?

Each enabled kind runs one query bounded by its time column (timestamp, occurred_at or start_time), the page size and a 30-day maximum window, and server-side truncation keeps payloads small, so a page load costs a few bounded scans rather than a full-table merge. Traces are off by default, so a default load runs three queries and keeps the heaviest table out until you ask for it.

Why not include runtime logs?

Volume. A chatty container can produce 1,500 lines per second. Mixing that with business signal would make the timeline unusable. The dedicated Logs page handles container logs with proper batching, pause/resume, and an interval mode that closes the WebSocket while you read.

Is Temps free to self-host?

Yes. Temps is dual-licensed MIT / Apache 2.0 and free to self-host — you only pay for the server you already run it on. Temps Cloud, a hosted control plane for your own servers, is coming soon from $29/month.

Related: Read the v0.1.0 beta release notes · Observe docs

#unified observability#opentelemetry#distributed tracing#error tracking#request logs#revenue analytics