What Is Observability? Logs, Metrics and Traces Explained

Observability is the ability to understand what's happening inside a running system by examining the data it produces — genuinely different from traditional monitoring, which typically answers pre-defined questions you thought to ask in advance ("is CPU usage above 80%?"). Observability is built specifically for the harder, more realistic scenario: something is wrong, you don't yet know why, and you need to explore the system's actual behavior to find out — a capability that depends on having rich, well-structured data available, not just a handful of pre-built dashboards covering the failures someone anticipated ahead of time.

The three pillars — logs, metrics, traces

• Logs — discrete, timestamped records of specific events ("user 4471 login failed: invalid password", "payment processing started for order #8821") — the most granular and detailed pillar, but also the most expensive to store and search at scale if not managed carefully.
• Metrics — numerical measurements aggregated over time (requests per second, average response latency, error rate, memory usage) — efficient to store and graph, excellent for spotting trends and setting alerting thresholds, but they summarize rather than explain individual events.
• Traces — a record of a single request's full journey through a distributed system, showing exactly which services it touched, in what order, and how long each step took — essential for understanding where time is actually being spent in a request that involves multiple services.

Why one pillar alone isn't enough

Metrics can tell you *that* p99 latency spiked at 2:14pm, but not *why* — for that, you need to drill into traces from around that time to see which specific requests were slow and which service in the chain caused the slowdown, then cross-reference with logs from that specific service to see the actual error or condition that caused it. This is the genuine, practical value of having all three pillars correlated together (often via a shared trace ID or request ID that appears in logs, metrics tags, and trace spans alike) rather than three separate, disconnected tools each showing a different slice of the same underlying system — the correlation between pillars is where real debugging speed comes from, not any single pillar in isolation.

Distributed tracing — why it matters more as architecture grows

In a simple, single-server application, a slow request is relatively easy to diagnose — there's only one place to look. In a microservices architecture (see our CI/CD and container-orchestration explainers for related infrastructure context), a single user-facing request might touch a dozen internal services — an API gateway, an auth service, three or four backend services, a database, a cache, a message queue — and without distributed tracing, figuring out which of those dozen hops actually caused a slow response means manually correlating logs across every service by timestamp, a genuinely painful and error-prone process at any real scale. A trace ID propagated through every hop turns this into a single, visual waterfall showing exactly where time was spent across the entire request's journey.

Structured logging — a prerequisite most teams get wrong initially

Plain-text log lines ("User login failed for john@example.com") are readable by a human scanning a terminal, but genuinely hard to search, filter and aggregate at scale — structured logging (emitting logs as key-value data, typically JSON, like `{event: "login_failed", user: "john@example.com", reason: "invalid_password", timestamp: ...}`) makes logs machine-queryable, letting you filter for exactly `event=login_failed AND reason=invalid_password` across millions of log lines instead of relying on fragile text pattern matching. Teams that start with unstructured, freeform log messages typically face a genuinely painful migration later once log volume grows past what manual reading can handle — starting with structured logging from day one, even for a small system, avoids this retrofit.

Cardinality — the hidden cost that catches teams by surprise

A metric's "cardinality" refers to how many distinct label/tag combinations it can produce — a metric tagged by `user_id` on a system with millions of users has extremely high cardinality, and most metrics systems (which are optimized for aggregating a bounded, manageable number of distinct time series) genuinely struggle or become prohibitively expensive at very high cardinality. This is a common, expensive mistake for teams new to observability: tagging a metric with a high-cardinality dimension (user ID, request ID, a raw timestamp) that seemed like it would be useful for filtering, only to discover it multiplies storage costs and query time dramatically — high-cardinality data generally belongs in logs or traces, which are built to handle it, not in metrics, which generally aren't.

Alerting — turning observability data into actionable signals

Observability data is only useful operationally if it actually surfaces problems to a human (or an automated response) at the right time — alerting rules defined on top of metrics (error rate exceeding a threshold, latency crossing a percentile bound, a queue depth growing unusually) are what turn passive data collection into an active early-warning system. A genuinely well-tuned alerting setup avoids two opposite failure modes: alert fatigue from too many low-signal alerts that get ignored, and missed incidents from thresholds set too loosely to catch a real, developing problem before it affects users significantly.

Sampling — the trade-off for tracing at scale

Capturing a full trace for every single request becomes prohibitively expensive in both storage and performance overhead at high request volumes, which is why most production tracing systems use sampling — capturing detailed traces for only a percentage of requests (or specifically for requests that error, exceed a latency threshold, or match other interesting criteria) rather than every single one. This is a genuine, deliberate trade-off: lower overhead and cost, at the cost of potentially not having a full trace for the exact specific request a user reports as broken — a common mitigation is "tail-based sampling," which decides whether to keep a trace only after seeing how the request actually turned out (keeping all error traces, for instance, while sampling normal ones more sparsely).

Step-by-step: building observability into a new service

1. Start with structured logging from the beginning — emit logs as key-value/JSON data with consistent field names across services.
2. Instrument key metrics — request rate, error rate, latency percentiles (not just averages, which hide outliers) for every meaningful operation.
3. Add distributed tracing with a propagated trace ID through every service hop, even before the system has many services — retrofitting this later is harder.
4. Correlate the three pillars by including the same trace/request ID in logs, metric tags where feasible, and trace spans.
5. Define alerting thresholds based on what actually affects users (error rate, latency percentiles) rather than raw infrastructure metrics alone (CPU, memory) that don't always correlate with real user impact.
6. Review and prune noisy or unused dashboards and alerts periodically — observability tooling accumulates cruft over time just like any other part of a codebase.

Common mistakes with observability

• Treating logs, metrics and traces as separate, disconnected tools instead of correlating them via a shared identifier.
• Tagging high-cardinality dimensions (user ID, raw timestamps) on metrics, causing storage costs and query performance to degrade badly.
• Starting with unstructured, freeform log messages and facing a painful migration once log volume grows past what manual review can handle.
• Alerting on raw infrastructure metrics alone without connecting them to actual user-facing impact, causing either alert fatigue or missed real incidents.
• Adding distributed tracing only after a system has grown complex enough that retrofitting trace-ID propagation across every existing service becomes a significant project.

OpenTelemetry — the move toward a shared, vendor-neutral standard

Historically, adopting observability tooling meant instrumenting an application against a specific vendor's proprietary SDK — genuinely useful, but creating real vendor lock-in (see our vendor lock-in explainer for the broader pattern), since switching observability vendors later meant re-instrumenting the entire codebase against the new vendor's SDK. OpenTelemetry has emerged as a widely-adopted, vendor-neutral open standard for instrumenting applications to emit logs, metrics and traces in a common format, which any compatible backend (whether an open-source tool or a commercial vendor) can then ingest — instrumenting against OpenTelemetry once, rather than a specific vendor's proprietary SDK, preserves meaningfully more flexibility to change observability backends later without a full re-instrumentation effort.

The cost of observability itself — a genuine budget line item

Storing, indexing and querying large volumes of logs, metrics and especially high-cardinality trace data at scale is a genuine, often underestimated cost — for a system generating meaningful traffic, observability infrastructure costs can become a non-trivial fraction of overall infrastructure spend if retention periods, sampling rates and cardinality aren't deliberately managed. This isn't a reason to under-invest in observability (the operational blindness of inadequate observability carries its own very real cost, usually paid during an incident when it's most expensive), but it is a reason to treat observability configuration — what gets logged at what verbosity, how long data is retained, what gets sampled — as a genuine, ongoing engineering and cost-management decision rather than a "log everything forever" default that quietly becomes expensive.

Observability during an actual incident — how the pillars get used together

A typical incident investigation flow illustrates why all three pillars matter together: an alert fires based on a metric threshold (elevated error rate), the responder pulls up a dashboard to see which service or endpoint is affected and roughly when it started, then drills into traces from that time window and affected endpoint to identify a specific pattern (a particular downstream dependency consistently slow or failing), and finally checks structured logs from that specific service around that specific time to find the actual underlying error or exception causing it. Each pillar narrows the investigation from a broad signal (something's wrong) to a specific, actionable root cause — which is exactly the workflow well-correlated observability data is designed to support, and exactly what's missing when logs, metrics and traces exist as disconnected tools that can't be cross-referenced by a shared identifier.

Service Level Objectives (SLOs) — connecting observability to what users actually experience

A Service Level Objective is a specific, measurable target for a service's reliability from the user's perspective ("99.9% of requests complete in under 300ms over a rolling 30-day window"), defined using the same metrics an observability system already collects — the practical value of an SLO is that it turns raw observability data into a concrete, agreed-upon target the team can alert against and make real trade-off decisions relative to, rather than treating every individual metric fluctuation as equally important. A team with a defined SLO and an "error budget" (how much unreliability is acceptable within the SLO before it's breached) can make genuinely informed decisions about whether to prioritize a risky deployment or additional feature work against the remaining error budget for that period — a decision that's much harder to make coherently without observability data organized around a specific, meaningful reliability target in the first place. Without an SLO, teams often default to treating every alert and every metric fluctuation as equally urgent, which is precisely the alert-fatigue problem a well-defined reliability target and error budget are designed to prevent. Defining even a small handful of genuinely meaningful SLOs for a system's most critical user journeys, rather than attempting to formalize every possible metric into a target, is usually a more practical starting point than trying to build a comprehensive SLO framework across an entire system all at once — the framework can always expand to cover more of the system once the initial, highest-value SLOs are proving genuinely useful in practice, rather than stalling out under the weight of trying to define everything comprehensively before any of it delivers value — a single well-chosen SLO on a system's most critical path is worth more in practice than a dozen poorly-considered ones spread thin.