Distributed Tracing in GCP: Traces, Spans, and Context Propagation

Simple explanation

Each service your request touches creates a span. All those spans share one trace ID. The result is a complete picture of where the request went and how long it spent at each stop, rendered as a waterfall diagram where you can see the bottleneck at a glance.

Why distributed tracing matters

Average latency metrics hide problems. If 99% of requests complete in 200ms but 1% take 10 seconds, your average looks healthy. When a slow request happens, logs from ten different services tell you what each one did, but not how they interacted for that specific request.

Distributed tracing solves both problems. It preserves the causal structure of a request: Service A called Service B, which called a database, which took 3.1 seconds. When you are working with microservices on GCP, tracing is often the only reliable way to answer “which service made this request slow?”

It also makes service dependencies visible. You can see which services a request depends on, how those dependencies compose, and what happens when one of them slows down or fails.

How distributed tracing works

The structure of a trace

Every trace starts when a request enters your system. The first service creates the root span. Each downstream service call creates a child span attached to its parent. Spans form a tree that mirrors the call graph of the request.

Here is what that looks like for the checkout example:

[ROOT] POST /checkout — api-gateway (4200ms)
  ├── [CHILD] process-order — order-service (3800ms)
  │     ├── [CHILD] check-inventory — inventory-service (3200ms)
  │     │     └── [CHILD] SELECT * FROM inventory — postgres (3100ms)  ← bottleneck
  │     └── [CHILD] reserve-items — inventory-service (120ms)
  └── [CHILD] log-audit — audit-service (90ms)

The Postgres query consumes 3100ms of a 4200ms request. That is your bottleneck. Without tracing, you would only know the total was slow.

What a span contains

Each span records:

• Trace ID: shared by every span in the request, a 128-bit random identifier
• Span ID: unique identifier for this span within the trace
• Parent span ID: the span that triggered this work (empty on the root span)
• Start time and end time: used to calculate duration and render the waterfall
• Attributes: key-value metadata such as HTTP method, status code, database query, order ID, user ID
• Status: OK, ERROR, or UNSET

Attributes are where the diagnostic value lives. A span that records http.method=POST, http.status_code=500, and db.statement=SELECT… tells you far more than one that only records duration.

The waterfall view

Cloud Trace renders traces as horizontal bars on a timeline. Each bar is one span. Nested bars show the parent-child relationship. You can immediately see which span is blocking the others.

Trace context and propagation

For a trace to connect across services, the trace ID and parent span ID must travel with every outgoing request. This is called context propagation, and it happens via HTTP headers.

The current standard is W3C Trace Context, which defines two headers:

• traceparent: carries the trace ID, parent span ID, and a flags byte. Format: 00-TRACE_ID-PARENT_SPAN_ID-FLAGS. Example: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
• tracestate: carries optional vendor-specific metadata. Usually empty.

When the order service calls the inventory service, it reads its current span’s IDs, formats the traceparent header, and includes it in the outgoing HTTP request. The inventory service reads that header, creates a new child span with the correct parent, and continues the trace.

If you use OpenTelemetry, the SDK handles propagation automatically through instrumented HTTP clients. You do not construct the headers manually.

How this works in Google Cloud

When you implement distributed tracing in GCP, three layers work together:

• Distributed tracing: the practice and concept, tracking requests across services by creating spans and connecting them into traces
• OpenTelemetry: the vendor-neutral instrumentation toolkit, SDKs for your application code that create spans, propagate context, and export data to a backend
• Cloud Trace: the GCP backend, stores span data, renders waterfall views, and surfaces latency outliers

Some Google Cloud services, including Cloud Run and GKE, can propagate trace context automatically at the platform level. This means a trace ID generated upstream can flow to downstream services without code changes.

That automatic propagation covers the routing of context. It does not create spans for your internal operations. Custom instrumentation with OpenTelemetry is still needed when you want spans for individual database queries, cache lookups, or external API calls, business-level attributes like user ID or order amount, and accurate visibility into code paths within a service rather than just between services.

Sampling strategies

Tracing every request in a high-traffic production system is expensive. Most tracing systems use sampling to record only a fraction of requests.

Common strategies:

• Head-based sampling: the decision is made at the root span and travels with the traceparent flags. Simple and low-overhead, but you cannot target slow or error requests specifically.
• Tail-based sampling: the decision is made after the entire trace is assembled, so you can capture 100% of slow or error traces while keeping only a small fraction of successful ones. More useful in production since you never lose a slow request because the sampler dropped it at the start.
• Fixed-rate sampling: sample a fixed percentage regardless of outcome. A 5% rate means roughly 1 in 20 requests is recorded.

When to use distributed tracing

Tracing is most useful when:

• A request is slow and you do not know which service is responsible. The waterfall view tells you immediately.
• A request fans out across multiple services. Monitoring Cloud Run or Monitoring GKE covers aggregate health. Tracing covers individual requests.
• You need to understand service dependency chains. Which service calls which, and how does latency compound?
• You are debugging an incident and logs alone are not enough. Link traces to your structured log entries using the trace ID and you can jump from a slow span directly to the logs it generated.
• You are working with event-driven systems. Propagate trace context through message headers to trace async flows across services and queues.

Tracing is less useful for simple single-service applications where logs or a profiler are sufficient, aggregate trend analysis (use metrics for that), and high-volume batch jobs where per-request tracing cost is prohibitive.

Distributed tracing vs logs vs metrics vs profiling

You get paged at 2am. Checkout is slow. Here is what each observability signal tells you:

More concisely:

• Metrics: aggregate trends and patterns over time. Best for alerting, capacity planning, and dashboards.
• Logs: detailed events and state within a service. Best for debugging what happened inside one service. Start in Logs Explorer.
• Traces: request path and latency across services. Best for debugging slow or broken cross-service requests.
• Profiling: CPU and memory usage inside a service at the code level. Best for finding hot spots in one process. See Cloud Profiler.

For a workflow that pulls all four signals together during a live incident, see Debugging Production Systems.