What Is Google Cloud Trace? Spans, Waterfalls, and Latency

The simple explanation

Cloud Trace is a storage and visualization layer for distributed traces in GCP. When a user makes a request to your application, that request touches multiple services: a load balancer, a Cloud Run instance, a database, maybe an external API. Cloud Trace records how long each step took, assembles those measurements into a timeline, and stores it so you can query it later.

The result is a waterfall view: a horizontal bar chart where each bar is one step (called a span), placed on a shared time axis. Wide bars are slow steps. That is the core insight Cloud Trace provides.

What Cloud Trace does

Cloud Trace gives you three things:

• A searchable history of traces. Filter by endpoint, status code, latency range, or time window to find the specific slow request you are trying to diagnose.
• A waterfall view for each trace. Shows every span as a bar on a shared timeline so you can see parent-child relationships and identify the slow step at a glance.
• Latency distribution analysis. Aggregates latency across many traces to show p50, p95, and p99 percentiles and how those change over time. This is how you catch regressions before they escalate.

Cloud Trace does not replace Cloud Monitoring for dashboards and alerting, or Logs Explorer for raw event details. It works alongside them. The three tools answer different questions about the same system.

When to use Cloud Trace

Reach for Cloud Trace when you have a latency problem and need to find where time is going:

• A Cloud Run endpoint that normally responds in 200ms is now taking 2 seconds
• A GKE microservice chain where one downstream dependency is slow but you do not know which one
• Intermittent latency spikes that are invisible in averages but visible in your p99 metric
• A deploy that went out yesterday and latency has been slowly creeping up since
• You want to know whether the slow step is in application code, a database query, a cache miss, or an external API call

When to reach for a different tool first

Cloud Trace is the right tool for latency questions. For other questions, start elsewhere:

• For infrastructure health, uptime, and alerting: Cloud Monitoring
• For detailed event investigation inside a service: Cloud Logging
• For CPU or memory hotspots inside your application code: Cloud Profiler

How Cloud Trace works

Cloud Trace is built around two concepts: traces and spans.

Traces and spans

A trace represents the complete journey of one request through your system. Every service that touches the request shares the same trace ID, a globally unique identifier that ties all the pieces together.

A span is one unit of work within that journey: one service handling part of the request, one database query, one external API call. Each span records a start time, an end time, and a set of attributes (key-value metadata like HTTP method, URL, status code, database query text, or order ID).

Spans form a parent-child tree. When Service A calls Service B, the span for Service B is a child of the span for Service A. The waterfall view renders this tree as indented horizontal bars on a shared timeline.

Context propagation

For spans from different services to join the same trace, the trace ID must travel with the request. This is done via HTTP headers. The standard is W3C Trace Context: the traceparent header carries the trace ID and the parent span ID. Each service reads the incoming header, creates its own child span, and passes an updated header with any outbound calls it makes.

When you use OpenTelemetry with an instrumented HTTP client, this propagation is handled automatically. For a deeper look at the underlying concepts, sampling strategies, and how context propagation works across service boundaries, see Distributed Tracing.

How traces reach Cloud Trace

Traces reach Cloud Trace through several paths, from most recommended to most low-level:

OpenTelemetry with OTLP (recommended for new setups)

OpenTelemetry is the vendor-neutral open standard for distributed tracing. Instrument your code with the OTel SDK, then export spans using the OTLP exporter. For GCP, you can route spans through an OpenTelemetry Collector configured with the googlecloud exporter, which handles authentication and delivery to Cloud Trace. This approach keeps your application code fully portable: if you switch backends or run locally with a Jaeger collector, you change the collector configuration, not your application code.

OpenTelemetry with the Cloud Trace exporter (direct GCP option)

For simpler GCP-only setups, the CloudTraceSpanExporter (available for Python, Go, Java, and Node.js) sends spans directly to Cloud Trace without needing a collector. This is a valid, working option for applications that run entirely on GCP. It trades portability for simplicity.

Automatic collection from GCP services

Cloud Run, GKE, App Engine, Cloud Functions, and the Cloud Load Balancer automatically emit traces for incoming requests. No instrumentation required for this baseline data. These automatic traces show you request boundaries but not what happens inside your application code, which is why adding application-level spans matters.

Cloud Trace API (low-level)

You can write trace data directly using the Cloud Trace REST or gRPC API. This is the most GCP-specific option and ties your instrumentation to GCP. For new projects, the OTel path is the better starting point.

A simple request flow example

Here is what a single checkout request looks like as a trace. The user submits a form:

[ROOT]  POST /checkout  (load balancer)  1850ms
  └── [CHILD]  POST /checkout  (Cloud Run: api-service)  1820ms
        ├── [CHILD]  check-inventory  (Cloud Run: inventory-service)  1400ms
        │     └── [CHILD]  SELECT * FROM inventory  (Cloud SQL)  1350ms  ← bottleneck
        ├── [CHILD]  charge-card  (external payment API)  310ms
        └── [CHILD]  send-confirmation  (Pub/Sub publish)  40ms

Each row is a span. The load balancer span is the root. The Cloud Run spans are children. The Cloud SQL query is a grandchild of the inventory service and takes 1350ms of a 1850ms total. That is where you look next. Not by guessing. The waterfall made it obvious.

Without tracing, you would see a slow endpoint in your Cloud Monitoring dashboard and have to work backwards through logs from each service. With tracing, the bottleneck is visible at a glance.

Instrumenting a Python application

The setup below uses the OpenTelemetry SDK with the Cloud Trace exporter, a direct path that works without a collector and is a practical starting point for GCP-only applications. Install the required packages:

pip install opentelemetry-sdk \
            opentelemetry-exporter-gcp-trace \
            opentelemetry-instrumentation-flask

Configure the tracer provider at application startup:

from opentelemetry import trace
from opentelemetry.exporter.cloud_trace import CloudTraceSpanExporter
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

provider = TracerProvider()
provider.add_span_processor(
    BatchSpanProcessor(CloudTraceSpanExporter())
)
trace.set_tracer_provider(provider)

tracer = trace.get_tracer(__name__)

# Create a custom span for a business operation
with tracer.start_as_current_span("process-order") as span:
    span.set_attribute("order.id", order_id)
    span.set_attribute("order.total_usd", order_total)
    # your business logic here

The BatchSpanProcessor buffers spans and exports them asynchronously, so your request handler is not blocked on the network call to Cloud Trace. The same OTel API works in Go, Java, and Node.js; only the exporter package changes. For Flask or FastAPI, the auto-instrumentation library (opentelemetry-instrumentation-flask) wraps HTTP handlers in spans automatically.

Reading the Cloud Trace console

Trace list

A table of recent traces sorted by time or latency. Each row shows the root span’s HTTP method, URL, total latency, span count, and status. Filter by URL pattern, status code, latency range, or time window. This is your starting point: find a representative slow request, then click into it.

Waterfall view

The waterfall renders the span tree as horizontal bars on a shared time axis. The x-axis is milliseconds from the start of the request. Each bar is one span. Child spans appear indented below their parent.

How to read it: look for the widest bars. A wide bar on a database query span means the database is the bottleneck. A gap between the end of a parent span and the start of its first child means time spent on something not captured as a span (network overhead, middleware, or uninstrumented code). The attributes panel on the right shows the HTTP method, URL, status code, database query, and any custom attributes you added.

Latency distribution

The latency distribution view aggregates data across many traces for a specific endpoint and shows a histogram: how many requests fell into each latency bucket. This reveals the shape of your latency, not just the average. A bimodal distribution (two humps) often means two distinct code paths with very different performance characteristics, which averages completely hide.

Comparing time periods and spotting regressions

Cloud Trace lets you compare latency distributions between two time windows: before and after a deploy, for example. If p99 moved from 800ms to 2 seconds after Wednesday’s release, the comparison view makes that shift immediately visible. The auto-analysis feature goes further: it scans a sample of traces and surfaces which spans are consistently slow across requests, instead of requiring you to click through individual traces manually.

How to know it is working

After deploying instrumentation, confirm each of these before relying on the data:

• Traces appear in the Cloud Trace list for your project
• A root span is visible for each incoming request
• Child spans are present for database calls, cache lookups, or external API calls you instrumented
• Span attributes (HTTP method, URL, status code) are populated
• If you configured log correlation, a Logs tab appears within individual traces in the console

If traces are not appearing at all, check that the service account running your application has the cloudtrace.agent IAM role, and that the GOOGLE_CLOUD_PROJECT environment variable is set correctly, especially for local development runs.

Jumping between traces and logs

Cloud Trace and Cloud Logging integrate when your structured logs include the trace correlation field. Set logging.googleapis.com/trace to projects/PROJECT_ID/traces/TRACE_ID in your structured JSON log output. Once linked, the Cloud Trace console shows a Logs tab within any trace that contains the log entries emitted during that specific request.

The reverse also works: in Logs Explorer, log entries with a trace link show a trace icon. Clicking it jumps directly to the trace in Cloud Trace. You find the error in the logs, then immediately see the full request timeline that produced it. This cross-signal jump is one of the most practical things in the GCP observability suite. For how all these tools fit into a structured incident investigation, see Debugging Production Systems.

Cloud Trace vs Cloud Monitoring vs Cloud Logging vs Cloud Profiler

Each tool answers a different question. Use them together, not as alternatives:

A common workflow: Cloud Monitoring fires an alert on elevated p99 latency, Cloud Trace pinpoints the slow span, Cloud Logging shows the error details for that request, and Cloud Profiler shows which function inside that slow span is consuming CPU. See also Monitoring Cloud Run and Monitoring GKE for platform-specific signal combinations.