How to Debug Production Issues in GCP: Metrics, Logs, Traces, and Rollbacks

Simple explanation

Each signal type answers a different question about a production incident:

• Metrics tell you when and where something changed: error rate spiked, latency increased, request volume dropped.
• Logs tell you what error happened: the message, stack trace, and request context for each individual failure.
• Traces tell you which operation in the request path was slow or failing. In distributed systems, the service that returns an error is often not where the error originated. Each traced operation is called a span.
• Profiler helps when the issue is CPU, memory, or lock contention over time. It shows where your code spends its runtime resources, not just when something failed.

Each signal costs time to search. Use them in order, from coarse to fine, so each step narrows the next one.

When to use this guide

This page is the right starting point when you are dealing with:

• A sudden spike in error rate or 5xx responses
• Rising p99 latency (the slowest 1% of requests) with no obvious cause
• Partial outages where some requests succeed and some fail
• Post-deploy regressions where a service behaved correctly before a recent change
• Intermittent failures that are hard to reproduce locally

If you already know which service is failing and what kind of failure it is, jump directly to the relevant troubleshooting page: Cloud Run container failed to start, GKE CrashLoopBackOff, Cloud SQL connection refused, or Cloud Functions failures.

Metrics vs Logs vs Traces vs Profiler

How the debugging workflow works

Production debugging is an operational decision tree. Each step narrows the search space for the next one.

1. Start with metrics to establish the blast radius (which services are affected, which regions, how many users) and the precise start time. Open your service dashboard and look at the last 2 hours.
2. Check what changed immediately after identifying the start time. Most sudden production incidents are caused by a recent deployment or configuration update. Check Cloud Audit Logs before spending time anywhere else.
3. Use logs to find the error pattern. With the time window from step 1, your log filter is now specific. Search for severity>=ERROR in the affected service during the incident window. Read the full log entry: stack traces, request IDs, and dependency errors are usually in structured log fields alongside the message.
4. Use traces to locate the failing hop when logs show an error but the root cause is in a different service. A span is a single operation within a distributed request. The trace shows all spans in sequence so you can see which one failed or took the most time.
5. Use profiler only when the issue points to resource behavior. If metrics show high CPU or memory and logs show no errors, Cloud Profiler reveals which function or code path is consuming the most resources over time.
6. Mitigate first, diagnose second. If a rollback is available and the incident is ongoing, roll back before spending time diagnosing. Diagnose the root cause once the service has recovered.

Step-by-step production debugging workflow

Step 1: Confirm what changed in metrics

Open Cloud Monitoring and check request rate, error rate, and latency for the affected service over the last 2 hours. Identify the exact minute the behavior changed. Note whether the change is sudden (deployment, config push) or gradual (resource exhaustion, traffic growth).

If you have alerting policies configured, they will tell you which metric threshold was breached and when. If you do not have alerts yet, this incident is a practical reason to add them after the postmortem.

Step 2: Check what changed in the environment

The most common cause of a sudden production incident is a recent deployment or configuration change. Before opening logs or traces, spend two minutes here.

In Logs Explorer, filter Cloud Audit Logs for admin activity. This filter shows all API-level changes across your project:

log_id("cloudaudit.googleapis.com/activity")

To narrow to Cloud Run deployments specifically:

log_id("cloudaudit.googleapis.com/activity")
protoPayload.serviceName="run.googleapis.com"
protoPayload.methodName="google.cloud.run.v1.Services.ReplaceService"

The protoPayload.authenticationInfo.principalEmail field shows who made each change. If the deployment timestamp aligns with the incident start time, roll back to the previous revision and confirm the service recovers before continuing your investigation.

Step 3: Narrow with logs

Move to Logs Explorer. Filter to the affected service and the incident time window. Start with severity>=ERROR to find the highest-priority signals first.

# Pull recent error logs from the command line
gcloud logging read \
  'severity>=ERROR resource.type="cloud_run_revision" resource.labels.service_name="api-service"' \
  --limit=50 \
  --format='value(timestamp,jsonPayload.message)' \
  --project=my-app-prod

# Pull all logs from the last 10 minutes
gcloud logging read \
  'resource.type="cloud_run_revision" resource.labels.service_name="api-service"' \
  --freshness=10m \
  --limit=100 \
  --project=my-app-prod

# Pull logs and pipe to jq for field extraction
gcloud logging read \
  'severity>=ERROR resource.type="cloud_run_revision"' \
  --limit=20 \
  --format=json \
  --project=my-app-prod | jq '.[].jsonPayload.message'

Read the full log entry, not just the message. Structured logs carry request IDs, user IDs, upstream service names, and error codes as separate queryable fields. These let you correlate errors across services and identify which specific request triggered the failure.

Step 4: Confirm the failing operation with traces

Open Cloud Trace and filter to the affected time window. Sort by latency or filter for traces with errors. Click a failing or slow trace to open the waterfall view.

In a microservices architecture, the service that returns an error to the user is often not where the error originated. The trace shows the full call chain. Each span represents one operation: an HTTP call, a database query, a Pub/Sub publish. Find the span that failed or consumed most of the time, then go directly to that service’s logs.

For more on how distributed tracing works across services, see Distributed Tracing.

Step 5: Escalate to profiler when the issue is resource behavior

If metrics show elevated CPU or memory, logs show no clear error, and traces show slow requests without an obvious failing span, the issue may be runtime resource behavior: a hot loop, a memory leak, or lock contention.

Cloud Profiler collects continuous, low-overhead profiling data from running services. Open Profiler in the Cloud Console, select the service and time range, and look for functions that consume a disproportionate share of CPU or memory. This is the right tool when the problem is a resource consumption pattern rather than a logic error.

Step 6: Mitigate first, diagnose second

During an active incident, recovery takes priority over root cause analysis. If a rollback is available: roll back, confirm the service is healthy, then investigate the failed revision in a non-production environment. If rollback is not available, consider routing traffic away from the affected region or instance first.

Common GCP production debugging scenarios

Error spike after a deployment

Open Cloud Audit Logs and filter for changes at the incident start time. For Cloud Run, filter for protoPayload.methodName=“google.cloud.run.v1.Services.ReplaceService”. If the timestamp matches, roll back immediately without further investigation. See Monitoring Cloud Run for revision traffic controls and error metrics.

Slow database-backed requests

Open Cloud Trace and filter for slow requests above your SLO threshold. Click a slow trace and look for a database span consuming most of the elapsed time. For Cloud SQL, cross-check with Query Insights for per-query execution time. See Cloud SQL Connection Refused if requests are failing rather than just slow.

Cloud Run 500s with no application error in logs

Search Logs Explorer for container-level events including OOM kills and startup failures:

resource.type="cloud_run_revision"
resource.labels.service_name="api-service"
"OutOfMemory"

Also check Cloud Monitoring for container/memory/utilization approaching 1.0 before the errors started. If memory is the cause, increase the limit or investigate the leak with Cloud Profiler. See Cloud Run container failed to start for the full startup failure checklist.

GKE pod crashes and CrashLoopBackOff

What you see: pods restart repeatedly, kubectl get pods shows CrashLoopBackOff, application traffic is intermittently disrupted.

Check GKE monitoring dashboards for pod restart count and container exit codes. Then pull logs from the last crash with kubectl logs pod-name —previous. See GKE CrashLoopBackOff Explained for the full diagnostic process and common root causes.

Cloud Functions runtime failures

What you see: invocations fail, Cloud Monitoring shows increased error count, but the failure rate is not 100% so the function is clearly running sometimes.

Filter Logs Explorer to resource.type=“cloud_function” and severity>=ERROR. Look for timeout errors, memory limit exceeded messages, and dependency import failures. Check whether execution time is approaching the configured timeout. See Debugging Cloud Functions Failures for a scenario-by-scenario breakdown.

Cloud SQL connection failures

What you see: the application returns database connection errors, the Cloud SQL instance shows as running, and the issue appeared after a deployment or restart.

See Cloud SQL Connection Refused for the most common root causes and fixes.

What replaced Cloud Debugger?

Cloud Debugger allowed attaching a snapshot debugger to a running production application without pausing it. Modern approaches cover the same ground more reliably:

• Structured logging: emit enough context in each log entry (request ID, user ID, relevant application state) to reconstruct what happened without a live debugger. Add spans with attributes to capture state at key decision points. This is the primary replacement for breakpoint-style debugging in production.
• Cloud Trace: trace the full path of a specific request and inspect the state captured in each span. Useful when the problem is latency or a failing downstream call.
• Cloud Profiler: continuous profiling for CPU, memory, and lock contention. Useful when the problem is runtime resource behavior rather than a logic error.
• Cloud Workstations: run a full development environment in GCP with VPC access. Debug against a staging environment that mirrors production rather than attaching to production directly.
• Feature flags and targeted diagnostic logging: enable verbose logging for a specific user or request subset in production without a full redeploy. Combine with structured logs to capture additional context on demand.