How to Create Alerts in Google Cloud Monitoring (GCP)

Simple explanation

An alerting policy has three parts:

• Condition: which metric to watch, how to aggregate it, and what threshold triggers the alert
• Duration (retest window): how long the condition must be continuously true before an incident is created. This is what separates a brief spike from a real problem.
• Notification channel: where to send the alert. Options include email, Slack, PagerDuty, Pub/Sub, and webhook.

When the condition is met for the full duration, Cloud Monitoring creates an incident and sends a notification. When the condition clears, the incident closes automatically and optionally sends a second notification so your team knows the problem has resolved.

Why alerts matter

Dashboards tell you what is happening when you are looking. Alerts tell you what is happening when you are not. Without alerts, you learn about incidents from user reports. With well-configured alerts, you know about degradation before most users are affected, and you have the metric data and runbook link right in the notification to start investigating immediately.

The challenge is calibration. Too many alerts, especially noisy ones that fire from brief spikes, train people to ignore them. Effective alerting is specific, actionable, and infrequent enough that when it fires, people take it seriously. Everything in this page is aimed at that goal.

When to create alerts

Create alerting policies for conditions that require a human response. Common categories:

• Error ratio: when 5xx responses exceed a percentage of total traffic on a production service
• Latency degradation: when p99 request latency exceeds a threshold that affects user experience
• Uptime failures: when an external health check cannot reach your service from multiple locations
• Metric absence: when a service or job stops reporting metrics entirely, which is a common sign of a crash or deployment problem
• Resource saturation: when CPU, memory, or disk utilization approaches limits
• Quota usage: when GCP quota consumption reaches a high percentage, before requests start being rejected
• Failed batch jobs: when a scheduled Cloud Run job or Dataflow pipeline does not complete successfully

If you are monitoring a Cloud Run service, see Monitoring Cloud Run for the specific metrics to prioritise. For GKE workloads, see Monitoring GKE.

How Cloud Monitoring alerts work

Condition

A condition specifies the metric to watch, the aggregation to apply, and the comparison threshold. Cloud Monitoring supports four condition types, covered in the next section. The condition is evaluated continuously against incoming metric data.

Alignment period

Before evaluating a condition, Cloud Monitoring resamples raw metric data into fixed time windows called the alignment period. For a 60-second alignment period, each data point represents the metric aggregated over the previous 60 seconds as a rate, sum, mean, or percentile depending on the aligner you choose. A shorter alignment period is more responsive but produces noisier results. 60 seconds is a reasonable default for most production alerts. See Metrics in GCP for how aligners and metric kinds interact.

Duration (retest window)

Duration is how long the condition must be continuously true before an incident is created. A duration of 0 fires the moment the condition is first met, which is useful for a complete service outage but noisy for most threshold alerts. A duration of 5 minutes (300s) means the condition must hold across five consecutive 60-second alignment windows. For most error rate and latency alerts, 2 to 5 minutes is a good starting point.

Incident creation

When the condition is met for the full duration, Cloud Monitoring creates an incident and sends notifications to your configured channels. Incidents are tracked under Monitoring > Alerting > Incidents. See Incident Response with Monitoring for how to use incidents during an investigation.

Notification channels

Notification channels are configured separately from alerting policies and reused across many policies. This means you can update a Slack webhook token in one place and all policies pointing to that channel pick up the change. Supported channel types: email, Slack, PagerDuty, webhook, and Pub/Sub.

Incident close and auto-close

An incident closes automatically when the condition is no longer true. You can also close it manually. Cloud Monitoring can send a notification when an incident closes, called the “incident closed” notification. Enable it so your team knows when to stand down. If a condition remains true for 7 days without manual intervention, Cloud Monitoring auto-closes the incident and reopens it if the condition is still met.

Severity and documentation

Each alerting policy can have a severity level (critical, error, warning) and a documentation field. Use the documentation field to link to a runbook. A notification that says “p99 latency is above 2000ms, see https://runbook/api-latency for diagnosis steps” is far more useful than one that says “alert fired”. Fill this field in from the start.

Prerequisites

• IAM permissions: you need the Monitoring Editor role (roles/monitoring.editor) or Monitoring Admin role (roles/monitoring.admin) to create alerting policies. Monitoring Viewer is read-only.
• Metric data must exist: alerting policies can be created before metrics arrive, but you cannot validate the condition or threshold until data is flowing. Deploy your service and send some traffic before configuring production alerts.
• Notification channels should be configured first: create channels before policies so you can reference them immediately. The GCP Console lets you create channels inline during policy creation, but doing it separately first makes Terraform easier.
• Billing must be enabled: Cloud Monitoring alerting requires an active billing account. The free tier includes a reasonable number of alerting policies for most teams.

Alert types and when to choose each

Metric threshold

What it is: fires when a metric value exceeds or falls below a threshold for the configured duration.

When to use it: for any continuously-emitted numeric metric: error count, latency percentile, CPU utilization, memory usage, request rate.

Example: alert when p99 request latency for your Cloud Run service exceeds 2000ms for 2 consecutive minutes.

Metric absence

What it is: fires when no data is received for a metric within a time window. The trigger is the absence of data, not a high value.

When to use it: for services or jobs that must report regularly. If a batch job normally writes a “job completed” metric every hour and stops doing so, absence alerting catches it. Also useful for detecting when a service has crashed entirely and is no longer emitting any metrics.

Example: alert if a Pub/Sub subscriber stops acknowledging messages and the subscription metric goes silent for 10 minutes.

Log-based metric alert

What it is: combines a log-based metric (a counter or distribution derived from log entries) with a threshold condition.

When to use it: for conditions that are clearest in logs: application error codes, specific warning patterns, or events that do not have a corresponding built-in metric. For example, alerting when a specific exception appears more than 10 times per minute.

Example: create a log-based metric that counts log entries matching severity=ERROR AND jsonPayload.code=“PAYMENT_FAILED”, then create a threshold alert on that metric.

Uptime check alert

What it is: fires when an uptime check fails from a configured number of geographic locations.

When to use it: for external reachability monitoring. Catches DNS failures, expired SSL certificates, load balancer misconfigurations, and complete service outages that internal metrics cannot see. Require at least 2 locations to fail before alerting to avoid false positives from single-location network blips.

Example: alert when your /health endpoint fails from 2 or more of 3 configured locations for 1 minute.

Step-by-step: create an alert in the Google Cloud Console

This walkthrough creates a metric threshold alert. The same flow applies to other condition types. Only the condition configuration step differs.

Step 1: Open alerting

In the GCP Console, navigate to Monitoring > Alerting. Click Create policy.

Step 2: Select a metric

Click Select a metric. Use the search box to find the metric you want to alert on. For a Cloud Run error alert, search for request_count and select the Cloud Run resource type. For a latency alert, search for request_latencies.

After selecting the metric, you can filter the time series. For a Cloud Run service, add a filter: resource.service_name = “your-service-name”. This scopes the alert to one service rather than all Cloud Run services in the project.

Step 3: Configure aggregation

The aggregation section controls the alignment period and how data is reduced across multiple time series. For a latency p99 alert, change the aligner from “mean” to “99th percentile”. For an error count alert, use “rate” to get requests per second. The Cloud Monitoring overview explains how the metrics pipeline works in more detail.

Step 4: Set the threshold condition

Choose whether the condition fires when the value is above or below the threshold. Set the threshold value, for example 2000 for a latency alert measured in milliseconds. Check the metric’s unit before entering a number. This is where beginners most often set the wrong value.

Step 5: Set the duration

This is the retest window. The default is often 0 or 1 minute. For most production alerts, increase this to 2 to 5 minutes to reduce noise. Do not leave it at 0 unless the condition represents a complete outage.

Step 6: Add notification channels

Click Next to move to the notification step. Select existing channels or create new ones. Create at least two channels of different types (for example, email and Slack) so that a single-channel failure does not silence your alerts.

Step 7: Add documentation

In the Alert details section, add a display name and fill in the Documentation field. Include a runbook link or a plain-English description of what to check first. This text is included in the notification.

Step 8: Enable incident closure notifications

In the notification settings, enable “Notify when incident is closed”. This tells your team when the condition has resolved so they know when to stop investigating.

Step 9: Review and save

Click Save policy. The policy is active immediately. It will not fire until metric data arrives and the condition is met for the full duration.

Example: Cloud Run 5xx error ratio alert

Cloud Monitoring’s condition_threshold block supports ratio conditions via denominator_filter. The numerator filter matches 5xx requests. The denominator filter matches all requests. Cloud Monitoring computes the ratio and compares it to the threshold (0.05 = 5%). Both sides use identical aggregations so the division is consistent.

resource "google_monitoring_alert_policy" "error_ratio_high" {
  display_name = "api-service: 5xx error ratio > 5%"
  combiner     = "OR"

  conditions {
    display_name = "5xx error ratio above 5% for 5 minutes"
    condition_threshold {
      # Numerator: rate of 5xx requests for api-service
      filter = join(" AND ", [
        "resource.type=\"cloud_run_revision\"",
        "metric.type=\"run.googleapis.com/request_count\"",
        "resource.labels.service_name=\"api-service\"",
        "metric.labels.response_code_class=\"5xx\""
      ])

      # Denominator: rate of all requests for api-service
      denominator_filter = join(" AND ", [
        "resource.type=\"cloud_run_revision\"",
        "metric.type=\"run.googleapis.com/request_count\"",
        "resource.labels.service_name=\"api-service\""
      ])

      duration        = "300s"  # 5 minutes
      comparison      = "COMPARISON_GT"
      threshold_value = 0.05    # 5% error ratio

      # Sum request rates across all revisions of api-service
      aggregations {
        alignment_period     = "60s"
        per_series_aligner   = "ALIGN_RATE"
        cross_series_reducer = "REDUCE_SUM"
        group_by_fields      = ["resource.labels.service_name"]
      }

      # Same aggregation for denominator so the ratio is consistent
      denominator_aggregations {
        alignment_period     = "60s"
        per_series_aligner   = "ALIGN_RATE"
        cross_series_reducer = "REDUCE_SUM"
        group_by_fields      = ["resource.labels.service_name"]
      }
    }
  }

  documentation {
    content   = "## api-service: 5xx error ratio above 5%\n\nMore than 5% of requests to api-service are returning 5xx errors.\n\nCheck recent deployments and review error logs:\nhttps://your-runbook-url/api-service-errors"
    mime_type = "text/markdown"
  }

  notification_channels = [google_monitoring_notification_channel.email.name]
}

Why 5 minutes? A brief spike during a deployment rollout can temporarily cause 5xx responses even when the deployment ultimately succeeds. A 5-minute duration means the alert only fires if errors persist, which is the signal worth responding to. For a more critical service, shorten this to 2 minutes (120s).

For more on the Cloud Run metrics involved here, see Monitoring Cloud Run.

Example: p99 latency alert

Average latency is misleading. A service where 90% of requests complete in 100ms and 10% take 5000ms will show an average of about 590ms, which looks acceptable. The 10% of users experiencing 5-second responses will not think so. p99 captures the experience of the slowest 1% of requests and is a better proxy for tail-latency problems.

run.googleapis.com/request_latencies is a distribution metric, which means you can query any percentile from it. The ALIGN_PERCENTILE_99 aligner extracts the 99th percentile from each 60-second window.

resource "google_monitoring_alert_policy" "p99_latency_high" {
  display_name = "api-service: p99 latency > 2000ms"
  combiner     = "OR"

  conditions {
    display_name = "p99 request latency above 2000ms for 2 minutes"
    condition_threshold {
      filter = join(" AND ", [
        "resource.type=\"cloud_run_revision\"",
        "metric.type=\"run.googleapis.com/request_latencies\"",
        "resource.labels.service_name=\"api-service\""
      ])
      duration        = "120s"  # 2 minutes
      comparison      = "COMPARISON_GT"
      threshold_value = 2000    # milliseconds

      aggregations {
        alignment_period     = "60s"
        per_series_aligner   = "ALIGN_PERCENTILE_99"
        cross_series_reducer = "REDUCE_PERCENTILE_99"
        group_by_fields      = ["resource.labels.service_name"]
      }
    }
  }

  documentation {
    content   = "## api-service: p99 latency above 2s\n\nThe 99th percentile request latency has exceeded 2000ms.\n\nCheck Cloud Trace for slow requests: https://your-runbook-url/api-service-latency"
    mime_type = "text/markdown"
  }

  notification_channels = [google_monitoring_notification_channel.email.name]
}

Choose your threshold based on your SLO or user-facing requirements. 2000ms is a reasonable starting point for an interactive API. Adjust lower if your service has tighter latency requirements. If this alert fires frequently, use distributed tracing to identify which operations are slow.

Notification channels and routing

Notification channels are configured separately from alerting policies in Monitoring > Alerting > Notification channels. Creating them separately means you can reuse one Slack channel across twenty policies without duplicating configuration.

Channel types

• Email: the simplest option. Good for low-urgency alerts and as a fallback channel. Use a team distribution list rather than an individual email address.
• Slack: suitable for visibility and team awareness. Not reliable as a sole channel for critical alerts since Slack can be unavailable during infrastructure incidents. Requires one-time OAuth authorization in the GCP Console.
• PagerDuty: the right choice for on-call rotation management and escalation. Requires an integration key from your PagerDuty account. Use PagerDuty for critical alerts that require a guaranteed response.
• Pub/Sub: sends notifications as JSON messages to a Pub/Sub topic. Use this to build custom notification logic, feed into ticketing systems, or trigger Cloud Functions for auto-remediation.
• Webhook: sends an HTTP POST to a URL you control. Useful for integrating with systems not natively supported.

Redundancy best practice

Do not rely on a single notification channel for critical alerts. Configure at least two channels of different types, for example Slack for visibility and PagerDuty for on-call paging. If Slack is down during an incident, PagerDuty will still reach the on-call engineer. Email alone is not sufficient for critical alerts because it may be slow and is easy to miss.

# Create an email notification channel
gcloud beta monitoring channels create \
  --display-name="Team Alerts Email" \
  --type=email \
  --channel-labels=email_address=oncall@example.com

# Create a Slack notification channel
# Note: Slack channels require authorization via the GCP Console first
gcloud beta monitoring channels create \
  --display-name="Slack #incidents" \
  --type=slack \
  --channel-labels=channel_name="#incidents"

# List channels to retrieve their IDs for use in Terraform or policy definitions
gcloud beta monitoring channels list --project=my-app-prod

How to test an alert safely

Creating an alert policy does not guarantee it works. A policy can be syntactically valid but point to the wrong metric, have a threshold that will never be reached, or reference a notification channel with an expired token. Test before you rely on it.

Validate the condition in non-production

Lower the threshold temporarily in a non-production environment. For example, set the error ratio threshold to 0.001 (0.1%) instead of 0.05 (5%). Then send a few requests that will result in errors and confirm that an incident is created, notifications are sent to the correct channels, and the incident closes automatically when the condition clears.

Confirm the full incident lifecycle

Check that you receive both the incident-opened notification and the incident-closed notification. Confirm that the incident appears in Monitoring > Alerting > Incidents. Verify that the incident links to the correct metric chart and includes the documentation text you added.

Restore the original threshold before going to production

After validating, restore the correct threshold and confirm the policy is saved. A common mistake is to forget to restore the threshold after testing, which results in a permanently noisy alert.

Metric threshold vs log-based metric vs uptime check

Choosing the wrong alert mechanism is one of the most common configuration errors. Here is a quick guide:

For most production services, you want all four: a metric threshold for errors and latency, a log-based metric for critical application events, an uptime check for external reachability, and a metric absence alert to detect complete silence.