Incident Response in the Cloud: What Actually Happens

Incident severity tiers

Most engineering teams use a severity classification system to communicate the urgency and impact of an incident without long explanations. The exact names and numbers vary by company, but the structure is similar everywhere.

The severity level is not always obvious at the start of an incident. It is normal to declare a higher severity and downgrade once you understand the scope. Declaring too low a severity and missing the impact is worse than over-declaring.

One important principle: the on-call engineer does not need to understand or fix the problem to declare an incident. If something looks wrong and you are not sure, declare the incident and get more eyes on it. The cost of a false alarm is low. The cost of a real incident that was not escalated promptly is high.

What on-call is actually like

On-call means you are the primary responder to production alerts during your rotation — evenings, nights, and weekends included. Most teams rotate every one to two weeks, meaning you are on-call for one week in every four to six depending on team size.

The honest picture: on-call quality varies enormously by team. A well-run team with good monitoring, a stable system, and meaningful runbooks might have you go through an entire on-call week without a single page. A team with a poorly monitored legacy system, alert fatigue from too many low-quality alerts, and no runbooks will page you multiple times per night.

Before joining a team, ask:

• How many pages does the average on-call engineer receive per shift?
• What percentage of alerts are actionable (as opposed to noise)?
• Is there a runbook for each alert type?
• How does the company compensate for on-call burden — additional pay, time off in lieu?
• What is the escalation path when the on-call engineer cannot resolve something alone?

These questions reveal a lot about engineering culture. A team that cannot answer them has likely not thought seriously about on-call health.

The incident timeline

Most incidents follow a recognisable sequence. Knowing the stages helps you move through them deliberately rather than reactively.

1. Detection

Something triggers an alert — a monitoring threshold breached, an error rate spike, a health check failing. Or a customer reports a problem before monitoring catches it. Detection time is the gap between the start of the problem and when someone is aware of it. Good monitoring reduces this gap. Alert fatigue (where engineers stop responding to frequent, low-quality alerts) increases it.

2. Triage and severity classification

The on-call engineer assesses: what is broken, how many users are affected, what is the business impact? Based on this, they declare a severity level and decide whether to escalate immediately or investigate first.

3. Incident declaration and communication

For SEV1 or SEV2 incidents: open an incident channel in Slack (e.g., #incident-2026-03-20-payment-errors), post an initial message stating what is known so far, and page any additional responders needed. The incident channel becomes the single source of truth for the duration.

4. Diagnosis

The team works to understand the root cause. This involves reading logs, checking dashboards, looking at what changed recently (deployments, config changes, external dependencies). The key discipline here is hypothesis-driven: form a theory, test it, confirm or rule it out, move to the next hypothesis. Random exploration of logs wastes time.

5. Mitigation

Stop the immediate harm. Mitigation is not the same as fixing the root cause — it is restoring service as quickly as possible. Rolling back a bad deployment, rerouting traffic away from a broken availability zone, disabling a feature that is causing errors. Speed matters here more than elegance. You can fix the root cause properly once the incident is resolved.

6. Recovery and monitoring

After mitigation, confirm that the service has recovered — error rates back to normal, health checks passing, user-visible functionality restored. Monitor for 15–30 minutes to confirm stability before closing the incident.

7. Post-mortem

A blameless review of what happened, why, and what changes would prevent recurrence. Usually happens within 24–72 hours while the details are fresh. Covered in detail below.

Communication during incidents

Poor communication during an incident is as damaging as slow diagnosis. Stakeholders left in the dark make decisions based on no information. Customers who receive no status updates lose confidence. Engineers who do not coordinate end up working on the same thing or undoing each other’s changes.

The incident commander role

On larger incidents, one person takes the incident commander (IC) role. The IC does not personally diagnose or fix — their job is to coordinate. They assign tasks, run status updates, manage communication to stakeholders, and make calls on escalation. Separating the coordination role from the technical investigation keeps both functioning well under pressure.

Status update cadence

For a SEV1 or SEV2 incident, post a status update to the incident channel every 15–20 minutes even if there is nothing new to report. “Still investigating, no change in status” is valuable information — it tells stakeholders that the incident is being actively worked and has not been forgotten. Silence during an incident causes escalation and distraction.

A useful status update format:

[14:32] STATUS UPDATE
Current state: Service is degraded — ~30% of checkout requests failing
What we know: Errors started at 14:15, correlated with deployment at 14:10
What we're doing: Investigating the deployment diff, considering rollback
Next update: 14:45

Communication to non-engineers

Engineering updates are too technical for product managers, customer success teams, and executives. Someone needs to translate: “The payment service is returning 500 errors” becomes “Customers are unable to complete purchases. We are working to restore normal service and estimate recovery by 15:00.” Keep non-technical stakeholder updates short, plain, and action-oriented.

Post-mortem culture

A post-mortem is a structured review of an incident. The goal is not to assign blame — it is to understand what happened well enough to prevent it from happening again.

A blameless post-mortem assumes that engineers made reasonable decisions given the information they had at the time. If an engineer made a change that caused an outage, the right question is not “why did they make that change?” but “what made it possible for that change to cause an outage, and what would have caught it before it reached production?”

What a good post-mortem includes

• Timeline — what happened, in chronological order, with precise timestamps
• Impact summary — how many users were affected, for how long, and what the business impact was
• Root cause — the underlying reason the incident happened (often different from the proximate cause)
• Contributing factors — things that made the incident worse, harder to detect, or harder to resolve
• What went well — detection was fast, the rollback procedure worked, communication was clear
• Action items — specific, assigned, time-bounded tasks to prevent recurrence or improve response

Post-mortems without action items are archaeology, not engineering. The value is in the changes made as a result.

What makes engineers valuable during incidents

When a SEV1 fires at 2am, the most valuable engineers are not necessarily the ones who know the most. They are the ones who remain calm, communicate clearly, form hypotheses systematically, and know when to escalate.

Specific behaviours that matter:

• Narrating your work — posting what you are checking and what you find in the incident channel, so others can follow along and avoid duplicating effort
• Checking recent changes first — most incidents are caused by something that changed recently. Check deployments, config changes, and dependency updates before diving into deeper diagnosis
• Prioritising mitigation over explanation — once you can stop the harm (by rolling back or disabling a feature), do it. You can understand why later
• Avoiding tunnel vision — if a hypothesis is not confirming after 10–15 minutes, step back and consider alternatives
• Knowing when to escalate — if you are stuck and the incident is still active after 20–30 minutes without progress, get more people involved. The ego cost of asking for help is much smaller than the business cost of a prolonged outage

Career advice around on-call readiness

On-call readiness is a significant differentiator between junior and senior cloud engineers. The practical steps to build it before you need it:

• Read runbooks proactively — do not wait for an alert to read the runbook for that alert. Review them during quiet periods so the information is already in your head
• Shadow on-call rotations — most teams will let you shadow a senior engineer on-call before you are primary. Take this seriously
• Practice rollbacks in staging — know how to roll back a deployment, restore a database from backup, and disable a feature flag before you need to do it under pressure
• Build mental models of the system — understand how the components connect, what the dependencies are, and what failure looks like for each component
• Contribute to runbooks — every time you investigate a new type of issue, write down what you checked, what worked, and what you ruled out. This improves the runbook and cements the knowledge in your own memory

Engineers who handle incidents well get noticed. It is visible, high-stakes work where your calm, systematic approach (or lack of it) is observed by the entire team. It is one of the fastest ways to build a reputation as a reliable, senior-minded engineer.