SRE Cheatsheet: Reliability Concepts, SLOs, and Incident Response

Site Reliability Engineering (SRE) is a discipline that applies software engineering practices to operations problems. This page is a quick reference covering the terminology, metrics, and frameworks you will use in an SRE role.

SRE Terms Glossary#

SLO vs SLA: The Key Distinction#

An SLO is an internal engineering target. It is tighter than what you promise customers so that you have a buffer before breaching contractual commitments. When you are close to consuming the full error budget, engineering teams focus on reliability rather than shipping new features.

An SLA is an external commitment — a legal or contractual agreement with customers. Breaching an SLA typically results in service credits or financial penalties. Because of this, SLAs are always set lower than SLOs.

Example:

• SLO: 99.95% availability over 30 days (internal target)
• SLA: 99.9% availability over 30 days (customer commitment)

This gives a buffer of 0.05 percentage points to absorb incidents before contractual obligations are at risk.

Availability Nines#

Moving from 99.9% to 99.99% is a 10x improvement in uptime. Each additional nine is progressively harder and more expensive to achieve.

Error Budgets#

How it is calculated — If your SLO is 99.9% availability over 30 days, you have 0.1% of requests that may fail. For a service handling 1 million requests per day, that is 1,000 failed requests per day, or ~30,000 per month.

Burning the budget — When incidents or bad deployments cause more failures than the SLO allows, you are “burning” the error budget. SRE teams track budget consumption in real time.

Why error budgets matter — The error budget creates a shared language between development and operations teams:

• If the budget is healthy (plenty remaining), development teams can move fast and deploy frequently.
• If the budget is nearly exhausted, the focus shifts to reliability work: fixing bugs, improving tests, reducing toil.
• This removes the adversarial dynamic where ops says “no deploys” and dev says “we need to ship.”

Error budget policies should be written down. A common policy: if the error budget is >50% consumed in the first half of the window, freeze feature releases and focus on reliability.

Key SRE Metrics#

Track p95 and p99 latency, not just averages. Averages hide tail latency, which is often what causes user complaints.

The Four Golden Signals#

Defined in the Google SRE book, these four metrics cover most service health scenarios:

Latency — The time it takes to serve a request. Track both success latency and error latency separately. Slow errors are different from fast errors.

Traffic — The demand on the system. Requests per second, queries per second, transactions per second — whichever unit is most meaningful for your service.

Errors — The rate of failed requests. Include both explicit failures (HTTP 5xx) and implicit failures (HTTP 200 with wrong content, requests that time out).

Saturation — How full the service is. A 90% saturated service is approaching its limit; small spikes will cause degradation. Also consider leading indicators like memory pressure and queue depth.

Incident Response Framework#

Mitigation before root cause — Do not spend 30 minutes debugging while users are affected. Rollback first, investigate second.

Runbook Structure#

A runbook is a document that guides an on-call engineer through diagnosing and resolving a specific alert. A good runbook contains:

1. Alert name and context — Which alert triggered this? What does it mean?
2. Service overview — What does this service do? What depends on it?
3. Diagnosis steps — What dashboards to check, what log queries to run, what to look for
4. Mitigation steps — Concrete commands or actions to reduce impact
5. Resolution steps — How to fully fix the problem
6. Escalation path — Who to contact if the runbook does not resolve the issue
7. Related links — Dashboard URLs, other runbooks, architecture diagrams

Post-Mortem Structure#

A post-mortem (also called an incident review) should be written within 24–48 hours of an incident being resolved.

Blameless culture — A blameless post-mortem focuses on systems and processes, not people. The assumption is that engineers acted in good faith with the information they had. The question is not “who broke it?” but “what allowed this to happen, and how do we make the system safer?”

Toil#

Toil is manual, repetitive, automatable work that scales with the size of the system. Examples: manually restarting a service when it crashes, manually provisioning user accounts, manually reviewing logs for known errors.

SRE teams aim to keep toil below 50% of their working time. The other 50% should be engineering work that reduces toil or improves reliability. Tracking toil is important because it surfaces automation opportunities and prevents the team from being consumed by operational busywork.

Common SRE Interview Questions#