AWS CodeDeploy Rollback Explained: EC2 vs ECS vs Lambda
AWS CodeDeploy can automatically revert a failed deployment before anyone notices. Rollback is triggered by a deployment failure or a CloudWatch alarm breach, and it works differently on each compute platform. EC2 rollbacks re-run the previous revision on the same instances. ECS rollbacks shift traffic back to the task set that was kept alive during deployment. Lambda rollbacks update a function alias to point back to the old version. How fast that recovery happens, and what risks you take, depends on which platform you are deploying to.
Simple explanation
CodeDeploy rollback is a version-control undo for your running application. When you deploy a new version and something breaks, CodeDeploy can automatically switch back to the last version that worked.
The key insight is that “reverting” means something different on each platform:
- EC2: The old code gets re-installed on your servers. It takes a few minutes because the installation steps run again.
- ECS: Traffic is flipped back to the old containers, which were never shut down. It takes seconds.
- Lambda: A pointer (the alias) is updated to point back to the old function version. It is near-instant.
Analogy: Think of a restaurant mid-service. ECS is like having a backup chef already at the stove. If the new chef makes a bad dish, you flip back to the backup in seconds. EC2 is like calling the backup chef in from home — they need time to arrive, change, and review the menu. Lambda is like updating a nameplate on the pass-through window: the previous chef never left the kitchen, you just redirect which name is active.
If you are new to CodeDeploy, read the AWS CodeDeploy Overview first to understand applications, deployment groups, and lifecycle hooks before diving into rollback configuration.
How CodeDeploy rollback works
When a rollback triggers, CodeDeploy creates a new deployment entry in the history marked as a rollback. It redeploys the revision from the last successful deployment rather than a new one. The original failed deployment is not deleted; it stays in the deployment history with a failed status.
Two events can trigger an automatic rollback:
- Deployment failure: The deployment itself fails. Instances fail health checks, a lifecycle hook script exits with a non-zero code, or not enough instances become healthy within the timeout window.
- CloudWatch alarm breach: The deployment succeeds, but a CloudWatch alarm fires during or after the deployment window. CodeDeploy stops the deployment and reverses it.
Rollback is configured per deployment group, not per application. If you have multiple deployment groups (such as my-app-prod and my-app-staging), you must enable rollback on each one separately. A missed deployment group means that environment has no automatic recovery.
Neither event triggers rollback automatically unless you configure it. Rollback is opt-in.
How to enable automatic rollback
Automatic rollback is configured at the deployment group level. You can set it through the AWS Console or the CLI. The following command enables both rollback triggers for an existing deployment group:
aws deploy update-deployment-group \
--application-name my-app \
--current-deployment-group-name my-app-prod \
--auto-rollback-configuration '{
"enabled": true,
"events": [
"DEPLOYMENT_FAILURE",
"DEPLOYMENT_STOP_ON_ALARM"
]
}' \
--alarm-configuration '{
"enabled": true,
"alarms": [
{"name": "my-app-5xx-rate"},
{"name": "my-app-latency-p99"}
]
}'The alarm names in alarm-configuration must match alarms that already exist in CloudWatch. If an alarm does not exist or is misspelled, CodeDeploy silently ignores it. Alarms must be added to the deployment group configuration specifically. Adding them to a separate monitoring dashboard does not connect them to rollback.
Not sure how to set up the CloudWatch alarms that feed into rollback? See Amazon CloudWatch Overview for how alarms work, and Creating Alerts for step-by-step alarm setup. Set thresholds tight enough to catch real problems: 5 to 10 times the normal error rate baseline is a reasonable starting point.
EC2 vs ECS vs Lambda: rollback comparison
| Platform | Rollback mechanism | Speed | What stays available | Main caveat | Best use case |
|---|---|---|---|---|---|
| EC2 | Re-runs last successful revision through full lifecycle hooks | Minutes (varies by instance count and hook scripts) | Instances stay up but may serve bad code until rollback completes | Hooks can fail during rollback too | Traditional apps, VM-based services |
| ECS | Shifts traffic back to original (blue) task set already running | Seconds | Original task set was never terminated during deployment | Rollback window closes when original task set is terminated | Containerized services on Fargate or EC2 |
| Lambda | Updates function alias to point to previous version | Near-instant | Previous version is immutable and always stored | Only works if you use aliases; direct version invocations are not affected | Serverless functions, event-driven workloads |
The speed difference matters most during an incident. An ECS or Lambda rollback completes before most users notice. An EC2 rollback may take long enough that you want to post a status update before it finishes.
EC2 rollbacks in detail
When CodeDeploy rolls back an EC2 deployment, it re-runs the last successful revision on the affected instances. The full AppSpec lifecycle runs again: BeforeInstall, Install, AfterInstall, ApplicationStart, and ValidateService, but with the previous revision’s artifacts.
Rollback time on EC2 depends on the number of instances, the deployment configuration (AllAtOnce, HalfAtATime, OneAtATime), and how long your lifecycle hook scripts take. Timing varies significantly by application; treat any published estimate as a rough approximation for your specific workload.
# Manually stop a deployment and trigger rollback
aws deploy stop-deployment \
--deployment-id d-XXXXXXXXX \
--auto-rollback-enabled
# Or manually redeploy the last good revision
aws deploy create-deployment \
--application-name my-app \
--deployment-group-name my-app-prod \
--deployment-config-name CodeDeployDefault.OneAtATime \
--revision '{
"revisionType": "S3",
"s3Location": {
"bucket": "my-deploy-bucket",
"key": "revisions/last-good-revision.zip",
"bundleType": "zip"
}
}'After a rollback, the deployment list shows a new entry marked as a rollback with a reference to the original failed deployment. This audit trail is useful when reviewing what happened during an incident.
ECS rollbacks in detail
ECS rollbacks work because CodeDeploy never terminates the original task set (the blue environment) during the deployment. It creates a new task set (the green environment), shifts traffic to it, and then waits before terminating the original. During that wait window, a rollback is just a traffic shift back to the original task set.
This is the blue-green deployment model applied to ECS. The original task set stays alive, so reversing is instant. For ECS-specific pipeline setup leading into this deployment model, see CI/CD Pipelines for ECS.
Analogy: Imagine a highway with two parallel lanes open at the same time. An ECS deployment sends traffic into the new lane while the old lane stays open. If the new lane has a problem, you flip traffic back in seconds. EC2 is more like repaving the road itself: the old surface gets replaced, and if something goes wrong during resurfacing, you have to wait for the work to finish before the road is usable again.
# Manual rollback for an ECS deployment
aws deploy stop-deployment \
--deployment-id d-XXXXXXXXX \
--auto-rollback-enabledThe terminateBlueInstancesOnDeploymentSuccess setting controls when the original task set is terminated after a successful traffic shift:
{
"blueGreenDeploymentConfiguration": {
"terminateBlueInstancesOnDeploymentSuccess": {
"action": "TERMINATE",
"terminationWaitTimeInMinutes": 15
}
}
}With a 15-minute wait, the original task set stays alive for 15 minutes after traffic shifts to the new version. During that window, CodeDeploy can roll back instantly if a CloudWatch alarm fires. After 15 minutes, the original task set terminates and the instant rollback window closes.
Choose a wait time that gives your monitoring enough time to detect problems. Ten to fifteen minutes is a common starting point. Services with faster alarm detection can use shorter windows.
Lambda rollbacks in detail
Lambda functions have versions and aliases. A version is an immutable snapshot of the function code and configuration. An alias is a pointer to a specific version. When CodeDeploy deploys a Lambda function, it publishes a new version and shifts the alias (such as production) to point to it.
A rollback is CodeDeploy updating that alias to point back to the previous version. Since the previous version is already stored in Lambda and never changes, the rollback is near-instant.
# Check which version the alias currently points to
aws lambda get-alias \
--function-name my-function \
--name production
# Manual rollback: point the alias back to the previous version
aws lambda update-alias \
--function-name my-function \
--name production \
--function-version 47After a rollback, the alias points to the old version. The new version still exists in Lambda but receives no traffic through the alias. You can delete it later, though this is rarely necessary.
For Lambda, CloudWatch alarms are especially important since there are no instances or health checks to fail. See Monitoring Lambda for how to set up function-level error rate and duration alarms that can feed into CodeDeploy rollback.
When to use rollback vs roll forward
Roll forward means shipping a new version with the bug fixed instead of reverting to the old one. Teams that prefer roll forward argue that it keeps the version in production predictable: it is always the latest one. Rollback-first teams argue that reverting is faster when the fix is not obvious.
Use both. Automatic rollback is the right first response to a broken deployment. It stops user impact quickly. Once the system is stable, diagnose the problem and deploy a fix as a new version. When using canary deployments, you already have a natural rollback point built into the traffic-shift phase, which limits how many users are affected before you decide to roll back or continue.
Rule of thumb: If you can diagnose and fix the problem in under 30 minutes, roll forward. If diagnosing will take longer, roll back first to restore service, then fix and redeploy. Never leave users on a broken version while you investigate.
For a structured approach to deciding between rollback and roll forward during a live incident, Incident Response with Monitoring covers the decision framework.
Database migrations and rollback safety
Rollback is not always safe, and database migrations are the main reason why.
Consider this scenario: your new version adds a user_verified column with a NOT NULL constraint. After deploying, CodeDeploy triggers a rollback to the old version. The old application does not write user_verified. Every insert now fails because the column is required and has no default.
Analogy: Imagine you added a new required field to a paper form while someone was filling out the old version. The old form does not have that field. Any submission using the old form is now incomplete and gets rejected. Database migrations work the same way: if you add a required column and then roll back the application, the old code submits rows without that column, and the database rejects them.
The safe pattern separates schema changes from application changes:
- Deploy the migration first as a backward-compatible addition (nullable column, a column with a default value, or a new table). The old application still works against the new schema.
- Deploy the application change that uses the new schema. Both old and new application versions can coexist with the schema.
- After the application change is fully deployed and stable, tighten the schema constraints if needed.
Never combine a schema migration and an application change in the same deployment if you want rollback to be safe. If you must combine them, plan a forward-only rollout with no rollback option and communicate that before the deployment window starts.
See Dev vs Staging vs Production for how to sequence and test migration changes across environments before they reach production.
How to test rollback in staging
Most teams find out their rollback is broken during a real incident. Test it in staging on a regular schedule instead.
- Deploy a version with a deliberately broken endpoint (returning 500 for every request) or injected latency above your alarm threshold.
- Wait for the CloudWatch alarm to fire.
- Verify CodeDeploy triggers the rollback automatically.
- Verify the rollback completes and the alarm returns to OK.
- Measure the total time from deployment to rollback completion. This is your real MTTR floor.
Schedule rollback drills as a recurring calendar event, once a month in staging. Treat a failed drill the same as a failed production post-mortem. The goal is to find configuration problems before they become 3am problems.
# Deploy the bad version to staging
aws deploy create-deployment \
--application-name my-app-staging \
--deployment-group-name my-app-staging \
--revision '{
"revisionType": "S3",
"s3Location": {
"bucket": "my-deploy-bucket",
"key": "staging/intentionally-bad-version.zip",
"bundleType": "zip"
}
}'
# Poll for the rollback deployment to appear
aws deploy list-deployments \
--application-name my-app-staging \
--deployment-group-name my-app-staging \
--query 'deployments[0]' \
--output textIf the alarm does not fire within a few minutes of deploying the bad version, your thresholds may be too loose. If the rollback does not trigger, check that autoRollbackConfiguration.enabled is true in the deployment group and that the alarm names match exactly. For keeping staging close enough to production to trust the results, see Secure CI/CD Pipelines.
Common mistakes
- Not enabling auto-rollback at all. Rollback is not on by default. Check every production deployment group to confirm
autoRollbackConfiguration.enabledis true. It takes one CLI call to verify. - Setting alarm thresholds too high. An alarm that fires at 50% error rate will not trigger during a canary that affects 5% of users. Set thresholds relative to your baseline: 5 to 10 times normal is a common starting point.
- Terminating the ECS blue task set immediately. Setting
terminationWaitTimeInMinutes: 0removes the instant rollback window. Keep the original task set alive for at least 10 to 15 minutes after a successful traffic shift. - Rolling back through a database migration. If the migration added schema the old code cannot handle, rollback breaks the application. Separate migrations from application changes.
- Never testing rollback. Test it in staging monthly. Deploy a bad version, verify the alarm fires, verify CodeDeploy rolls back, and measure the total recovery time. This is maintenance work that pays off during production incidents.
Summary
- Rollback is not enabled by default. Enable it at the deployment group level with
DEPLOYMENT_FAILUREandDEPLOYMENT_STOP_ON_ALARMevents. Check every production deployment group. - EC2 rollbacks re-apply the last successful revision through the full lifecycle. Speed depends on instance count and hook scripts.
- ECS rollbacks shift traffic back to the original task set kept running during deployment. This takes seconds.
- Lambda rollbacks update the function alias to the previous version. This is near-instant because versions are immutable and already stored.
- Rollback is the fastest response to a broken deployment. Roll forward is the right follow-up once the system is stable and the fix is ready.
- Rolling back through a database migration is dangerous. Separate schema changes from application changes so the old version can still work against the new schema.
- Test rollback in staging monthly. Do not discover it is broken during a real incident.
Frequently asked questions
What is the difference between rollback, redeploy, and roll forward?
A rollback redeploys the last known-good revision already stored in CodeDeploy. It is fast because no new artifact needs to be built or fetched. A redeploy is a fresh deployment of a specific revision through the full pipeline. Roll forward means shipping a new version with the bug fixed rather than reverting. Use rollback for speed during incidents, redeploy for planned corrections, and roll forward when the fix is ready quickly.
Does CodeDeploy rollback work the same way for EC2, ECS, and Lambda?
No. For EC2, CodeDeploy re-runs the last successful revision on the existing instances, going through all lifecycle hooks again. This takes several minutes. For ECS, CodeDeploy shifts traffic back to the original task set that was kept running during deployment. This takes seconds. For Lambda, CodeDeploy updates the function alias to point back to the previous version. This is near-instant because function versions are immutable and already stored.
Is it safe to roll back a deployment that included a database migration?
Not always. If the migration added a non-nullable column the old application does not write, rolling back causes insert failures. If it renamed or removed a column the old code reads, the rollback breaks queries. The safe pattern is to separate migrations from application changes: deploy the migration first as a backward-compatible addition, then deploy the application change. That way, the old application version still works if you need to roll back.
Can I test rollback before I ever need it in production?
Yes, and you should. Deploy a version with a deliberately broken endpoint or injected latency into staging and verify that the CloudWatch alarm fires and CodeDeploy triggers the rollback automatically. This confirms your alarm thresholds are correct and your rollback configuration actually works. Most teams discover broken rollback configuration during a real incident. Test it in staging monthly instead.
Does the rollback configuration apply to every deployment, or can I override it per deployment?
Rollback is configured at the deployment group level, which means it applies to every deployment in that group by default. You can override the rollback setting per deployment when calling the CreateDeployment API by specifying autoRollbackConfiguration in the request. The per-deployment override only affects that single deployment. Deployment group config is the baseline; per-deployment config is the exception.