Shows a simulated Amazon ECS rolling deployment failure, explains why ECS does not auto rollback, and outlines mitigations like deployment circuit breaker, health checks, CodeDeploy, and CI/CD safeguards
Welcome back. This lesson demonstrates what happens when a rolling deployment in Amazon ECS fails and how that failure can leave your service unhealthy or stuck. We’ll simulate a faulty build artifact, push it through the CI/CD pipeline, and observe ECS behavior so you can learn how to guard against and recover from such failures.Environment: AWS Cloud9 (you may also use VS Code in similar labs).What we’ll do
Introduce a deliberately broken application build.
Push the change to the repository and trigger a CodeBuild run that produces a new container image.
Update an ECS service to use the new task revision and observe the failed rolling deployment.
Cover why ECS does not automatically rollback by default and outline mitigations.
Application (fixed, simplified Flask app)
The app shown below is the simplified Flask application used in the demo. In the lab, a developer intentionally removes or comments out critical code (for example, a route or import) to create a broken image that will fail at runtime or fail health checks.
# pythonfrom flask import Flask, render_template, request, redirect, url_forapp = Flask(__name__)# Default credentialsDEFAULT_USERNAME = "admin"DEFAULT_PASSWORD = "password123"@app.route('/', methods=['GET', 'POST'])def login(): error = None if request.method == 'POST': username = request.form.get('username') password = request.form.get('password') # Check if provided credentials match the default ones if username == DEFAULT_USERNAME and password == DEFAULT_PASSWORD: return redirect(url_for('welcome')) else: error = 'Invalid Credentials. Please try again.' return render_template('login.html', error=error)@app.route('/welcome')def welcome(): return render_template('product.html')@app.route('/place_order')def place_order(): # Code to place order return render_template('receipt.html')if __name__ == '__main__': app.run(host='0.0.0.0', port=5000)
Commit and push the faulty change
After deliberately introducing the fault, the source changes are committed and pushed from the Cloud9 environment:
Trigger CodeBuild to create a new image
A push to the repository triggers a CodeBuild project which builds a new container image and pushes it to Amazon ECR. Wait for the build to complete and then verify the new image in your repository.
Verify the image in ECR
Once CodeBuild finishes successfully, it pushes the image to ECR. Confirm the new image and its tag in your repository before updating ECS.
Update the ECS service
Update the ECS service to point to the latest task definition revision (the one that references the newly built image). In the demo, we also reduce the desired task count to 1 so the effect of the broken deployment is easier to observe.
Observed failure behavior
When the new task launches, the container image contains the intentional fault, so the task either crashes or fails load-balancer/container health checks. ECS will keep attempting to start replacement tasks, which results in a repeating failure cycle.Common symptoms:
New tasks never reach RUNNING + healthy state.
Service desired count is not satisfied (for example, shows 0/1 or 0/N).
Continuous restart attempts that consume compute and increase cost.
End-user impact due to unavailable application.
Why ECS does not auto-rollback
By default, ECS persistently tries to deploy the requested task definition revision. ECS will not automatically revert to the previous working task definition unless you configure automatic rollback using the deployment circuit breaker or orchestrate rollback via external automation (CI/CD, CloudWatch alarms, or CodeDeploy).Mitigation and best practices
Use one or more of the following features and practices to prevent or reduce the impact of failed rolling deployments:
Feature / Practice
Purpose
Example / How-to
ECS Deployment Circuit Breaker
Automatically stop a failing deployment and optionally rollback
Enable deploymentConfiguration with deploymentCircuitBreaker set to enable: true and rollback: true (docs)
Container & LB health checks
Detect unhealthy tasks quickly so ECS can respond
Define healthCheck in the task definition and configure target group health checks in the load balancer
AWS CodeDeploy (Blue/Green)
Safer traffic shifting and automated rollback for ECS
Use CodeDeploy blue/green deployments to perform controlled traffic shifting and automatic rollback on failure (docs)
CI/CD gates and testing
Catch application-level defects before images are published
Add unit/integration tests, static analysis, and end-to-end checks in CodeBuild pipelines
CloudWatch alarms + automation
Trigger alerts or automated remediation/rollback
Configure CloudWatch alarms on task failures or target group unhealthy counts and tie to Lambda or Systems Manager Automation runbooks
Enable the ECS deployment circuit breaker and configure container and load-balancer health checks to automatically abort failed rolling updates and roll back to a previously stable task revision.
Next steps
We will next examine ECS deployment circuit breaker settings in detail and demonstrate an automatic rollback during a failed deployment. For further reading and to implement the suggestions above, see the links below.Links and references
