Understanding Rolling Update again

Welcome back. In this walkthrough we recreate a failing ECS rolling update and analyze what went wrong. We’ll:

Rebuild the Docker image and push a new task definition revision from CodeBuild.
Deploy the new task definition revision to an ECS service and observe the failed rolling update.
Manually roll back to the previously working revision and explain why ECS did not automatically restore the service.

This guide preserves the exact commands and configuration used in the reproduction, while clarifying the sequence and root causes so you can apply the same troubleshooting steps in your environment.

1) Build pipeline: CodeBuild buildspec

Below is the buildspec used in CodeBuild to build, tag, and push the Docker image. Note that the tagging was corrected so both latest and the commit-hash tag are pushed.

version: 0.2
phases:
  pre_build:
    commands:
      - echo Logging in to Amazon ECR...
      - aws ecr get-login-password --region $AWS_DEFAULT_REGION | docker login --username AWS --password-stdin 666234738304.dkr.ecr.eu-central-1.amazonaws.com/cryptoproject
      - COMMIT_HASH=$(echo $CODEBUILD_RESOLVED_SOURCE_VERSION | cut -c 1-7)
      - IMAGE_TAG=${COMMIT_HASH:-latest}

  build:
    commands:
      - echo Build started on `date`
      - echo Building the Docker image...
      - docker build -t $REPOSITORY_URI:latest .
      - docker tag $REPOSITORY_URI:latest $REPOSITORY_URI:$IMAGE_TAG

  post_build:
    commands:
      - echo Build completed on `date`
      - echo Pushing the Docker image...
      - docker push $REPOSITORY_URI:latest
      - docker push $REPOSITORY_URI:$IMAGE_TAG
      - sed -i "s|REPOSITORY_URI|$REPOSITORY_URI:$IMAGE_TAG|g" task-definition.json
      - aws ecs register-task-definition --cli-input-json file://task-definition.json

Tips:

Prefer using immutable tags (commit hash or image digest) in the task definition to avoid ambiguity during rollbacks.
Pushing both latest and a commit-tag helps during development, but relying on latest for production rollbacks can cause recovered revisions to pull the same broken image.

2) Application change that caused the failure

We intentionally introduced a code change that caused the container to fail at runtime. The fixed snippet below shows the application after adding app and handling error in the login route while keeping the commented-out route and the commented blocks that originally introduced the issue.

from flask import Flask, render_template, request, redirect, url_for

app = Flask(__name__)

# Default credentials
DEFAULT_USERNAME = "admin"
DEFAULT_PASSWORD = "password123"

@app.route('/login', methods=['GET', 'POST'])
def login():
    error = None
    if request.method == 'POST':
        username = request.form['username']
        password = request.form['password']
        
        # Check if provided credentials match the default ones
        if username == DEFAULT_USERNAME and password == DEFAULT_PASSWORD:
            return redirect(url_for('welcome'))
        else:
            error = 'Invalid Credentials. Please try again.'
    
    return render_template('login.html', error=error)

# @app.route('/welcome')
def welcome():
    return render_template('product.html')

@app.route('/place-order')
def place_order():
    product_id = request.args.get('product')
    return render_template('place_order.html', product_id=product_id)

This is the exact runtime state that CodeBuild packaged and pushed.

3) Git commands used to prepare the image

Commands executed to amend a commit and push a new change (output examples included):

git add .
git commit --amend --reset-author
# Example output:
# 3 files changed, 7 insertions(+), 6 deletions(-)
# Enumerating objects: 9, done.
# Counting objects: 100% (9/9), done.
# Compressing objects: 100% (6/6), done.
# Writing objects: 100% (5/5), 213.80 KiB | 16.00 KiB/s, done.
# Total 5 (delta 3), reused 0 (delta 0)
# To https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/aws-microservice-project
#    87216782..e1389e2  master -> master

Then a new commit and push:

git commit -m "Understand rolling update"
git push origin master
# Example push output:
# Enumerating objects: 5, done.
# Counting objects: 100% (5/5), done.
# Compressing objects: 100% (3/3), done.
# Writing objects: 100% (3/3), 367.00 KiB | 0 bytes/s, done.
# Total 3 (delta 2), reused 0 (delta 0), pack-reused 0
# remote: Validating objects...
# To https://git-codecommit.eu-central-1.amazonaws.com/v1/repos/aws-microservice-project
#    52afb79..3769762  master -> master

4) Task definition registration

When CodeBuild completed, it updated the task-definition.json and registered a new task definition revision. The relevant portion of the registered task-definition (revision 6) is shown below:

{
    "taskDefinition": {
        "taskDefinitionArn": "arn:aws:ecs:eu-central-1:666234783044:task-definition/aws-crypto-app:6",
        "containerDefinitions": [
            {
                "name": "kodeklud-crypto-coin",
                "cpu": 0,
                "memory": 512,
                "portMappings": [
                    {
                        "containerPort": 80,
                        "hostPort": 80,
                        "protocol": "tcp"
                    },
                    {
                        "containerPort": 5000,
                        "hostPort": 5000,
                        "protocol": "tcp"
                    }
                ],
                "essential": true,
                "logConfiguration": {
                    "logDriver": "awslogs",
                    "options": {
                        "awslogs-create-group": "true",
                        "awslogs-group": "/ecs/aws-microservice",
                        "awslogs-region": "eu-central-1",
                        "awslogs-stream-prefix": "ecs"
                    }
                }
            }
        ],
        "family": "aws-crypto-app",
        "executionRoleArn": "arn:aws:iam::666234783044:role/ecsTaskExecutionRole",
        "networkMode": "awsvpc",
        "revision": 6,
        "status": "ACTIVE"
    }
}

The build pipeline substituted the commit hash into the task-definition.json, so the task definition revision is correlated with the image tag.

5) Deploying the new revision in ECS

From the ECS console we updated the service to use revision 6. On the Update Service page we selected revision 6 and clicked Update.

The image shows a webpage interface of the Amazon Elastic Container Service (ECS) with various optional service configurations like Service Connect, Service Discovery, Load Balancing, and more. There are buttons for actions like "Cancel" and "Update" at the bottom.

What happened after the update:

ECS started tasks for revision 6, but they failed to reach a RUNNING state.
The deployment hung in a loop: tasks were repeatedly started and stopped due to container start errors caused by the introduced change.
ECS kept trying to bring up the failing revision instead of returning to the previously working revision.

6) Manual rollback performed

To recover we manually reverted the service back to the last-known working revision:

Open the service in the ECS console and click Update service.
Select the previous working task definition revision (revision 5).
Click Update.

After that:

ECS started tasks using revision 5 and the corresponding image.
The new failing deployment was drained and the service returned to a healthy RUNNING state.

Why ECS did not automatically roll back

Automatic rollback in ECS is not guaranteed by default. Two common blockers are:

No automatic detection and abort: ECS will continue attempting a deployment until it either succeeds or you enable the deployment circuit breaker or external automation to stop it.
Non-immutable image tags: If both the old and new task definitions reference a moving tag like latest, rolling back the task definition may still pull the same broken image.

Best practices: use immutable image tags (commit-hash or image digest like repo@sha256:...) and enable the ECS deployment circuit breaker or monitoring automation to detect and recover from faulty deployments.

Quick reference: Common causes and mitigation

Problem	Why it causes failed rollouts	Mitigation
Using `latest` or mutable tags	Rollback to a previous task definition can still pull the same broken image if `latest` was overwritten	Use immutable tags (`<commit-hash>` or image digest).
No deployment circuit breaker	ECS will keep trying a failing deployment indefinitely	Enable the ECS deployment circuit breaker or add health checks/alarms to trigger rollback automation. See AWS docs below.
Application runtime errors	The container exits during startup, causing new tasks to fail	Add container-level logging and health checks; test images locally before pushing.
Missing logging or observability	Hard to determine why tasks fail	Configure CloudWatch logs and structured logs in the container.

Note: In the table cell above latest is shown as inline code to avoid parsing/ambiguity.

Recap and recommended hardening steps

Reproduced a failing rolling update by pushing a new task definition revision with a runtime error.
ECS attempted the rolling update, but the deployment repeatedly failed and did not auto-rollback.
Manual rollback to the previous task definition revision restored the service.
Root cause: mutable image tags (latest) and lack of an automatic deployment abort mechanism.

Recommended actions:

Use immutable image tags (commit-hash tags or digests) in task definitions.
Enable the ECS deployment circuit breaker to automatically stop unhealthy deployments:
- AWS docs: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-type-ecs.html
Implement health checks (load balancer and container-level) and CloudWatch alarms to detect failed deployments quickly.
Ensure CI/CD injects the immutable image identifier into the task definition that you register.

​1) Build pipeline: CodeBuild buildspec

​2) Application change that caused the failure

​3) Git commands used to prepare the image

​4) Task definition registration

​5) Deploying the new revision in ECS

​6) Manual rollback performed

​Why ECS did not automatically roll back

​Quick reference: Common causes and mitigation

​Recap and recommended hardening steps

Watch Video