Demo General Experiment Setup AZ

In this lesson you’ll prepare the prerequisites for running an Availability Zone (AZ) power-interruption experiment using AWS Fault Injection Simulator (FIS). We’ll create an IAM role that FIS can assume and attach the task policy containing the permissions required to simulate an AZ-level power interruption. Overview:

Create a trust role that only FIS can assume.
Inspect the experiment folder and key files.
Review the essential task-policy statements required by the experiment.
Attach the task policy to the role so the experiment can run.

1) Create the IAM role (trust policy)

The AZ experiment folder contains a trust policy file named fis-az-experiment-policy.json. This trust policy allows only the FIS service to assume the role. Run the following command (assumes fis-az-experiment-policy.json is in the current directory):

aws iam create-role \
  --role-name fis-az-role \
  --assume-role-policy-document file://fis-az-experiment-policy.json

Example output:

ubuntu:~/environment/workshopfiles/fis-workshop/az-experiment (main) $ aws iam create-role --role-name fis-az-role --assume-role-policy-document file://fis-az-experiment-policy.json
{
    "Role": {
        "Path": "/",
        "RoleName": "fis-az-role",
        "RoleId": "AROA4YVVCUMJ57GAXUKSR",
        "Arn": "arn:aws:iam::877559718675:role/fis-az-role",
        "CreateDate": "2024-08-05T05:24:20+00:00",
        "AssumeRolePolicyDocument": {
            "Version": "2012-10-17",
            "Statement": [
                {
                    "Effect": "Allow",
                    "Principal": {
                        "Service": [
                            "fis.amazonaws.com"
                        ]
                    },
                    "Action": "sts:AssumeRole"
                }
            ]
        }
    }
}
ubuntu:~/environment/workshopfiles/fis-workshop/az-experiment (main) $

2) Review the experiment folder

List the experiment folder to inspect the files that accompany the AZ experiment:

ubuntu:~/environment/workshopfiles/fis-workshop/az-experiment (main) $ ls -l
total 60
-rw-rw-r-- 1 ubuntu ubuntu 15987 Aug  4 14:41 az-impairment-dashboard.yaml
-rw-rw-r-- 1 ubuntu ubuntu 2280 Aug  4 14:41 disable-enable-cross-zone-alb.sh
-rw-rw-r-- 1 ubuntu ubuntu 212 Aug  4 14:41 fis-az-experiment-policy.json
-rw-rw-r-- 1 ubuntu ubuntu 5448 Aug  4 14:41 fis-az-experiment-template.json
-rw-rw-r-- 1 ubuntu ubuntu 5094 Aug  4 14:41 fis-az-task-policy.json
-rw-rw-r-- 1 ubuntu ubuntu 950 Aug  4 14:41 restart-user-traffic.sh
-rw-rw-r-- 1 ubuntu ubuntu 2328 Aug  4 14:41 shift-traffic-out-of-az-all-albs.sh
-rw-rw-r-- 1 ubuntu ubuntu 2144 Aug  4 14:41 shift-traffic-out-of-az-one-alb.sh
-rw-rw-r-- 1 ubuntu ubuntu 4533 Aug  4 14:41 withzonalshift.sh
ubuntu:~/environment/workshopfiles/fis-workshop/az-experiment (main) $

Quick reference: key files and their purpose

File	Purpose
`fis-az-experiment-policy.json`	Trust policy that allows `fis.amazonaws.com` to assume the role.
`fis-az-task-policy.json`	Task policy containing permissions the FIS role needs to simulate AZ failures (key file).
`fis-az-experiment-template.json`	FIS experiment template that defines actions, targets, and recovery steps.
`az-impairment-dashboard.yaml`	Dashboard configuration for visualizing impairment and experiment metrics.
`*.sh` scripts	Helper scripts to shift and restore traffic across ALBs and AZs (e.g., `shift-traffic-out-of-az-one-alb.sh`).

This role requires broad permissions because an AZ power interruption touches multiple services (network ACLs, compute, databases, caching, and tagging). The policy also scopes destructive actions using tags to minimize blast radius.

3) Key policy statements (fis-az-task-policy.json)

Below is a concise, cleaned representation of the relevant statements from fis-az-task-policy.json. These statements grant the FIS role the ability to log experiment activity, manage temporary network ACLs to simulate network isolation, and perform failover/restart operations on database and cache resources.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowFISExperimentLoggingActionsCloudwatch",
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogDelivery",
        "logs:PutResourcePolicy",
        "logs:DescribeResourcePolicies",
        "logs:DescribeLogGroups"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": "ec2:CreateTags",
      "Resource": "arn:aws:ec2:*:*:network-acl/*",
      "Condition": {
        "StringEquals": {
          "ec2:CreateAction": "CreateNetworkAcl",
          "aws:RequestTag/managedByFIS": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": "ec2:CreateNetworkAcl",
      "Resource": "arn:aws:ec2:*:*:network-acl/*",
      "Condition": {
        "StringEquals": {
          "aws:RequestTag/managedByFIS": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:CreateNetworkAclEntry",
        "ec2:DeleteNetworkAcl"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:network-acl/*",
        "arn:aws:ec2:*:*:vpc/*"
      ],
      "Condition": {
        "StringEquals": {
          "ec2:ResourceTag/managedByFIS": "true"
        }
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "rds:FailoverDBCluster"
      ],
      "Resource": [
        "arn:aws:rds:*:*:cluster:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "rds:RebootDBInstance"
      ],
      "Resource": [
        "arn:aws:rds:*:*:db:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "elasticache:DescribeReplicationGroups",
        "elasticache:InterruptClusterAzPower"
      ],
      "Resource": [
        "arn:aws:elasticache:*:*:replicationgroup:*"
      ]
    },
    {
      "Sid": "TargetResolutionByTags",
      "Effect": "Allow",
      "Action": [
        "tag:GetResources"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:StartInstances",
        "ec2:StopInstances"
      ],
      "Resource": "arn:aws:ec2:*:*:instance/*"
    }
  ]
}

Notes on the primary statements:

CloudWatch Logs actions are permitted so FIS can create and manage log delivery for experiments.
Network ACL creation, tagging, entry creation/deletion, and deletion are restricted by the managedByFIS tag to limit impact to resources the experiment manages.
RDS and ElastiCache actions allow failing over or rebooting database and cache resources to simulate AZ-level failures.
Tag-based resource resolution (tag:GetResources) enables FIS to select targets by tags.
EC2 instance StartInstances and StopInstances allow lifecycle operations as part of the simulation.

4) Attach the task policy to the role

After creating the role, attach the task policy using put-role-policy:

aws iam put-role-policy \
  --role-name fis-az-role \
  --policy-name fis-az-policy \
  --policy-document file://fis-az-task-policy.json

Example (no output on success):

ubuntu:~/environment/workshopfiles/fis-workshop/az-experiment (main) $ aws iam put-role-policy --role-name fis-az-role --policy-name fis-az-policy --policy-document file://fis-az-task-policy.json
ubuntu:~/environment/workshopfiles/fis-workshop/az-experiment (main) $

Always follow the principle of least privilege. Before running experiments that perform destructive actions, validate the role and policy in a controlled environment and ensure that tag scoping (for example, managedByFIS) is correctly applied to limit the blast radius.

Now the FIS role is created and associated with the task policy that enables the AZ power-interruption simulation. You can run the experiment using this role and the fis-az-experiment-template.json experiment template.

Introduction

Building a Basic FIS experiment

Introduction to Real life Application

Chaos Engineering on Compute - EC2

Chaos Engineering on Database - Aurora

Chaos Engineering on Serverless - Fargate

Chaos Engineering on Kubernetes- EKS

Chaos Engineering on Availability Zone

Conclusion

Chaos Engineering Fundamentals

Demo General Experiment Setup AZ

1) Create the IAM role (trust policy)

2) Review the experiment folder

3) Key policy statements (fis-az-task-policy.json)

4) Attach the task policy to the role

Links and references

Watch Video

​1) Create the IAM role (trust policy)

​2) Review the experiment folder

​3) Key policy statements (fis-az-task-policy.json)

​4) Attach the task policy to the role

​Links and references

Watch Video

1) Create the IAM role (trust policy)

2) Review the experiment folder

3) Key policy statements (fis-az-task-policy.json)

4) Attach the task policy to the role

Links and references