Reboot Reader Node Scenario on Aurora

​

Architecture Overview

​

Prerequisites

• An existing Aurora PostgreSQL cluster with one writer and one reader in multi-AZ.
• AWS CLI v2 configured with permissions for AWS FIS and RDS.
• IAM role for FIS with fis:StartExperiment and rds:RebootDBInstance permissions.
• Cluster identifiers:
  • Writer: aurora-writer-1
  • Reader: aurora-reader-1

​

AWS FIS Experiment Components

1. Target: The AWS resource(s) to inject faults into.
2. Action: The fault to inject (e.g., reboot).
3. Role ARN: IAM role that grants FIS the required permissions.
4. Stop conditions (optional): When to halt the experiment automatically.

​

Step-by-Step: Injecting a Reader Node Reboot

​

1. Define the FIS Experiment Template

{
  "description": "Reboot Aurora PostgreSQL reader node",
  "roleArn": "arn:aws:iam::123456789012:role/FIS-Experiment-Role",
  "targets": {
    "ReaderInstance": {
      "resourceType": "aws:rds:db-instance",
      "resourceArns": [
        "arn:aws:rds:us-east-1:123456789012:db:aurora-reader-1"
      ]
    }
  },
  "actions": {
    "RebootReader": {
      "actionId": "aws:rds:reboot-db-instance",
      "description": "Reboot the reader DB instance",
      "parameters": {
        "dbInstanceIdentifier": "aurora-reader-1"
      },
      "targets": {
        "InstanceTarget": "ReaderInstance"
      }
    }
  }
}

​

2. Start the Experiment

aws fis start-experiment \
  --cli-input-json file://fis-reboot-reader.json \
  --region us-east-1

{
  "experimentId": "exp-abc123xyz",
  "state": "initiating"
}

​

3. Monitor the Experiment

• CLI:
  aws fis get-experiment --id exp-abc123xyz
  
• Console: Visit the AWS FIS Experiments page.

​

Expected Results & Hypothesis

1. Reader goes offline: Aurora shifts read traffic to the writer.
2. Writer handles all requests: No downtime for your application.
3. Reader rejoins: After reboot, reads distribute back across both nodes.

​

Cleanup

1. Delete the FIS experiment template (if created separately).
   aws fis delete-experiment-template --id tpl-xyz123
   
2. Verify that both Aurora instances are healthy:
   aws rds describe-db-instances \
     --db-instance-identifier aurora-reader-1 \
     --query 'DBInstances[].DBInstanceStatus'
   

​

References

• Amazon Aurora PostgreSQL
• AWS Fault Injection Simulator
• AWS CLI Command Reference: fis start-experiment