> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Infrastructure as Code for SRE
> Guide for SREs on Infrastructure as Code covering fundamentals, tools, best practices, testing and CI/CD, Terraform IAM example, policy as code, and drift detection and remediation.
Welcome. In this lesson we cover Infrastructure as Code (IaC) fundamentals for Site Reliability Engineers (SREs): what IaC is, why it matters, essential practices, tooling, testing, policy as code, a real-world Terraform example (IAM users), CI/CD patterns, drift detection, and remediation strategies.
At its core, Infrastructure as Code (IaC) is the practice of treating infrastructure the same way you treat application code: store it in version control, make changes via pull requests, review and test them, and apply changes in a repeatable automated workflow. This shifts infrastructure from a manual, error-prone process into a reliable, auditable, and testable pipeline.
Historically, teams provisioned infrastructure by SSHing into hosts, editing config files, restarting services, and hoping nothing broke. That approach leads to infrastructure drift — live systems that diverge from the documented or desired state, and no reliable audit trail of who changed what.
IaC reverses that model: declare your desired state in code, review changes through Git workflows, run automated checks and previews, and apply updates with tooling (Terraform, CloudFormation, etc.). The result: consistent environments, auditable changes, and the ability to test updates before they reach production.
## Why SREs adopt IaC
* Eliminate snowflake servers — provisioned resources are consistent and reproducible.
* Predictable disaster recovery — rebuild environments from code.
* Auditability — Git history provides a trace of who changed what and when.
* Safer deployments — preview and test infrastructure changes before applying them, reducing incidents and on-call interruptions.
## Popular IaC tools
Use the right tool for the job: provisioning, configuration management, or policy enforcement. Below is a quick reference.
| Tool | Type / Primary Use | Notes |
| - | - | - |
| Terraform / OpenTofu | Declarative provisioning | Widely used for multi-cloud infrastructure as code. See [Terraform docs](https://www.terraform.io/docs). |
| AWS CloudFormation | Declarative provisioning (AWS) | Native AWS IaC with deep service integration. |
| Pulumi | Imperative/declarative using general-purpose languages | Use familiar programming languages for infra. |
| Ansible | Configuration management & orchestration | Agentless, uses YAML playbooks. |
| Chef / Puppet | Configuration management | Longstanding CM tools for large fleets and complex policies. |
| Open Policy Agent (OPA) | Policy-as-code | Enforce fine-grained policies in CI/CD. |
## IaC best practices (practical checklist)
* Version control everything. If it isn't in Git, it effectively doesn't exist — avoid manual cloud-console changes.
* Use variables and parameterization instead of hard-coded values to support multiple environments.
* Create reusable modules and follow DRY (Don't Repeat Yourself) patterns.
* Treat infrastructure changes like application code: require pull requests, peer review, and automated checks (lint, security scans, tests).
* Isolate secrets and store them in secure vaults or CI/CD secrets (never in source).
Example: keep Terraform code in Git instead of console changes:
```bash theme={null}
# Good: All infrastructure code in Git
git add main.tf variables.tf
git commit -m "Add production database with backup retention"
# Bad: Making changes through the AWS console (untracked, unreviewed)
# (No commit message can capture that regret)
```
## Testing infrastructure changes
Infrastructure must be validated like application code. Integrate local checks and CI pipeline scans.
Common local commands:
| Purpose | Command |
| - | - |
| Preview changes | `terraform plan` |
| Validate syntax | `terraform validate` |
| Lint for best practices | `tflint` |
| Security scanning | `tfsec .` |
Example local usage:
```bash theme={null}
# Preview changes
terraform plan
# Validate syntax
terraform validate
# Lint for best practices
tflint
# Security scanning
tfsec .
```
CI/CD pipelines should run these steps automatically on PRs and block merges when checks fail. Example GitHub Actions workflow that runs plan and a security scan:
```yaml theme={null}
name: Terraform CI
on:
pull_request:
push:
branches:
- main
- dev-test
jobs:
terraform-plan-and-scan:
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
- name: Terraform Init
run: terraform init -input=false
- name: Terraform Plan
run: terraform plan -out=tfplan
- name: Security Scan (tfsec)
run: tfsec .
- name: Require Manual Approval for prod changes
if: contains(github.event.pull_request.title, 'prod')
uses: trstringer/manual-approval@v1
```
## Policy as code
Encode guardrails so issues are caught before any apply step runs. Policy-as-code tools like Open Policy Agent (OPA) and HashiCorp Sentinel can prevent unsafe changes (for example: deny publicly-readable S3 buckets, require backup tags on critical resources, or block production-affecting changes without explicit approvals).
## Real-world example: automating AWS IAM user creation
Manual IAM user creation at scale is error-prone: missing MFA, inconsistent policy attachments, or different tags across accounts. IaC lets you define users in variables, iterate to create them, attach managed or inline policies, and enforce consistent metadata (tags, MFA enforcement via policy, etc.).
Example Terraform patterns for IAM users
1. Structured variable for users:
```terraform theme={null}
variable "iam_users" {
description = "List of IAM users to create"
type = list(object({
name = string
groups = list(string)
}))
default = [
{
name = "pablo.developer"
groups = ["developers"]
},
{
name = "julia.sre"
groups = ["sre", "developers"]
}
]
}
```
2. Create users dynamically and tag them:
```terraform theme={null}
resource "aws_iam_user" "users" {
for_each = {
for user in var.iam_users : user.name => user
}
name = each.value.name
force_destroy = true
tags = {
Environment = "production"
ManagedBy = "terraform"
}
}
```
3. Attach a managed policy example:
```terraform theme={null}
resource "aws_iam_user_policy_attachment" "mfa_policy" {
for_each = aws_iam_user.users
user = each.value.name
policy_arn = "arn:aws:iam::aws:policy/IAMUserChangePassword"
}
```
## Repository walkthrough (summary)
The sample repo (KodeKloud Records Terraform Infrastructure) demonstrates these concepts: remote state in S3, modular IAM user creation, and CI/CD workflows for plan/apply/destroy. Clone or fork the repo to follow along.
Top-level Terraform configuration (remote state and S3 backend excerpt):
```terraform theme={null}
terraform {
required_version = ">= 1.3.0"
backend "s3" {
key = "global/terraform.tfstate"
region = "eu-north-1"
encrypt = true
# The bucket will be provided at terraform init via -backend-config or pre-created.
}
}
resource "random_id" "bucket_suffix" {
byte_length = 4
}
resource "aws_s3_bucket" "terraform_state" {
bucket = "terraform-state-${random_id.bucket_suffix.hex}"
lifecycle {
prevent_destroy = true
}
}
resource "aws_s3_bucket_versioning" "versioning" {
bucket = aws_s3_bucket.terraform_state.id
versioning_configuration {
status = "Enabled"
}
}
resource "aws_s3_bucket_server_side_encryption_configuration" "encryption" {
bucket = aws_s3_bucket.terraform_state.id
rule {
apply_server_side_encryption_by_default {
sse_algorithm = "AES256"
}
}
}
```
Module usage and inline policy (module inputs excerpt):
```terraform theme={null}
module "iam_users" {
source = "./modules/iam-user"
iam_usernames = var.iam_usernames
managed_policy_arns = [
"arn:aws:iam::aws:policy/ReadOnlyAccess",
"arn:aws:iam::aws:policy/IAMUserChangePassword"
]
inline_policy_document = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"s3:ListAllMyBuckets",
"s3:GetBucketLocation",
]
Effect = "Allow"
Resource = "*"
},
{
Action = [
"s3:List*",
"s3:Get*",
]
Effect = "Allow"
Resource = [
"arn:aws:s3:::dev-bucket",
"arn:aws:s3:::dev-bucket/*"
]
},
{
Action = [
"s3:ListBucket",
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
]
Effect = "Allow"
Resource = [
"arn:aws:s3:::${aws_s3_bucket.terraform_state.bucket}",
"arn:aws:s3:::${aws_s3_bucket.terraform_state.bucket}/*",
]
},
]
})
}
```
Example tfvars:
```hcl theme={null}
aws_region = "eu-north-1"
iam_usernames = [
"iamuser-pablo",
"iamuser-julia",
"iamuser-diego"
]
```
Module implementation excerpt (modules/iam-user/main.tf):
```terraform theme={null}
resource "aws_iam_user" "new_users" {
for_each = toset(var.iam_usernames)
name = each.value
path = "/"
# Prevent Terraform from trying to update existing users' attributes
lifecycle {
ignore_changes = all
}
}
resource "aws_iam_access_key" "user_keys" {
for_each = aws_iam_user.new_users
user = each.value.name
}
resource "aws_iam_user_policy_attachment" "user_policy_attachments" {
for_each = {
for pair in setproduct(keys(aws_iam_user.new_users), var.managed_policy_arns) : "${pair[0]}-${replace(pair[1], ":", "_")}" => {
user = pair[0]
policy_arn = pair[1]
}
}
user = aws_iam_user.new_users[each.value.user].name
policy_arn = each.value.policy_arn
}
resource "aws_iam_user_policy" "inline_policy" {
for_each = var.inline_policy_document != null ? aws_iam_user.new_users : {}
name = "${each.value.name}-inline-policy"
user = each.value.name
policy = var.inline_policy_document
}
```
## GitHub Actions to apply Terraform
Store AWS credentials as repository secrets and give the CI user least privilege required for the workflow. Use a workflow that checks out code, sets up Terraform, runs init, and applies.
Example apply workflow (trigger on specific branches):
```yaml theme={null}
name: Terraform Apply
on:
push:
branches:
- main
- dev-test
jobs:
terraform-apply:
name: "Terraform Apply"
runs-on: ubuntu-latest
steps:
- name: Checkout code
uses: actions/checkout@v3
- name: Setup Terraform
uses: hashicorp/setup-terraform@v2
- name: Terraform Init
run: terraform init -input=false
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
- name: Terraform Apply
run: terraform apply --auto-approve
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
```
Store credentials in GitHub repository secrets and never commit them to the repo. Grant the CI user least privilege needed for the workflow.
The lesson repo includes an apply workflow and a separate destroy workflow you can run manually when you need to tear down resources.
After pushing a branch that triggers the apply workflow, Terraform will create the IAM users defined in your variables. The example run created several users as shown in the IAM console.
There is also a "Terraform Destroy" workflow for cleaning up test resources when you're done.
## Infrastructure drift: detection and remediation
A common challenge is infrastructure drift: live infrastructure that no longer matches code. Example: someone urgently grants CloudWatch access in the AWS console instead of updating Terraform — the configuration diverges.
Detect drift
* Run `terraform plan` against the live environment. Terraform compares real resources to the state file and highlights differences.
* Integrate periodic drift detection (scheduled CI jobs) to detect drift proactively.
Fixing drift — three options
1. Accept the manual change by updating your IaC (recommended when the manual change is valid).\
Example: Add CloudWatch permissions in Terraform:
```terraform theme={null}
resource "aws_iam_user_policy" "cloudwatch_policy" {
for_each = aws_iam_user.new_users
name = "${each.value.name}-cloudwatch-policy"
user = each.value.name
policy = jsonencode({
Version = "2012-10-17"
Statement = [
{
Action = [
"cloudwatch:GetMetricStatistics",
"cloudwatch:ListMetrics"
]
Effect = "Allow"
Resource = "*"
}
]
})
}
```
2. Reject the manual change and restore the declared state:
```bash theme={null}
terraform apply --auto-approve
```
3. Import the manually-created resource into Terraform so it becomes part of the managed state:
```bash theme={null}
terraform import aws_iam_user_policy manual_policy pablo:manual-cloudwatch-access
```
Whichever path you choose, keep code and infrastructure in sync to ensure reproducibility and reliability.
## Closing notes
This lesson covered IaC fundamentals for SREs: why IaC matters, tooling, best practices, testing and CI/CD, policy as code, an IAM automation example, and drift detection/remediation. Configuration management (Ansible, Chef, Puppet) is a complementary topic focused on ensuring consistent software and runtime configuration across fleets and should be studied alongside IaC for full-stack reliability.
## Links and references
* Terraform documentation: [https://www.terraform.io/docs](https://www.terraform.io/docs)
* tfsec: [https://github.com/aquasecurity/tfsec](https://github.com/aquasecurity/tfsec)
* tflint: [https://github.com/terraform-linters/tflint](https://github.com/terraform-linters/tflint)
* GitHub Actions docs: [https://docs.github.com/actions](https://docs.github.com/actions)
* Open Policy Agent (OPA): [https://www.openpolicyagent.org/](https://www.openpolicyagent.org/)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.