IAM Policy Debugging With IAM

Welcome back. This lesson explains practical techniques for debugging AWS IAM policies, with workflows, tools, and concrete examples to help you quickly identify and fix permission problems. Agenda

Common IAM issues and examples
Tools and workflows: Policy Simulator, CloudTrail, IAM Access Analyzer, Access Advisor
Policy evaluation sequence: identity vs resource policies, conditions, and explicit deny
Permission boundaries and cross-account debugging
Automating policy validation in CI/CD

Troubleshooting flow (high level)

Capture the failing API call and context (CloudTrail).
Recreate the failing action and analyze effective permissions (Policy Simulator, Access Analyzer).
Identify the policy or boundary that caused the denial (explicit deny, permission boundary, SCP).
Implement a fix, retest, and document the resolution.

Goal: find the effective principal, enumerate all applicable policies and constraints, propose a minimal change, test it in a controlled environment, and then apply the change in production if appropriate.

A slide showing a flowchart titled "IAM Policy Debugging: Finding and Fixing Permission Issues." It depicts components like IAM Policy, Policy Simulator, IAM Access Analyzer, CloudTrail, and CLI/SDK with arrows showing how they interact to find and recommend fixes for permission problems.

Example: S3 read-only policy (valid JSON)

This policy grants list and get permissions for a specific bucket and its objects. JSON does not support inline comments, so the example below is valid JSON that you can paste into the console or CLI.

Remember: JSON policy documents must be valid JSON (no inline comments). Use external documentation or separate comments when annotating policies.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "S3ReadOnlyAccess",
      "Effect": "Allow",
      "Action": [
        "s3:GetObject",
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::example-bucket",
        "arn:aws:s3:::example-bucket/*"
      ]
    }
  ]
}

This policy intentionally excludes write actions such as s3:PutObject. A common troubleshooting scenario: developer pipelines (CodeCommit/CodeBuild/CodeDeploy) fail with AccessDenied. To debug: reproduce the failing API call, inspect CloudTrail for the denied event, and use Policy Simulator to determine which policy denied the action. Common IAM issues (summary)

Category	Typical causes	Troubleshooting tips
Syntax & structure	Invalid ARN, malformed JSON	Validate policy JSON, run linter or JSON parser
Logical & semantic	Missing permissions, explicit Deny	Identify which statement denies or allows the action
Context & scope	Wrong principal, missing condition keys, permission boundary, SCP	Check identity, resource policy, conditions, and org-level SCPs

A slide titled "Common IAM Policy Issues" showing three numbered boxes: 1) Syntax and Structure (Invalid ARN, JSON parse error), 2) Logical and Semantic (Missing permission, Explicit deny), and 3) Context and Scope (Missing context key, Wrong principal). The boxes use colored arrow badges and there's a copyright notice for KodeKloud.

Policy evaluation sequence When AWS receives a request it evaluates policies in roughly this order:

Identity-based policies attached to the principal (user, group, or role).
Resource-based policies attached to the target resource (S3 bucket policy, KMS key policy, Lambda resource policy, etc.).
Context conditions used by either policy (IP address, MFA, VPC endpoint, tags, time-of-day, session tags).
Explicit Deny check: any explicit Deny in any applicable policy results in an overall Deny.

Notes:

By default, actions are implicitly denied unless an explicit Allow exists.
Service Control Policies (SCPs) and permission boundaries further restrict effective permissions:
- SCPs apply at the organization/OU level and can block actions across accounts.
- Permission boundaries limit what permissions an identity can obtain; effective permissions are the intersection of identity policy and permission boundary.

A stepped flowchart titled "Policy Evaluation Flow" showing boxes from "Request Received" through "Identity-Based Policy", "Resource-Based Policy", "Context Conditions Checked", and "Explicit Deny Check" to a final "Final Decision". The slide has a © Copyright KodeKloud notice in the corner.

Policy Simulator and permissive checks

Use the IAM Policy Simulator to predict whether a principal can perform a specific action with a given context (resource ARN, source IP, session tags).
For controlled testing, a permissive wildcard policy can help determine whether a denied action is caused by policy or by external constraints:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "*",
      "Resource": "*"
    }
  ]
}

Do not deploy wildcard Allow policies in production. Use them only in isolated test environments to confirm whether a denial is policy-related. The Policy Simulator does not evaluate organization-level SCPs or permission boundaries—verify those separately.

CloudTrail: source of truth for API failures

CloudTrail records API calls with outcome, error codes, requester identity, resources, and region. Use CloudTrail before changing policies to confirm the exact failing API and request context.
Example CloudTrail event (trimmed):

{
  "eventName": "ListObjectsV2",
  "errorCode": "AccessDenied",
  "errorMessage": "Access Denied",
  "userIdentity": {
    "type": "IAMUser",
    "arn": "arn:aws:iam::123456789012:user/Alice",
    "userName": "Alice"
  },
  "resources": [
    { "ARN": "arn:aws:s3:::example-bucket" }
  ],
  "awsRegion": "us-west-2"
}

Best practices:

Query CloudTrail with Athena for pattern searches.
Feed CloudTrail events into EventBridge for automated detection and remediation.
If you need object-level S3 data events, enable S3 data event logging explicitly.

IAM Access Analyzer

Access Analyzer inspects policies and generates findings for overly permissive or cross-account access. It can be used interactively in the console or invoked via API to validate templates and policies in CI/CD pipelines.

Practical debugging workflow (step-by-step)

Reproduce the issue. If not reproducible, gather CloudTrail, logs, request IDs, and any environment details.
Inspect the principal’s identity-based policies, attached groups, and permission boundaries.
Inspect resource-based policies on the target resource (bucket policy, KMS key policy, Lambda resource policy).
Simulate the exact action in the Policy Simulator with matching context (source IP, session tags, region).
Review CloudTrail for AccessDenied or other error codes to confirm the failing API name and resource.
Use IAM Access Analyzer to find mis-scoped or overly-permissive policies.
Apply a minimal fix, retest, and then document and automate the validation.

A flowchart titled "Debugging Workflow" that outlines steps for diagnosing permission issues (identify and reproduce issue, analyze policies, test with policy simulator, review CloudTrail logs) with decision diamonds for "Issue reproduced?" and "Missing permissions?".

IAM Access Advisor

Access Advisor reports which services a principal recently used. Use it to identify unused permissions to tighten policies and enforce least privilege.

Identity-based vs Resource-based policies

Identity-based policies: attached to users, groups, or roles; specify what that identity can do.
Resource-based policies: attached to resources such as S3 buckets, KMS keys, Lambda functions; specify which principals may access the resource.
Both can contain conditions. Any explicit Deny in either applicable policy will override Allows.

A presentation slide titled "Resource-Based Policy Debugging" containing a side-by-side table comparing Identity-Based and Resource-Based policies. It lists aspects like what they're attached to (user/group/role vs resource such as S3/KMS), what they specify, types, cross-account behavior, evaluation rules, and examples.

Permission boundaries vs SCPs

Permission boundaries: a maximum set of permissions an IAM principal can obtain. Effective permissions = identity policy ∩ permission boundary.
Service Control Policies (SCPs): organization/OU-level policies that restrict actions across accounts; SCPs apply before identity policies when determining effective permissions at the account scope.
Permission boundaries are account-level constructs; SCPs are managed at the organization level.

A presentation slide titled "Permission Boundary Debugging" showing a Venn diagram with two overlapping ovals: a blue "Permission Boundary" and an orange "Identity Policies." The overlap is labeled "Effective Permissions (Intersection)."

When permission boundaries are present

Compare the permission boundary policy and identity-based policies to compute effective permissions.
For testing, temporarily relax the permission boundary in a controlled (non-production) environment to confirm whether the boundary is the limiting factor.
Prefer service-specific permission boundary templates to consistently restrict permissions for particular services.

A vertical flowchart titled "Permission Boundary Debugging" that outlines steps from "Start" through comparing boundary vs identity policies, temporarily modifying boundaries for testing, analyzing effective permissions, and using service-specific boundary templates. It ends with an "Issue Resolved" oval at the bottom.

Automating policy validation in CI/CD

Integrate policy checks into your pipeline to detect permission or resource-policy issues before deployment.
Combine IaC template validation, IAM Access Analyzer checks, and configuration snapshots during the build stage.

A slide titled "Automating Policy Validation" showing a CI/CD pipeline flow: Source -> Build (CodeBuild) -> Policy Validation (IAM Access Analyzer API / AWS Config Rules) -> Test (Integration & Compliance) -> Deploy (Production).

Example CodeBuild buildspec: validate CloudFormation templates with cfn-policy-validator

version: 0.2

phases:
  install:
    runtime-versions:
      python: 3.8
    commands:
      - pip install cfn-policy-validator
  build:
    commands:
      - cfn-policy-validator validate --template-path ./template.json --region us-east-1

Cross-account access debugging Common root causes:

Trust relationship: the role in the trusting account must include the caller account (or principal) in its trust policy.
AssumeRole permissions: the calling principal must have sts:AssumeRole permission to initiate the assume-role call.
Resulting session permissions: after assume-role succeeds, the role’s identity policy and any resource policies may still block the target action.

A flowchart titled "Debugging Cross-Account Access" showing Account A (Trusted Account) and Account B (Trusting Account) linked by a trust relationship. It highlights an IAM role, permission and resource-based policies and labels common issues like trust relationship errors, AssumeRole permission issues, and missing resource policies.

Checklist for validating cross-account access

Step	What to check
Trust policy	Confirm the role’s trust relationship allows the calling account/principal
AssumeRole permission	Ensure the caller has `sts:AssumeRole` for the target role
External ID / tags	Validate `ExternalId` or session tags if required by the trust policy
Session & resource policies	Verify the assumed role’s identity policy and any resource-based policies for the desired action

Service-specific debugging tips

Service	Key checks
S3	Bucket policy, ACLs, object ownership settings, public access block, IAM policies
Lambda	Execution role permissions, resource-based function policy (who can invoke)
DynamoDB	Identity policies for table access, attribute-level restrictions
KMS	Key policy, grants, and IAM policy interactions (KMS evaluates key policy first)

A slide titled "Service-Specific IAM Debugging" showing AWS service icons for S3, Lambda, DynamoDB, and KMS. Each service box includes a short debugging tip (e.g., check path permissions and ACLs, execution role and resource policy, attribute-level permissions, and grant/key policy interaction).

Common AccessDenied examples (CloudTrail snippets)

Typical AccessDenied events you may see:

[
  { "eventName": "Invoke", "errorCode": "AccessDenied" },
  { "eventName": "GetObject", "errorCode": "AccessDenied" },
  { "eventName": "StartBuild", "errorCode": "AccessDenied" }
]

Troubleshooting checklist by scenario

Scenario	Quick checks
CI/CD pipeline failures	Verify CodeBuild/CodePipeline service roles, policies, and trust relationships
Lambda invocation failures	Check function resource policy (who can invoke) and execution role (what it can do)
Console vs API access	Confirm whether the user has programmatic (API) permissions vs only console access
S3 access denied	Inspect bucket policy, ACLs, IAM policies, object ownership, and public access block
Cross-account calls	Start with trust policy, then `sts:AssumeRole`, then session-level permissions

A slide titled "Common Debugging Scenarios" that lists AWS troubleshooting checks (CI/CD pipeline role permissions, Lambda invocation/execution roles, console vs API permission differences, S3 access policies, and cross-account trust/assume-role issues) with short remediation notes. The items are shown with circular icons and the slide is © KodeKloud.

Final notes and best practices

Start from evidence (CloudTrail). Reproduce if possible.
Simulate the exact action and context with the Policy Simulator.
Use IAM Access Analyzer and Access Advisor to find mis-scoped and unused permissions.
Inspect identity policies, resource policies, condition keys, permission boundaries, and SCPs.
Automate validation in CI/CD to catch policy issues early and help enforce least privilege.

Best practice: instrument policy validation into your CI/CD pipeline and maintain a short feedback loop—policy errors found early are simpler and safer to fix.

References and further reading

Thanks for listening — we’ll continue with more hands-on examples in the next lesson.

Introduction

Domain 1 - SDLC Automation

Domain 2 - Configuration Management and IaC

Domain 3 - Resilient Cloud Solutions

Domain 4 - Monitoring and Logging

Domain 5 - Incident and Event Response

Domain 6 - Security and Compliance

IAM Policy Debugging With IAM

Watch Video