Amazon Macie Watching S3 for PII

In this lesson you’ll learn how Amazon Macie helps discover, classify, and respond to sensitive data stored in Amazon S3 (for example, credit card numbers, Social Security numbers, and other personally identifiable information). We cover what Macie does, how it works, the finding lifecycle, managed and custom identifiers, multi-account deployment patterns, automated responses, a CloudFormation example, a sample remediation Lambda, and optimization guidance.

Amazon Macie has evolved into “Macie Classic” → “Macie (Macie2)” APIs. When implementing IaC or EventBridge rules, verify whether your account/region uses aws.macie or aws.macie2 and consult the official AWS Macie documentation for the latest resource types and schemas.

Why use Amazon Macie?

Macie addresses the challenge of detecting sensitive information in S3 so you can monitor, report, and remediate according to regulatory and security requirements. It automates discovery, classification, and generates findings that feed into security workflows and reporting.

A diagram showing Amazon Macie scanning multiple S3 buckets and analyzing their contents. Macie identifies sensitive data categories like PII, financial data, and credentials/keys, with an AWS Certified DevOps Engineer badge shown on the right.

High-level capabilities

Macie’s main functions include:

Automated sensitive data discovery and classification inside S3.
Tagging and categorization of data for policy and compliance workflows.
Generating findings (data findings and configuration findings) with details such as affected resources, confidence scores, and severity.
Integrations with Security Hub, EventBridge, CloudWatch, and other automation targets.

A slide titled "Macie – Overview" showing a flowchart of Primary Functions with four boxes: Automated sensitive data discovery, Data classification and tagging, Security posture evaluation for S3, and Compliance monitoring and reporting. The boxes are connected under a single "Primary Functions" heading.

Core components and console

At a high level, Macie:

Enumerates S3 buckets (inventory).
Samples and analyzes object content (sensitive-data detection).
Produces findings (including configuration issues like public access).

The Macie console provides a central dashboard for findings, alerts, compliance reports, and job management — but production automation usually routes findings to EventBridge or Security Hub.

A presentation slide titled "Macie – Overview." It shows a "Key Components" box connected to four boxes labeled "S3 bucket inventory," "Sensitive data discovery jobs," "Managed data identifiers," and "Custom data identifiers."

A simple flow diagram titled "Macie – Overview" showing a blue "Macie Console" box connected to four feature boxes: Dashboard overview, Findings & alerts, Compliance reports, and Job management. The layout illustrates the console's main functions.

How Macie works (discovery → detection → findings)

Discovery jobs sample or scan objects, Macie applies managed and custom identifiers to detect sensitive content, and findings are generated. Findings can be data findings (e.g., PII, credentials) or configuration findings (e.g., public buckets). Findings include contextual metadata used for triage and remediation.

A presentation slide titled "How Macie Works" showing a left-to-right flowchart: S3 Discovery Process → Sensitive Data Detection → Finding Generation. A note beside the final box mentions S3 bucket policy findings (public access, etc.), and a © KodeKloud copyright appears at the bottom.

Findings and lifecycle

Findings typically follow a lifecycle:

Generation and notification (Macie creates a finding).
Investigation and triage (security team or automation reviews the finding).
Remediation and resolution (manual action or automated remediations are applied).
Suppression or archiving (resolved or noise findings are suppressed/archived).

Treat findings according to severity and context — for example, immediately remediate high-severity public buckets containing PII; schedule reviews for low-severity findings.

A slide titled "Macie Findings" showing a "Finding Lifecycle" flowchart. It lists four stages: Generation and notification; Investigation and triage; Remediation and resolution; and Suppression or archiving.

Managed vs. custom data identifiers

Use managed identifiers for common patterns (SSN, credit cards, passports). For organization-specific formats, create custom data identifiers using regular expressions, optional keyword proximity, and confidence thresholds to lower false positives.

A presentation slide titled "Managed and Custom Data Identifiers" comparing features of managed data identifiers (pre-configured categories, built-in detection, updates) and custom identifiers (regex patterns, keyword context, configurable thresholds). Below is an "Implementation Strategy" box with brief recommendations for using and combining identifiers and ongoing testing.

Common identifier categories

Category	Examples	When to use
PII	SSN, passport numbers	Use managed rules for standard PII formats
Financial	Credit card numbers, bank account numbers	Managed rules exist; custom regex for nonstandard formats
Credentials	API keys, passwords	Use both managed and custom identifiers; validate via sampling
Health	Medical records, PHI	Use custom identifiers if formats are specific to your org

A slide titled "Managed and Custom Data Identifiers – Identifier Table." It shows a three-column table of identifier types (PII, Financial, Credentials, Health) with examples (SSN, passport number, credit card, passwords, medical records) and brief descriptions.

Custom identifiers configuration is simple: provide a regex, set the confidence threshold, optionally set keyword proximity or maximum match distance, then save and apply.

A slide titled "Managed and Custom Data Identifiers" showing a flow diagram for "Custom Data Identifier Configuration." It contains connected boxes labeled "Regex pattern input," "Confidence threshold slider/input," "Keyword proximity options," and "Save/Apply buttons."

When creating custom identifiers, test regex patterns and keyword proximity on representative sample data. Overly broad regexes increase false positives; overly strict patterns miss true positives. Always validate in a staging environment before wide deployment.

Multi-account deployment (AWS Organizations)

For enterprise environments, integrate Macie with AWS Organizations, designate a delegated administrator account, and enable automated enrollment for member accounts. Centralized configuration, standardized policies, and aggregated findings make scaling Macie manageable.

A slide titled "Multi-Account Macie Implementation" showing a blue "Management Strategies" box linked to three boxes labeled "Standardized configuration," "Centralized finding aggregation," and "Cross-account remediation."

A diagram titled "Multi-Account Macie Implementation" showing an Administrator Account on the left connecting to two Member Accounts, which feed into AWS Organizations on the right via an "Automated Enrollment Flow."

Recommended model: Organizations integration with a delegated admin account that aggregates findings centrally and triggers cross-account remediation workflows.

Automated response patterns

Route Macie findings to EventBridge and match attributes from the finding payload. EventBridge rules can trigger Lambda, Step Functions, or Systems Manager Automation documents to remediate, quarantine, notify, or create tickets.

A flowchart titled "Automated Response With Macie" showing a remediation workflow from an Amazon Macie finding through EventBridge, AWS Lambda, remediation actions, and status tracking (with optional Systems Manager and notification feedback). The diagram uses AWS icons and labeled boxes to illustrate the event flow from detection to resolution.

When matching EventBridge rules, ensure you use the correct event source (aws.macie or aws.macie2) and conditionally match attributes in the finding payload to avoid false triggers.

Sample CloudFormation fragment (IaC)

The following CloudFormation YAML fragment demonstrates enabling a Macie session, creating a custom data identifier, and adding a findings filter. Validate resource types against the current CloudFormation Macie2 schema in your account/region.

Resources:
  MacieSession:
    Type: AWS::Macie::Session
    Properties:
      Status: ENABLED
      FindingPublishingFrequency: ONE_HOUR  # Options: FIFTEEN_MINUTES | ONE_HOUR | SIX_HOURS

  PCIIdentifier:
    Type: AWS::Macie::CustomDataIdentifier
    Properties:
      Name: PCI-Data-Detector
      Regex: "(?i)\\b(4\\d{12}(\\d{3})?|5[1-5]\\d{14})\\b"  # Example credit card regex
      MaximumMatchDistance: 10

  HighRiskFilter:
    Type: AWS::Macie::FindingsFilter
    Properties:
      Action: ARCHIVE
      FindingCriteria:
        criterion:
          severity:
            Eq: ["High"]

Note: resource names, property names, and top-level resource types may differ between Macie Classic and Macie2. Always check the AWS CloudFormation documentation before deployment.

Example automated remediation Lambda

A common automated remediation pattern is EventBridge → Lambda. The Lambda inspects the finding payload, determines the affected S3 bucket, and performs remediation actions such as enabling default SSE and applying a restrictive bucket policy.

# lambda_function.py
import json
import boto3

def lambda_handler(event, context):
    # Extract detail from the EventBridge event. The exact path may vary based on EventBridge transforms.
    detail = event.get("detail", {})
    bucket_name = detail.get("resource", {}).get("affectedResources", [{}])[0].get("s3Bucket", {}).get("name")
    region = detail.get("awsRegion", "us-east-1")

    if not bucket_name:
        return {
            "statusCode": 400,
            "body": json.dumps("No bucket name found in event")
        }

    try:
        encrypt_bucket(bucket_name, region)
        apply_restrictive_policy(bucket_name, region)
        return {
            "statusCode": 200,
            "body": json.dumps("Remediation completed successfully")
        }
    except Exception as e:
        return {
            "statusCode": 500,
            "body": json.dumps(f"Remediation failed: {e}")
        }

def encrypt_bucket(bucket_name, region):
    s3 = boto3.client("s3", region_name=region)
    s3.put_bucket_encryption(
        Bucket=bucket_name,
        ServerSideEncryptionConfiguration={
            "Rules": [
                {
                    "ApplyServerSideEncryptionByDefault": {
                        "SSEAlgorithm": "AES256"
                    },
                    "BucketKeyEnabled": True
                }
            ]
        }
    )
    print(f"Encryption enabled for bucket {bucket_name}")

def apply_restrictive_policy(bucket_name, region):
    s3 = boto3.client("s3", region_name=region)
    policy = {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "RestrictedAccess",
                "Effect": "Deny",
                "Principal": "*",
                "Action": "s3:GetObject",
                "Resource": f"arn:aws:s3:::{bucket_name}/*",
                "Condition": {
                    "StringNotEquals": {
                        "aws:PrincipalTag/DataAccessRole": "PII-Approved"
                    }
                }
            }
        ]
    }
    s3.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(policy))
    print(f"Restrictive policy applied to bucket {bucket_name}")

Runtime recommendation: python3.8 or later (choose a supported runtime). Set the Lambda timeout based on expected remediation time (for example, 300 seconds). Ensure the Lambda execution role has least-privilege IAM permissions for s3:PutBucketEncryption, s3:PutBucketPolicy, and read access for the EventBridge/Macie payloads.

A slide titled "Automated Remediation Example" about automating S3 encryption with Lambda. The diagram shows an AWS Macie session using a custom data identifier to trigger a remediation Lambda that enforces bucket encryption, while findings are saved to a findings bucket.

Configuring discovery jobs

When you create discovery jobs, choose:

Job type: one-time, scheduled, or continuous.
Bucket scope: all buckets, specific buckets, or filtered buckets.
Data identifiers: managed identifiers, custom identifiers, or both.

Complete the job configuration and monitor results to fine-tune frequency and scope.

A flowchart titled "Optimizing Macie Discovery" that outlines the steps to configure an AWS Macie discovery job. It shows decision points for job type (one-time or scheduled), bucket scope (all or specific), and whether to use custom data identifiers, leading to "Complete Job Configuration."

Performance and cost optimization

Consider these optimizations:

Use targeted or sample-based discovery for large datasets to reduce cost.
Set job concurrency limits and leverage regional deployment strategies.
Parallelize across accounts carefully to avoid throttling and high costs.
Tune discovery frequency and scope to match risk appetite.

A slide titled "Optimizing Macie Discovery" with a blue "Performance Considerations" header. Three connected boxes list "Job concurrency limits," "Regional deployment strategy," and "Account-level parallelism."

A bar chart titled "Optimizing Macie Discovery" comparing estimated AWS Macie discovery costs for four methods: Targeted discovery (lowest), Sample-based, Incremental, and Bucket inclusion/exclusion (highest). The y-axis shows cost values up to about 1200.

Use CloudWatch dashboards to visualize job run status, findings trend by severity, and S3 security posture (number of encrypted, public, or unencrypted buckets).

A CloudWatch dashboard titled "Monitoring and Metrics for Macie" showing a bar chart of sensitive data findings by severity and a donut chart of discovery job status. Below is an S3 security posture summary with totals for buckets, encrypted, public, and unencrypted buckets.

Common issues & troubleshooting

Missing findings: often caused by insufficient Macie permissions to access S3 objects or incorrect Organizations integration.
False positives: usually due to overly broad custom identifiers; refine regex and keyword proximity.
High cost: frequent full scans across many buckets; reduce frequency or scope.
Multi-account problems: frequently tied to Organizations integration or cross-account permissions.

References and further reading

AWS Macie documentation: https://docs.aws.amazon.com/macie/latest/userguide/what-is-macie.html
Amazon EventBridge: https://docs.aws.amazon.com/eventbridge/latest/userguide/what-is-amazon-eventbridge.html
AWS CloudFormation documentation: https://docs.aws.amazon.com/cloudformation/index.html
AWS Lambda best practices: https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html

That covers the overview, workflow, customization, automated remediation patterns, sample IaC, and optimization considerations for Amazon Macie. See you in the next lesson.

Introduction

Domain 1 - SDLC Automation

Domain 2 - Configuration Management and IaC

Domain 3 - Resilient Cloud Solutions

Domain 4 - Monitoring and Logging

Domain 5 - Incident and Event Response

Domain 6 - Security and Compliance

Amazon Macie Watching S3 for PII

Why use Amazon Macie?

High-level capabilities

Core components and console

How Macie works (discovery → detection → findings)

Findings and lifecycle

Managed vs. custom data identifiers

Common identifier categories

Multi-account deployment (AWS Organizations)

Automated response patterns

Sample CloudFormation fragment (IaC)

Example automated remediation Lambda

Configuring discovery jobs

Performance and cost optimization

Common issues & troubleshooting

References and further reading

Watch Video

​Why use Amazon Macie?

​High-level capabilities

​Core components and console

​How Macie works (discovery → detection → findings)

​Findings and lifecycle

​Managed vs. custom data identifiers

​Common identifier categories

​Multi-account deployment (AWS Organizations)

​Automated response patterns

​Sample CloudFormation fragment (IaC)

​Example automated remediation Lambda

​Configuring discovery jobs

​Performance and cost optimization

​Common issues & troubleshooting

​References and further reading

Watch Video

Why use Amazon Macie?

High-level capabilities

Core components and console

How Macie works (discovery → detection → findings)

Findings and lifecycle

Managed vs. custom data identifiers

Common identifier categories

Multi-account deployment (AWS Organizations)

Automated response patterns

Sample CloudFormation fragment (IaC)

Example automated remediation Lambda

Configuring discovery jobs

Performance and cost optimization

Common issues & troubleshooting

References and further reading