SNSSQS Demo

In this lesson you’ll build a simple, scalable serverless pipeline using Amazon S3 → Amazon SNS → Amazon SQS → AWS Lambda. This pattern is ideal for processing uploaded media files (for example, converting videos and generating thumbnails) while keeping processing decoupled and resilient. High-level flow:

A user uploads a video (MP4) to an S3 bucket.
S3 publishes an event to an SNS topic (Video Uploaded).
SNS fans out the event to two SQS queues:
- video-processing → triggers a Lambda that converts the video (e.g., to HLS) and writes to the processed-videos bucket.
- thumbnail-processing → triggers a Lambda that generates thumbnails and writes to the thumbnails bucket.

This article walks through creating the S3 buckets, SNS topic, SQS queues, Lambda functions, wiring S3 event notifications to SNS, testing the end-to-end flow, and cleaning up resources. Buckets created in advance:

raw-videos-kodekloud
processed-videos-kodekloud
thumbnails-kodekloud

A screenshot of the AWS S3 Management Console showing a list of five S3 buckets (including thumbnails-kodekloud, raw-videos-kodekloud, processed-videos-kodekloud) and a green banner confirming the successful creation of the "thumbnails-kodekloud" bucket. The table displays each bucket's AWS region, access settings, and creation date.

These are plain buckets with no additional configuration yet — we will add the necessary S3 event notification and IAM permissions during the demo.

Quick resource overview

Resource	Purpose
S3 `raw-videos-kodekloud`	Upload source video files (input)
S3 `processed-videos-kodekloud`	Store processed outputs (HLS playlist + segments)
S3 `thumbnails-kodekloud`	Store generated thumbnails
SNS topic `video-uploaded`	Fan-out notification on object creation
SQS `video-processing`	Queue for video conversion workers
SQS `thumbnail-processing`	Queue for thumbnail generator workers
Lambda functions	Workers subscribed to SQS queues

Open the SNS console → Topics → Create topic.
Name the topic (for example: video-uploaded). Use the Standard topic type unless you need FIFO semantics or strict ordering.
Review options (encryption, access policy). For the demo keep defaults and create the topic.

A screenshot of the Amazon Web Services SNS console showing a newly created topic named "video-uploaded" with its details (ARN, owner, type) and subscription options. A green success banner at the top confirms the topic was created and there are buttons to edit, delete, or publish messages.

Note: later you will add a policy statement to this topic to allow S3 to publish events. Create two Standard SQS queues:

video-processing (Standard)
thumbnail-processing (Standard)

Default settings (visibility timeout, delivery delay, retention) are fine for this demo. If you need cross-account publishers or consumers, add the appropriate access policy later. Example default owner-only policy structure for an SQS queue (replace account ID and queue ARN as appropriate):

{
  "Version": "2012-10-17",
  "Id": "__default_policy_ID",
  "Statement": [
    {
      "Sid": "__owner_statement",
      "Effect": "Allow",
      "Principal": {
        "AWS": "841860927337"
      },
      "Action": [
        "SQS:*"
      ],
      "Resource": "arn:aws:sqs:us-east-1:841860927337:video-processing"
    }
  ]
}

After creating each queue, subscribe it to the SNS topic:

SNS topic → Subscriptions → Create subscription
Protocol: Amazon SQS
Endpoint: select the queue ARN

SNS will then fan out published messages to both queues and SQS will store them until a consumer processes them.

A screenshot of the AWS Management Console showing the Amazon SQS "Create queue" page with a queue named "video-processing" and configuration fields. It shows settings like visibility timeout (30 seconds), message retention (4 days), maximum message size (256 KB), delivery delay, and encryption options.

You can quickly verify subscription behavior by publishing a test message to the SNS topic (next section) and observing message counts in the SQS console.

A screenshot of the AWS Simple Queue Service (SQS) console listing two queues: "thumbnail-processing" and "video-processing." Both are Standard queues with one message available each and show creation times and Amazon SQS encryption.

Publish a test message to SNS

From the SNS topic page, use Publish message. The message body should include the information your workers require (for example, S3 bucket name and object key). Use JSON to keep messages structured. Example message body:

{
  "bucket": "raw-videos-kodekloud",
  "key": "b415c94e-de85-4f6a-949c-2eb2e93bf830"
}

After publishing:

Both SQS queues should show the message available.
Messages remain in the queues until a consumer (Lambda) receives them.

Publish multiple messages to verify counts increment appropriately.

Create the Lambda functions

We need two Lambda functions:

video-processing — downloads the S3 object, runs ffmpeg to produce HLS output, uploads to processed-videos-kodekloud.
thumbnail-processing — downloads the S3 object (or extracts a frame), generates thumbnails, and uploads to thumbnails-kodekloud.

Create (or reuse) an IAM role for the Lambdas with these permissions:

SQS receive permissions: Use AWSLambdaSQSQueueExecutionRole or equivalent.
S3 permissions: s3:GetObject, s3:PutObject for the relevant buckets (prefer least privilege in production).
CloudWatch Logs: AWSLambdaBasicExecutionRole.

For a demo you can attach AmazonS3FullAccess, but in production scope down permissions to specific bucket ARNs. Example role name: Lambda-SQS-S3.

Screenshot of the AWS Lambda "Create function" web console. It shows options to author a function (from scratch, blueprint, or container image) and form fields for function name, runtime (Node.js 18.x) and architecture selection.

SQS event structure and Lambda handler

When Lambda is triggered by SQS, the event contains a Records array. Each record’s body is a string that contains the SNS notification JSON. SNS wraps your original message inside the Message field (as a JSON-encoded string). Typical shape (simplified):

{
  "Records": [
    {
      "messageId": "953cc77f-7de4-40b0-8413-c96c18049d98",
      "body": "{\n  \"Type\": \"Notification\",\n  \"MessageId\": \"d341b75b-...\",\n  \"TopicArn\": \"arn:aws:sns:us-east-1:123456789012:video-uploaded\",\n  \"Message\": \"{\\n  \\\"bucket\\\": \\\"raw-videos-kodekloud\\\",\\n  \\\"key\\\": \\\"b415c94e-...\\\"\\n}\",\n  \"Timestamp\": \"2023-10-12T03:22:05.836Z\"\n}",
      "eventSource": "aws:sqs",
      "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:video-processing",
      "awsRegion": "us-east-1"
    }
  ]
}

To extract your original payload:

Parse record.body to an object.
Parse the Message property (a JSON string) to obtain the original message.

Minimal Node.js handler showing this parsing:

export const handler = async (event) => {
  for (const record of event.Records) {
    // SNS notification JSON is stored in record.body
    const snsNotification = JSON.parse(record.body);
    // The original message is in snsNotification.Message (stringified JSON)
    const message = JSON.parse(snsNotification.Message);

    console.log("Received message:", message);
    // Use message.bucket and message.key for S3 operations
  }

  return {
    statusCode: 200,
    body: JSON.stringify("Processed batch"),
  };
};

After deploying the Lambda, add the SQS trigger:

Configure batch size (1..10 for Standard queues). Larger batches reduce invocations but require the handler to process multiple messages.
Optionally configure batch window to allow more messages to be batched.

A screenshot of the AWS Lambda console "Add trigger" panel for an SQS event source, showing options like "Activate trigger," "Batch size" (set to 10), "Batch window," and "Maximum concurrency." The page includes an "Add" button at the bottom to create the trigger.

After wiring the trigger, publish test messages to SNS and check the Lambda CloudWatch logs to confirm message parsing.

A screenshot of the AWS Amazon SQS console showing two queues — "thumbnail-processing" (2 messages available) and "video-processing" (0 messages) — with type, creation time, and encryption details. Browser tabs and the AWS top navigation bar are visible at the top.

Concise handler example that logs bucket and key:

export const handler = async (event) => {
  for (const record of event.Records) {
    const snsNotification = JSON.parse(record.body);
    const message = JSON.parse(snsNotification.Message);
    const bucket = message.bucket;
    const key = message.key;

    console.log(`Bucket: ${bucket}, Key: ${key}`);
    // Add S3 getObject and processing logic here
  }
};

Video processing Lambda (example implementation)

Below is an example Lambda using AWS SDK v3 and fluent-ffmpeg to download a video, create HLS output, and upload generated files to the processed bucket. This is a focused example to illustrate structure — production code should include more robust error handling, streaming, and /tmp cleanup.

import fs from "fs";
import path from "path";
import ffmpeg from "fluent-ffmpeg";
import ffmpegInstaller from "@ffmpeg-installer/ffmpeg";
import { S3Client, GetObjectCommand, PutObjectCommand } from "@aws-sdk/client-s3";

ffmpeg.setFfmpegPath(ffmpegInstaller.path);

export const handler = async (event) => {
  const client = new S3Client({});

  for (const record of event.Records) {
    const snsNotification = JSON.parse(record.body);
    const message = JSON.parse(snsNotification.Message);
    const bucket = message.bucket;
    const key = message.key;

    console.log(`Processing ${bucket}/${key}`);

    // Download object
    const getCommand = new GetObjectCommand({ Bucket: bucket, Key: key });
    const getResponse = await client.send(getCommand);
    const tmpInputPath = `/tmp/input-${Date.now()}.mp4`;
    const writeStream = fs.createWriteStream(tmpInputPath);
    await new Promise((resolve, reject) => {
      getResponse.Body.pipe(writeStream)
        .on("finish", resolve)
        .on("error", reject);
    });

    // Process with ffmpeg (example: create HLS playlist)
    const outputDir = `/tmp/output-${Date.now()}`;
    fs.mkdirSync(outputDir);
    const outputM3U8 = path.join(outputDir, "output.m3u8");

    await new Promise((resolve, reject) => {
      ffmpeg(tmpInputPath)
        .addOptions([
          "-profile:v baseline",
          "-level 3.0",
          "-start_number 0",
          "-hls_time 10",
          "-hls_list_size 0",
          "-f hls",
        ])
        .output(outputM3U8)
        .on("end", resolve)
        .on("error", reject)
        .run();
    });

    // Upload generated files to processed bucket
    const processedBucket = "processed-videos-kodekloud"; // replace with your bucket
    const files = fs.readdirSync(outputDir);
    for (const file of files) {
      const filePath = path.join(outputDir, file);
      const fileStream = fs.createReadStream(filePath);
      const putCommand = new PutObjectCommand({
        Bucket: processedBucket,
        Key: `${path.basename(key, path.extname(key))}/${file}`,
        Body: fileStream,
        ContentType: file.endsWith(".m3u8") ? "application/vnd.apple.mpegurl" : "video/MP2T",
      });
      await client.send(putCommand);
    }

    // Cleanup temp files (omitted for brevity)
  }

  return {
    statusCode: 200,
    body: JSON.stringify("Processing complete"),
  };
};

Lambda environments do not include ffmpeg by default. Provide a working ffmpeg binary via a Lambda layer or use a container image runtime that bundles ffmpeg. Also ensure the Lambda role has s3:PutObject permission for the processed-videos-kodekloud bucket.

Thumbnail-processing Lambda

Create the thumbnail Lambda using the same role (Lambda-SQS-S3) or a role with equivalent S3 and CloudWatch permissions. Add thumbnail-processing SQS as the trigger. Implement logic to download the input object (or stream a frame), generate thumbnails (using ffmpeg, sharp, or similar), and upload to thumbnails-kodekloud.

Screenshot of the AWS Lambda "Add trigger" page showing the Trigger configuration panel with "sqs" typed into the source dropdown and the SQS option highlighted. The Cancel and Add buttons are visible at the bottom.

Configure S3 to publish events to SNS

You have two options to notify SNS when a new object is created:

Have your upload client publish a message to SNS via the AWS SDK.
Configure an S3 Event Notification to publish to SNS automatically when new objects are created.

To configure S3 Events:

Go to S3 → Bucket (raw-videos-kodekloud) → Properties → Event notifications → Create event notification.
Name: video-uploaded
Optionally add Prefix/Suffix filters (for example filter by path or file extension).
Event types: choose “All object create events” (PUT/POST/COPY/Multipart Upload).
Destination: SNS topic → select video-uploaded.

S3 must be allowed to publish to the SNS topic. Add a topic policy statement that allows s3.amazonaws.com to call SNS:Publish for your topic only for the desired bucket and account. Example SNS topic policy statement (replace values with your ARNs and account ID):

{
  "Sid": "AllowS3Publish",
  "Effect": "Allow",
  "Principal": {
    "Service": "s3.amazonaws.com"
  },
  "Action": [
    "SNS:Publish"
  ],
  "Resource": "arn:aws:sns:us-east-1:841860927337:video-uploaded",
  "Condition": {
    "ArnLike": {
      "aws:SourceArn": "arn:aws:s3:::raw-videos-kodekloud"
    },
    "StringEquals": {
      "aws:SourceAccount": "841860927337"
    }
  }
}

Add the statement into the topic’s Access Policy (Topic → Edit → Access policy). Insert only the statement object into the existing Statement array — do not overwrite the whole policy unless you intend to.

When adding the statement to the topic policy, ensure you only add the policy element (the statement object) into the existing policy’s Statement array. Do not duplicate top-level Version or Id keys if they already exist.

If S3 cannot publish to the topic, creating the S3 notification may fail or the messages may never appear in SNS/SQS. Verify the topic ARN, bucket ARN, and account ID in the policy.

The screenshot shows the AWS S3 console's notification configuration page with fields for Prefix ("images/") and Suffix (".jpg"). The Event types section is visible with "All object create events" checked and other object create/remove options listed.

Test the end-to-end flow

Confirm both SQS queues are empty.
Upload a small test video file to raw-videos-kodekloud (via S3 console or CLI).
S3 event → SNS topic video-uploaded → SNS fans out to both SQS queues → each Lambda processes its queue.

Verify results:

HLS files (playlist .m3u8 + .ts segments) in processed-videos-kodekloud under a generated ID folder.
Thumbnails in thumbnails-kodekloud under a matching ID folder.

A screenshot of the AWS S3 console showing the Properties tab for an S3 bucket named "raw-videos-kodekloud" with bucket overview details (region, ARN, creation date) and a green success banner about a created event notification.

After processing, you should see HLS output files (playlist + segments):

A screenshot of the Amazon S3 web console showing a folder (ID ee33fc58-...) with three objects listed: output.m3u8 and two .ts files. The table shows their types, sizes and last-modified timestamps.

And the generated thumbnails should be present in the thumbnails bucket (open an image to verify the screenshot).

Cleanup

To avoid ongoing charges delete the demo resources when finished:

Delete SQS queues.
Delete SNS topic.
Delete Lambda functions.
Empty and delete S3 buckets (you must empty before deletion).

A screenshot of the AWS S3 Management Console showing a list of S3 buckets with their names, AWS regions, access settings, and creation dates. The toolbar with actions like Create bucket, Delete, and Copy ARN is visible.

A screenshot of the AWS S3 console showing the "Delete bucket" confirmation for the bucket "thumbnails-kodekloud". It shows warning text about irreversible deletion and an input field to type the bucket name to confirm.

That completes the walkthrough. You now have a working serverless pattern that uses S3 → SNS → SQS → Lambda to process uploaded videos and produce derived assets (processed HLS streams and thumbnails).

Introduction

Services - Networking

Services - Storage

Services - Compute

Services - Database

Services - Application Integration

Services - Data and ML

Services - Migration and Transfer

Services - Management and Governance

Services - Security

Bringing it all together

Designing for Security

Designing for Reliability

Designing for Performance

Designing for Cost-Optimization

Applying your Design Skills

Quick resource overview

Publish a test message to SNS

Create the Lambda functions

SQS event structure and Lambda handler

Video processing Lambda (example implementation)

Thumbnail-processing Lambda

Configure S3 to publish events to SNS

Test the end-to-end flow

Cleanup

Links and references

Watch Video

​Quick resource overview

​Create the SNS topic

​Create the SQS queues and subscribe them to SNS

​Publish a test message to SNS

​Create the Lambda functions

​SQS event structure and Lambda handler

​Video processing Lambda (example implementation)

​Thumbnail-processing Lambda

​Configure S3 to publish events to SNS

​Test the end-to-end flow

​Cleanup

​Links and references

Watch Video

Quick resource overview

Create the SNS topic

Create the SQS queues and subscribe them to SNS

Publish a test message to SNS

Create the Lambda functions

SQS event structure and Lambda handler

Video processing Lambda (example implementation)

Thumbnail-processing Lambda

Configure S3 to publish events to SNS

Test the end-to-end flow

Cleanup

Links and references