Demo Setting up SNSSQS to send Messages

• A user uploads a raw video to an S3 bucket (e.g., MP4).
• S3 publishes an event to an SNS topic (for example, video-uploaded).
• SNS fans out the notification to two SQS queues:
  • video-processing queue — for transcoding to a streaming format (HLS) and storing output in the processed-videos bucket.
  • thumbnail-processing queue — for extracting still frames and storing them in the thumbnails bucket.
• Two Lambda functions poll the queues, read the original object from S3, perform work, then write results back to their respective S3 buckets.

​

Prerequisites

​

Mapping: service responsibilities

​

1) Create an SNS topic

• Name: video-uploaded (or your preferred name)
• Type: Standard — ordering is not required for this demo
• Leave encryption and other settings at defaults for the demo

​

2) Create two SQS queues and subscribe them to SNS

• video-processing
• thumbnail-processing

• Subscription protocol: Amazon SQS
• Select the video-uploaded SNS topic

​

3) Create Lambda functions that process SQS messages

• video-processing — downloads the original object from S3, transcodes to HLS (e.g., with ffmpeg), uploads the HLS outputs to processed-videos-kodekloud.
• thumbnail-processing — downloads the original object, extracts thumbnails, and uploads them to thumbnails-kodekloud.

• S3 read/write as required (for demo you can use AmazonS3FullAccess, but scope down to least privilege in production)
• CloudWatch Logs (Lambda managed policy handles logging)

• Choose the appropriate queue per function (video-processing ↔ queue, thumbnail-processing ↔ queue)
• Batch size: how many messages delivered per invocation (demo uses 1; production may batch)
• Batch window: how long to wait to collect messages for a batch (0s for immediate invocation)

// javascript
export const handler = async (event) => {
  console.log(JSON.stringify(event, null, 2));
  return {
    statusCode: 200,
    body: JSON.stringify('OK'),
  };
};

• event.Records — array of SQS messages (length equals number of messages in batch)
• event.Records[n].body — contains the SNS notification as a JSON string
  • Parse this string to get the SNS envelope, then parse Message to get the original payload (often JSON)

// javascript
export const handler = async (event) => {
  for (const record of event.Records) {
    // body is an SNS notification (string)
    const snsNotification = JSON.parse(record.body);
    // Message is the original application payload (string)
    const message = JSON.parse(snsNotification.Message);

    console.log('Received message:', message);
    // message.bucket and message.key contain the S3 bucket and key
    // Proceed to get the object from S3 and process it
  }

  return {
    statusCode: 200,
    body: JSON.stringify('Processed'),
  };
};

• Parse the message to obtain the bucket and key
• GetObject from S3
• Run ffmpeg (or use a media processing service) to convert to HLS
• Upload M3U8 and TS segment files back to the processed bucket

// javascript
import fs from "fs";
import ffmpeg from "fluent-ffmpeg";
import ffmpegInstaller from "@ffmpeg-installer/ffmpeg";
import {
  GetObjectCommand,
  S3Client,
  PutObjectCommand
} from "@aws-sdk/client-s3";

ffmpeg.setFfmpegPath(ffmpegInstaller.path);

export const handler = async (event) => {
  const s3 = new S3Client({});

  for (const record of event.Records) {
    const snsNotification = JSON.parse(record.body);
    const message = JSON.parse(snsNotification.Message);
    console.log("Processing message:", message);

    const getCommand = new GetObjectCommand({
      Bucket: message.bucket,
      Key: message.key,
    });

    try {
      const response = await s3.send(getCommand);
      const filePath = `/tmp/${message.key.split("/").pop()}`;
      const writeStream = fs.createWriteStream(filePath);
      await new Promise((resolve, reject) => {
        response.Body.pipe(writeStream)
          .on("finish", resolve)
          .on("error", reject);
      });

      const outputDir = `/tmp/output-${Date.now()}`;
      fs.mkdirSync(outputDir, { recursive: true });

      await new Promise((resolve, reject) => {
        ffmpeg(filePath)
          .outputOptions([
            "-profile:v baseline",
            "-level 3.0",
            "-start_number 0",
            "-hls_time 10",
            "-hls_list_size 0",
            "-f hls"
          ])
          .output(`${outputDir}/output.m3u8`)
          .on("end", resolve)
          .on("error", reject)
          .run();
      });

      const files = fs.readdirSync(outputDir);
      for (const filename of files) {
        const body = fs.readFileSync(`${outputDir}/${filename}`);
        const putCommand = new PutObjectCommand({
          Bucket: "processed-videos-kodekloud",
          Key: `${message.key}/${filename}`,
          Body: body,
        });
        await s3.send(putCommand);
      }

      console.log("Processing complete for", message.key);
    } catch (err) {
      console.error("Error processing message", err);
      throw err;
    }
  }

  return {
    statusCode: 200,
    body: JSON.stringify("Done"),
  };
};

​

4) Configure S3 to publish events to the SNS topic

• Name: video-uploaded
• Event types: All object create events (s3:ObjectCreated:*)
• Destination: SNS topic → select video-uploaded

• Amazon S3 Event Notifications: https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html

{
  "Version": "2012-10-17",
  "Id": "example-ID",
  "Statement": [
    {
      "Sid": "AllowS3ToPublish",
      "Effect": "Allow",
      "Principal": {
        "Service": "s3.amazonaws.com"
      },
      "Action": [
        "SNS:Publish"
      ],
      "Resource": "arn:aws:sns:us-east-1:841860927337:video-uploaded",
      "Condition": {
        "ArnLike": {
          "aws:SourceArn": "arn:aws:s3:::raw-videos-kodekloud"
        },
        "StringEquals": {
          "aws:SourceAccount": "841860927337"
        }
      }
    }
  ]
}

• Open the SNS topic → Edit → Access policy → add the statement to the policy’s Statement array and save.
• Return to the S3 event notification and save it. If saving fails, check the topic policy for invalid JSON, wrong ARNs, or account IDs.

​

5) Test the end-to-end flow

1. Upload a short video to raw-videos-kodekloud (Console, API, or CLI).
2. S3 emits an object-created event → SNS receives it → SNS fans out to both SQS queues.
3. Lambda functions subscribed to the queues pick up messages, process, and upload results.

• video-processing should create HLS outputs (M3U8 + TS segments) in processed-videos-kodekloud:

• thumbnail-processing should create thumbnails in thumbnails-kodekloud:

• If you don’t see outputs, check CloudWatch Logs for both Lambda functions to verify message receipt and errors.
• Verify SNS topic policy and S3 event notifications if S3 cannot publish to SNS.
• Confirm SQS queue policies allow SNS to deliver messages.

​

Cleanup

• Delete SQS queues
• Delete SNS topic
• Delete Lambda functions
• Empty & delete S3 buckets

​

Links and references

• Amazon S3 Event Notifications — https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html
• Amazon SNS documentation — https://docs.aws.amazon.com/sns/
• Amazon SQS documentation — https://docs.aws.amazon.com/sqs/
• AWS Lambda documentation — https://docs.aws.amazon.com/lambda/