SNSSQS Demo

In this lesson, you will learn how to integrate Amazon SNS and Amazon SQS to build a video processing application. The goal is to trigger workflows when a user uploads a video file (e.g., MP4) to an S3 bucket. This upload starts an SNS notification that fans out to two SQS queues. One queue invokes a Lambda function for video conversion (e.g., converting the video into HLS format), while the other triggers a Lambda function to generate video thumbnails.

Below is an overview of the architecture along with detailed configuration steps.

Architecture Overview

1. S3 Bucket for Raw Videos:
   When a user uploads a video, the file is stored in an S3 bucket in its original format.

2. SNS Topic Notification:
   The S3 bucket invokes an SNS notification (e.g., topic name video-uploaded) each time a video is uploaded.

3. Fan-out to SQS Queues:
   The SNS topic sends out notifications to two SQS queues:

   • Video Processing Queue:
     A subscribed Lambda function retrieves the file name from the message, downloads the video from S3, processes it (e.g., converts the video), and then uploads the converted video to a designated S3 bucket.

   • Thumbnail Processing Queue:
     A second Lambda function automatically creates video thumbnails and stores them in a specified thumbnails S3 bucket.

Below is the architecture diagram illustrating the entire flow:

Step 1: Configuring the SNS Topic

1. Open the SNS service console and navigate to "Topics." Click on "Create topic".
2. Enter the topic name (for example, video-uploaded) and select the Standard type (message ordering is not required for this application).

3. Optionally, set a display name and retain default settings for encryption and access policies. Click "Create topic" to finalize the configuration.

Step 2: Creating SQS Queues

Video Processing Queue

1. Go to the SQS console and click "Create queue."
2. Select the Standard queue type and name the queue video-processing.
3. Accept the default settings for visibility timeout, delivery delay, and message retention.
4. Under the access policy, leave it at its default setting (only the queue owner can send/receive messages).

5. After queue creation, subscribe the queue to the SNS topic. Click "Subscribe to Amazon SNS topic," select the video-uploaded topic, and hit "Save."

Thumbnail Processing Queue

1. Create another Standard queue and name it thumbnail-processing.
2. Use the default settings and subscribe this queue to the video-uploaded SNS topic.

At this point, any message published to the SNS topic is delivered to both queues. Each queue holds the incoming messages until the respective Lambda functions process them.

Step 3: Testing SNS and SQS Integration

1. Navigate to the SNS console and select the video-uploaded topic.
2. Click "Publish message" and enter the message body. The message can be structured in plain text or JSON. For example, a JSON message might look like this:

{
  "bucket": "raw-videos-kodekloud",
  "key": "b415c94e-de85-4f6a-949c-2eb2e293bf30"
}

3. After publishing the message, refresh the SQS console to verify that the message appears in both queues. If no consumer (Lambda or EC2) is configured, the messages remain in the queues.
4. Publish another message with different details to see the message count increment.

Step 4: Configuring Lambda Functions

Video Processing Lambda Function

1. Sign in to the Lambda console and click "Create function." Select "Author from scratch."
2. Set the function name as video-processing, choose the appropriate runtime (e.g., Node.js 18.x), and assign a role with SQS and S3 permissions. You may create a new role (e.g., Lambda_SQS_S3) and attach additional policies (such as S3 Full Access) via the IAM console.

3. Once the Lambda function is created, add an SQS trigger by selecting the video-processing queue. Configure the batch size (for example, 1 for individual processing or a higher number for batching) and set the appropriate batch window.

4. Update the function code to process the event. The sample code below logs the event and extracts the message from the first record:

export const handler = async (event) => {
    console.log("Received event:", event);
    // Extract and parse the message from the SQS event
    const body = JSON.parse(event.Records[0].body);
    const message = JSON.parse(body.Message);
    console.log("Message details:", message);

    // Processing logic (e.g., retrieving the video from S3, converting it, and uploading) goes here.
    const response = {
        statusCode: 200,
        body: JSON.stringify("Hello from Lambda!"),
    };
    return response;
};

After deploying the function, check the SQS queue’s message count. A reduction indicates that the Lambda function is processing messages. You can also review CloudWatch logs for detailed execution information.

Thumbnail Processing Lambda Function

1. Create another Lambda function named thumbnail-processing using the same steps as above. Use the same role (e.g., Lambda_SQS_S3) because this function also requires SQS and S3 access.
2. Add an SQS trigger for the thumbnail-processing queue and configure the batch settings.
3. Implement the code to handle thumbnail generation, then deploy the function.

Step 5: Configuring S3 Event Notifications

Automate the workflow by configuring your S3 bucket (storing raw videos) to trigger an event notification to the SNS topic each time a video is uploaded:

1. Open the S3 console and select the raw videos bucket (e.g., raw-videos-kodekloud).
2. Under the "Properties" tab, scroll down to "Event notifications" and create a new notification.
3. Set an event name (e.g., video-uploaded). Optionally, specify a prefix (files are assumed to be uploaded to the root for this demonstration).
4. Under "Event types," select the object creation events (PUT, POST, COPY, etc.).
5. For the destination, select SNS and choose the video-uploaded topic.

Important:
Before S3 can publish notifications to SNS, update the SNS topic's access policy to allow the Amazon S3 service to publish messages. For example, add the following policy statement to your SNS topic (replace placeholders with actual values):

{
  "Sid": "ExampleSNSPublishPolicy",
  "Effect": "Allow",
  "Principal": {
    "Service": "s3.amazonaws.com"
  },
  "Action": "sns:Publish",
  "Resource": "arn:aws:sns:us-east-1:841860927337:video-uploaded",
  "Condition": {
    "ArnLike": {
      "aws:SourceArn": "arn:aws:s3:::raw-videos-kodekloud"
    },
    "StringEquals": {
      "aws:SourceAccount": "841860927337"
    }
  }
}

To update the SNS topic policy:

1. Open the SNS topic configuration page.
2. Click "Edit" next to Access Policy.
3. Add the above JSON statement to the existing policy and save your changes.

Finally, save the event notification within the S3 console.

Step 6: Testing and Verification

1. Send Test Messages:
   Publish a test message via the SNS console and verify that both SQS queues receive it.

2. Upload a Video File:
   Upload a short video file to the raw videos bucket. This triggers an SNS notification which, in turn, invokes the linked Lambda functions.

3. Verify Outputs:

   • Check the processed videos bucket for the converted video files (e.g., an .m3u8 file along with corresponding .ts chunk files).
   • Review the thumbnails bucket to ensure the thumbnail images have been generated appropriately.

Cleanup

After verifying the configuration, it is important to clean up resources to avoid incurring unnecessary charges:

1. Delete SQS Queues:
   In the SQS console, delete both the video-processing and thumbnail-processing queues.

2. Delete the SNS Topic:
   Open the SNS console and delete the video-uploaded topic.

3. Remove Lambda Functions:
   Delete both Lambda functions (video-processing and thumbnail-processing).

4. Delete S3 Buckets:
   Empty the raw videos, processed videos, and thumbnails buckets and then delete them.

This demonstration has shown how to integrate SNS with SQS and Lambda for automated video processing and thumbnail generation. Leveraging AWS services enables efficient and scalable event-driven processing for video applications.

Happy Learning!