Demo Kinesis in real time consumption and production

• Create a Kinesis Data Stream.
• Create a Kinesis Data Firehose delivery stream that uses the Kinesis stream as its source.
• Configure S3 as the Firehose destination (create a bucket if needed).
• Produce test records into the Kinesis stream using the AWS SDK for JavaScript (v3).

​

Architecture overview

​

1) Create the Kinesis Data Stream

• Provisioned — choose explicit shard count when you want fine-grained throughput control and cost predictability.
• On-demand — automatic scaling with no shard management; simpler but can be more expensive for high sustained throughput.

​

2) Create the Firehose delivery stream and choose S3 as destination

• In the Firehose creation workflow, choose Kinesis Data Stream as the source and select the stream you created.
• Choose S3 as the destination. Either pick an existing bucket or create a new one during the delivery stream setup.
• Optionally enable buffering, compression, or format conversion (JSON → Parquet/ORC). For this demo we forward raw JSON into S3.

• Buffering settings (size and interval) control how frequently Firehose writes objects to S3. Smaller intervals yield more objects; larger intervals reduce PUT count but increase object size.
• Leaving transformation/format conversion disabled will deliver raw payloads as-is.

​

3) Confirm S3 delivery and inspect the results

{"date":1698107939318,"price":"$46941"}{"date":1698107939259,"price":"$342561"}{"date":1698107939368,"price":"$301171"}{"date":1698107939418,"price":"$274001"}{"date":1698107939469,"price":"$6611"}

​

4) Produce test records using AWS SDK for JavaScript (v3)

• Node.js (v14+ recommended)
• AWS credentials configured (environment variables, shared credentials file, or AWS CLI setup)
• Install the Kinesis client:
  • npm install @aws-sdk/client-kinesis

// send-kinesis.js
import { KinesisClient, PutRecordCommand } from "@aws-sdk/client-kinesis";

const REGION = "us-east-1"; // change to your region
const STREAM_NAME = "crypto-stock-price"; // change to your stream name

const client = new KinesisClient({ region: REGION });

async function putRecord(payload) {
  const input = {
    StreamName: STREAM_NAME,
    PartitionKey: "test", // simple partition key for demo purposes
    Data: Buffer.from(JSON.stringify(payload)),
  };

  const command = new PutRecordCommand(input);
  return client.send(command);
}

function startSending(intervalMs = 50) {
  setInterval(async () => {
    try {
      const payload = {
        date: Date.now(),
        price: "$" + (Math.floor(Math.random() * 40000) + 1),
      };
      const response = await putRecord(payload);
      console.log("PutRecord response:", response);
    } catch (err) {
      console.error("Error sending record:", err);
    }
  }, intervalMs);
}

// Start sending records every 50 ms
startSending(50);

node send-kinesis.js

​

Optional: Data transformation and format conversion

• Enable Firehose data transformation and provide a Lambda function to process each batch.
• Enable format conversion to convert newline-delimited JSON to Parquet/ORC — this improves query performance and reduces storage for analytic workloads.

​

Troubleshooting checklist

• Are AWS credentials and region configured? Use aws sts get-caller-identity to verify.
• Is the Firehose delivery stream in an Active state?
• Are IAM permissions correct for Firehose to write to your S3 bucket and for Kinesis to be used as a source?
• Check CloudWatch logs for Firehose or the transformation Lambda (if enabled).

​

Links and references

• AWS SDK for JavaScript (v3) — https://docs.aws.amazon.com/sdk-for-javascript/v3/developer-guide/welcome.html
• Kinesis Data Streams documentation — https://docs.aws.amazon.com/kinesis/index.html
• Kinesis Data Firehose documentation — https://docs.aws.amazon.com/firehose/index.html
• Amazon S3 documentation — https://docs.aws.amazon.com/s3/index.html