Skip to main content
Welcome back. In this lesson you’ll configure and run Kafka Connect (standalone) with the Confluent Amazon S3 Sink connector to stream messages from a Kafka topic into an S3 bucket. High-level checklist
  • Keep your Kafka broker process running in the original terminal. Do not stop it.
  • Open a second terminal on the EC2 instance for the Kafka Connect steps.
  • Ensure the EC2 instance has permission to access the S3 bucket used for the connector.
Example Kafka broker logs (verify your broker is running and accepting connections):
Step-by-step
  1. Open a new terminal on the EC2 instance and become root
  1. Download the Confluent / Amazon S3 Sink connector plugin
Attempt to download:
If you get a 403 Forbidden error, the EC2 instance role needs permission to access the bucket. Attach an appropriate S3 policy to the EC2 role (the demo uses AmazonS3FullAccess; for production, use least-privilege policies).
The image shows an AWS Identity and Access Management (IAM) interface displaying details for a role named "kafka_S3_demo," including its summary, permissions, and other settings.
Attach the S3 policy to the role:
The image shows an AWS IAM permissions management interface where a user is selecting policy options related to Amazon S3, with "AmazonS3FullAccess" highlighted.
For production use, follow least-privilege principles. Grant only the S3 actions and bucket resources the connector requires (for example s3:PutObject, s3:ListBucket on the specific bucket).
After attaching the policy, re-run the aws s3 cp command. Successful download example:
Verify and unzip:
  1. Configure the Kafka Connect worker (standalone)
Open the worker properties shipped with Kafka (adjust the path if your Kafka distro is elsewhere):
Edit or confirm the following key settings. Replace the bootstrap.servers value with your broker’s IP/hostname and set plugin.path to the directory where you unpacked the connector. Example connect-standalone.properties:
Save and exit the editor. Quick reference — important connect-standalone.properties entries
  1. Create the S3 Sink connector configuration
Create a connector properties file that describes connector behavior (topics to read, S3 bucket, formatting, etc.):
Example s3-sink-connector.properties — replace s3.bucket.name, s3.region, and topics as appropriate:
Connector properties explained (high-level)
  • topics: Kafka topic(s) to sink to S3.
  • s3.bucket.name / s3.region: Target S3 bucket and region.
  • flush.size: Number of records before writing to S3.
  • format.class: Output format (JSON in this example).
  • plugin.path in the worker must include the connector JARs for Connect to load the io.confluent.connect.s3.S3SinkConnector class.
  1. Create the S3 bucket (if you haven’t already)
Create the bucket via the AWS Console and ensure the region matches s3.region in the connector config.
The image shows the AWS S3 bucket creation interface, displaying encryption options and a message indicating that the specified bucket name is already taken.
  1. Start Kafka Connect (standalone)
Change into the Kafka installation directory (or reference the full path to the script) and start Connect with the worker and connector config files:
Run the connect-standalone.sh script from the Kafka root directory (the directory that contains bin/), or supply the full path. If you run it from the wrong folder you will see No such file or directory.
On startup the worker will scan plugin paths and attempt to load the S3 connector. Example logs showing plugin scanning and consumer group assignment:
Important notes and troubleshooting
  • Topic existence: If the configured topic (for example cartevent) does not exist, Kafka Connect will not create it for you. Topic creation is controlled by the broker setting auto.create.topics.enable. Create the topic ahead of time with your desired partitions and replication: kafka-topics.sh --create ...
  • plugin.path: Ensure plugin.path points to the directory containing the unpacked connector JARs. Each plugin typically lives in its own subdirectory under plugin.path; if Connect cannot find the S3 connector classes it will not load the plugin.
  • Permissions: Confirm the EC2 instance IAM role has S3 permissions for the bucket (PutObject, ListBucket).
  • Converters: Keep worker and connector converters compatible. If you disable schemas (schemas.enable=false) ensure your messages are formatted accordingly.
Recap — what we did
  • Downloaded and unpacked the Confluent S3 Sink connector.
  • Updated connect-standalone.properties to reference the connector via plugin.path.
  • Created a connector configuration describing topics, S3 bucket, converters, and formatting.
  • Started Kafka Connect in standalone mode and verified plugin scanning and consumer assignment via logs.
Next steps
  • Produce events to the configured topic (cartevent) and observe Kafka Connect writing files to the S3 bucket.
  • Monitor the S3 bucket for objects written by the connector and adjust flush.size / rotate.schedule.interval.ms to balance latency and file sizes.
See you in the next lesson.

Watch Video