Demo Hands on with PubSub Subscription and its integration
Hands on demo connecting Pub/Sub topic to Cloud Storage so published messages are written as files for zero infrastructure ingestion and downstream processing.
Welcome back. In this hands-on demo you’ll extend your Pub/Sub knowledge by connecting a Pub/Sub topic to Cloud Storage so that published messages land as files in a bucket. This pattern is useful for simple, zero-infrastructure ingestion pipelines (for example, later loading objects into BigQuery or processing them with Spark).Prerequisites
A Google Cloud project with billing enabled.
Pub/Sub API and Cloud Storage enabled.
A Pub/Sub topic already created (this demo uses kodekloud-demo-topic).
What we’ll do
Create a Cloud Storage bucket.
Inspect the Pub/Sub topic.
Create a subscription that writes messages to the bucket.
Publish messages and confirm files appear in Cloud Storage.
Clean up resources.
Step 1 — Create a Cloud Storage bucket
In the Cloud Console search for “Buckets” → Create.
Provide a globally unique name (demo uses kodekloud-pubsub-bucket).
Leave other settings at their defaults and click Create.
Step 2 — Open the Pub/Sub topic and view metrics
Open the topic you created earlier (kodekloud-demo-topic). The topic details page shows helpful monitoring metrics (published message count, publish requests, throughput) and export options (BigQuery, Cloud Storage).
Step 3 — Create a subscription that writes to Cloud Storage
Create a subscription that delivers messages from the topic into your Cloud Storage bucket:
Go to Pub/Sub → Subscriptions → Create subscription.
Enter a subscription ID (demo: kodekloud-storage-sub).
Select the topic projects/<your-project>/topics/kodekloud-demo-topic.
For Delivery type choose Write to Cloud Storage.
Browse and select the bucket kodekloud-pubsub-bucket.
Choose file format: Avro or JSON (demo uses JSON).
When “Write to Cloud Storage” is selected, Pub/Sub needs permission to create objects in the bucket. The Console will detect missing permissions and offer a one-click option to set them.
Pub/Sub requires permission to write objects into the destination bucket (typically the Pub/Sub service account needs Object Creator and Object Viewer roles). The Console can auto-assign these when you click “Set permission” during subscription creation.
Finalize the subscription creation. With a 1 minute batching interval, Pub/Sub will write files to the bucket approximately once per minute (subject to batching thresholds).
Step 4 — Publish messages and verify Cloud Storage objects
Publish messages to the topic to trigger the delivery. You can use a small publish script from Cloud Shell. Example output from a Cloud Shell publish run:
Wait for the subscription’s configured batching interval (1 minute in this demo), then inspect the bucket kodekloud-pubsub-bucket in the Cloud Console. You should see new objects created by Pub/Sub; download them to view the JSON payloads.Example Cloud Storage object URLs produced by this integration:
