Connecting to Cloud PubSub

​

Architecture overview

• Publisher side — applications that send messages (for example, microservices, data collection APIs, or logging agents like Fluent Bit).
• Subscriber side — applications that receive messages from a subscription (for example, other microservices, data pipeline workers, or serverless functions).

​

How client libraries communicate

• Protocols: gRPC is used by default for performance and low latency. REST/JSON endpoints are also available for certain use cases and tooling.
• Endpoint: Client libraries communicate with the Pub/Sub endpoint at pubsub.googleapis.com unless you explicitly configure a different endpoint.
• Responsibilities handled by libraries: authentication (Application Default Credentials or explicit service account keys), automatic retries, publish batching, and subscription flow control/backpressure.

​

Authentication and IAM

• In Google Cloud runtimes (Compute Engine, GKE, Cloud Run, Cloud Functions), ADC is usually available automatically.
• When running outside Google Cloud (for example, local development), set GOOGLE_APPLICATION_CREDENTIALS to a service-account key file:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account.json"

​

Publish vs. subscribe patterns

• Publishing: Clients publish messages to a topic. Libraries support synchronous and asynchronous (future/promise-based) publish APIs and provide batching options for higher throughput.
• Pull subscribers: Clients pull messages from a subscription. Modern libraries support streaming pull with callback handlers and automatic lease (ack deadline) management.
• Push subscribers: Pub/Sub sends HTTP POST requests to a configured HTTPS endpoint when new messages arrive. Push endpoints must acknowledge the request (e.g., return HTTP 200) to avoid redelivery.

​

Client library packages (common languages)

# Python
pip install google-cloud-pubsub

# Node.js
npm install @google-cloud/pubsub

​

Minimal code examples

from google.cloud import pubsub_v1

project_id = "my-project"
topic_id = "my-topic"

publisher = pubsub_v1.PublisherClient()
topic_path = publisher.topic_path(project_id, topic_id)

data = "Hello, Pub/Sub!".encode("utf-8")
future = publisher.publish(topic_path, data, origin="python-sample")
message_id = future.result()
print(f"Published message ID: {message_id}")

from google.cloud import pubsub_v1

project_id = "my-project"
subscription_id = "my-subscription"

subscriber = pubsub_v1.SubscriberClient()
subscription_path = subscriber.subscription_path(project_id, subscription_id)

def callback(message):
    print(f"Received message: {message.data.decode('utf-8')}")
    message.ack()

streaming_pull_future = subscriber.subscribe(subscription_path, callback=callback)
print(f"Listening for messages on {subscription_path}...")

try:
    streaming_pull_future.result()
except KeyboardInterrupt:
    streaming_pull_future.cancel()

const {PubSub} = require('@google-cloud/pubsub');
const pubsub = new PubSub({projectId: 'my-project'});

async function publishMessage() {
  const topicName = 'my-topic';
  const dataBuffer = Buffer.from('Hello, Pub/Sub!');
  const messageId = await pubsub.topic(topicName).publish(dataBuffer);
  console.log(`Published message ID: ${messageId}`);
}

publishMessage().catch(console.error);

​

Operational considerations

• Batching and latency: Publishers batch messages to improve throughput. Tune batch size and delay to balance latency and throughput.
• Flow control: Configure max outstanding messages/bytes on subscribers to avoid resource exhaustion and to maintain predictable memory/CPU usage.
• Retries and dead-letter topics: Configure retry policies and consider a dead-letter topic for messages that fail processing repeatedly so they can be inspected and reprocessed.
• Push endpoints: Ensure push endpoints use HTTPS, validate requests as needed (e.g., authentication or token validation), and respond with the correct status codes to control acknowledgment behavior.
• Monitoring and alerting: Track metrics such as publish/ack latency, undelivered message backlog, and subscriber errors to detect and respond to issues quickly.

​

Security warning

​

Quick reference: patterns and choices

​

Links and references

• Cloud Pub/Sub documentation: https://cloud.google.com/pubsub
• Pub/Sub client libraries: https://cloud.google.com/pubsub/docs/reference/libraries
• Authentication overview (ADC): https://cloud.google.com/docs/authentication/production

• Publishers and subscribers connect to Cloud Pub/Sub via Google Cloud client libraries.
• Libraries abstract network, auth, retries, batching, and flow control so you can focus on message handling.
• Choose publish/pull/push based on latency, throughput, and runtime environment.
• Use Application Default Credentials or secure credential provisioning and follow least-privilege IAM.