Skip to main content
Welcome back — in this lesson we’ll cover Cloud Logging: what it is, how it works, and why it matters for data engineers maintaining pipelines and storage systems. Think of Cloud Logging as the central nervous system of your Google Cloud environment. It ingests logs from applications, VMs, containers, and managed services; indexes and stores them in log buckets; and exposes them for searching, alerting, visualization, and export.
A teal-to-aqua gradient slide with the title "Cloud Logging" centered in white text. A small "© Copyright KodeKloud" notice appears in the bottom-left.
Logging guarantees each entry is captured reliably, retained according to your policy, and accessible through tools like Logs Explorer for interactive investigation.

​
How Cloud Logging works (high level)

  • Every Google Cloud resource (Compute Engine VMs, GKE, Cloud Functions, App Engine, Cloud Storage, and managed services) can emit logs. Logs arrive via:
    • platform integrations,
    • agents (for VMs and some container setups), or
    • the Cloud Logging API and client libraries.
  • Incoming logs are ingested into the Logging service, which provides:
    • A log router that filters and routes entries to sinks.
    • Sinks that export logs to destinations such as Cloud Storage, BigQuery, Pub/Sub, or other log buckets (including cross-project/organization sinks).
    • Log buckets where logs are stored and retention policies are applied.
  • For interactive investigation, use Logs Explorer in the Cloud Console. It supports advanced queries, filters, and visualizations.
This pipeline captures console actions, API calls, service events, application logs, and audit events across your environment.

​
Why Cloud Logging is powerful

  • Low-latency ingestion: Logs appear quickly after they are emitted, enabling near real-time troubleshooting.
  • Fully managed: Google handles scaling, availability, and the underlying infrastructure.
  • Log-based metrics: Turn specific log patterns into metrics to power alerts and dashboards (e.g., count of failed jobs or application errors).
  • Rich querying: Logs Explorer offers a powerful query language and UI for filtering, inspecting, and visualizing trends.
  • Audit logs: Capture admin activity, data access, and system events—essential for security investigations and compliance.
Note: Data Access audit logs are not enabled by default for many services due to volume and cost. Enable them intentionally if you need detailed access records for troubleshooting or compliance.
Warning: Log retention and export settings affect cost and compliance. Configure retention policies and export sinks to balance long-term storage needs against billing.

​
Key components (at-a-glance)

ComponentPurposeExample use-case
Log router & sinksFilter and route log entries to destinationsExport deletes to Pub/Sub to trigger recovery workflows
Log buckets & retentionStore logs and enforce retention policiesArchive logs for 365 days in a separate bucket
Audit logsTrack admin activity, data access, and system eventsIdentify who deleted objects from a bucket
Log-based metricsCreate metrics from log contentsAlert on repeated job failures
IAM rolesControl access to view, write, and manage logsroles/logging.viewer, roles/logging.logWriter

​
Configuration patterns important to data engineers

  • Sinks: Use sinks to route audit or application logs to BigQuery for analysis, Cloud Storage for long-term retention, or Pub/Sub for streaming/real-time reaction.
  • Retention: Set retention at the log-bucket level to enforce compliance and manage cost.
  • Audit Logging: Admin Activity, Data Access, and System Event audit logs give different levels of visibility—enable Data Access selectively.
  • Log-based metrics & alerts: Create metrics from log patterns and surface them in Cloud Monitoring to monitor pipeline health.
  • IAM granularity: Apply least privilege using logging-specific roles.

​
Quick examples

Logs Explorer query to find Cloud Storage object deletes: 
resource.type="gcs_bucket"
protoPayload.methodName="storage.objects.delete"
protoPayload.resourceName:"your-bucket-name"
Create a sink to export delete events to BigQuery (replace PROJECT_ID and DATASET): 
gcloud logging sinks create export-deletes \
  bigquery.googleapis.com/projects/`PROJECT_ID`/datasets/`DATASET` \
  --log-filter='resource.type="gcs_bucket" protoPayload.methodName="storage.objects.delete"'
Example: read exported logs in BigQuery with SQL (simplified): 
SELECT
  protoPayload.authenticationInfo.principalEmail AS principal,
  timestamp,
  protoPayload.requestMetadata.callerIp AS caller_ip,
  protoPayload.resourceName AS resource
FROM
  `PROJECT_ID.DATASET.cloudaudit_googleapis_com_activity_*`
WHERE
  protoPayload.methodName = "storage.objects.delete"
ORDER BY timestamp DESC
LIMIT 100;

​
Example scenario: deleted CSV files in a Cloud Storage bucket

  1. Cloud Storage emits audit logs for object create/delete/read operations.
  2. Use Logs Explorer to locate storage.objects.delete events and see the principal (user or service account), timestamp, and API call.
  3. For deeper analysis, export audit logs to BigQuery to run SQL queries over historical events.
  4. For real-time reaction (e.g., automatic recovery), route delete events to Pub/Sub and trigger a Cloud Function or Dataflow pipeline.

​
Troubleshooting checklist for data engineers

  • Verify the relevant audit logs are enabled (Admin Activity vs Data Access).
  • Check sink filters and destinations to ensure exports include the desired log entries.
  • Confirm IAM: does the user or service account have roles/logging.viewer or proper sink writer permissions?
  • Inspect log retention for the bucket where logs are stored—older logs may be expired.
  • Use log-based metrics to detect recurring failures before they escalate.

​
References

Closing Cloud Logging is more than a repository: it’s an integrated observability platform for troubleshooting, security investigations, monitoring, and analytics. For data engineers, mastering log routing, audit logs, and log-based metrics helps maintain reliable and observable data systems. That’s it for this lesson. See you in the next one.

Watch Video