Kodekloud Record Store App Overview and Installation

This lesson uses the small demo application “KodeKloud Record Store Web App” to demonstrate observability, distributed tracing, metrics, and how distributed components interact. The full repository is available on GitHub: jaked-page91/kodekloud-records-store-web-app. Fork it to experiment with code, compose configurations, and observability settings.

A screenshot of a GitHub repository page for "kodekloud-records-store-web-app" showing the file and folder list, branches, and recent commits. The right sidebar displays the repo's About section, releases, and package/deployment info.

The repository README includes a concise introduction and an architecture overview — this is intentionally a small, extendable demo you can treat as a playground for instrumentation and observability experiments.

Overview and architecture

Key components:

FastAPI web service: handles HTTP requests and serves the API.
Celery workers: handle background processing with RabbitMQ as the broker.
PostgreSQL: persistent storage for products, orders, and other state.
Observability stack: Prometheus, Grafana, Jaeger, Loki, Fluent Bit, Pushgateway, Blackbox Exporter, and Alertmanager collect metrics, logs, and traces.

The diagram below shows the main components and their interactions.

A dark-themed system architecture diagram showing an observability stack and a Kodekloud records store. It labels components like Blackbox Exporter, FastAPI web service, RabbitMQ, Celery worker, Prometheus and PostgreSQL with arrows indicating data flows.

Request flow (how requests propagate and where signals are produced)

Read-only requests (example: GET /products) are handled fully by the FastAPI app querying PostgreSQL.
Stateful/business requests (example: POST /checkout) create orders and emit background tasks to RabbitMQ. Celery workers then process payments, update inventory, and persist results back to PostgreSQL.
Traces, logs, and metrics are exported so you can follow end-to-end request paths in Jaeger, correlate logs in Loki, and inspect metrics in Prometheus/Grafana.

The sequence diagram below illustrates these interactions and where observability signals are generated for different endpoints.

A dark-themed sequence diagram titled "Request Flow" showing lanes for Client, FastAPI App, Database, RabbitMQ, and Celery Worker. It diagrams GET /products and POST /checkout interactions (query products, create order, queue background task, process payment/inventory) with notes about Jaeger/Loki tracing and Prometheus metrics.

Project structure (simplified)

Below is a simplified tree of the files and directories you’ll interact with in the course:

├── main.py                # FastAPI application entry point
├── routes.py              # API endpoints (products, orders, checkout)
├── models.py              # Database models (Product, Order)
├── database.py            # Database connection and session management
├── worker.py              # Celery background tasks
├── telemetry.py           # OpenTelemetry setup
├── metrics.py             # Prometheus metrics definitions (BEST PRACTICES)
├── requirements.txt       # Python dependencies
config/
└── monitoring/            # Observability configuration
    ├── prometheus.yml
    ├── alertmanager.yml
    ├── alert_rules.yml
    ├── sli_rules.yml
    └── grafana-provisioning/
deploy/
└── environments/
    ├── setup-local-env.sh # Environment setup script (creates .env.dev)
    └── templates/
scripts/
├── generate_logs.sh
└── demo_request_correlation.sh
docker-compose.yaml
Dockerfile
test_traffic.sh
black_box_monitor.sh

Get started (local development)

Clone the repository and prepare a development environment file:

git clone <your-repo-url>
cd kodekloud-records-store-web-app

# Create a .env.dev file with safe defaults:
./deploy/environments/setup-local-env.sh

# Inspect the created file:
cat deploy/environments/.env.dev

Example: deploy/environments/.env.dev

# Application Settings
DEBUG=true
LOG_LEVEL=DEBUG
WEB_PORT=8000

# Message Queue
RABBITMQ_HOST=rabbitmq

# Monitoring & Observability
GRAFANA_ADMIN_PASSWORD=dev_admin_123
PROMETHEUS_RETENTION_TIME=7d

# OpenTelemetry
OTEL_SERVICE_NAME=kodekloud-record-store-api-dev
OTEL_EXPORTER_OTLP_ENDPOINT=jaeger:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_TRACES_SAMPLER=parentbased_always_on

# Prometheus Pushgateway
PROMETHEUS_PUSHGATEWAY=pushgateway:9091

# Python Path
PYTHONPATH=/app/src

# Environment Tag
ENVIRONMENT=development

Start the full stack with Docker Compose using the .env.dev file:

docker-compose --env-file deploy/environments/.env.dev up -d

Verify containers are running:

docker-compose ps

Test key endpoints and generate traces/metrics:

# Root / health
curl http://localhost:8000/
curl http://localhost:8000/health

# Prometheus metrics exposed by the app
curl http://localhost:8000/metrics

# Endpoints that create traces/errors for testing
curl http://localhost:8000/trace-test
curl http://localhost:8000/error-test

Available services (default ports and access)

Service	URL	Notes / Credentials
Records Store API	http://localhost:8000	API root
API Docs	http://localhost:8000/docs	Swagger / OpenAPI UI
Grafana	http://localhost:3000	`admin` / `dev_admin_123`
Prometheus	http://localhost:9090	Prometheus UI
Jaeger	http://localhost:16686	Traces search and timeline
Loki	http://localhost:3100	Log ingestion/query API
Alertmanager	http://localhost:9093	Alerting UI
RabbitMQ Management	http://localhost:15672	`guest` / `guest`

Inspecting Prometheus and Jaeger

After the stack is up, verify metrics and traces in the Prometheus and Jaeger UIs.

A screenshot of the Prometheus web UI (localhost:9090) showing the Query page with an empty expression input, an "Execute" button, and the message "No data queried yet." The interface also shows tabs for Table/Graph/Explain and a visible "Add query" button.

A screenshot of the Jaeger UI search page with a left-side search form. On the right is a large cartoon blue gopher wearing a green hat peeking up from the bottom of the screen.

Docker Compose highlights

The repository provides a single docker-compose.yaml describing the application and the observability stack. The api service reads many values via environment variables from the external .env.* file. Example excerpt showing environment injection:

services:
  api:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: kodekloud-record-store-api
    restart: always
    depends_on:
      - db
      - rabbitmq
      - jaeger
      - fluent-bit
    ports:
      - "${WEB_PORT:-8000}:8000"
    environment:
      POSTGRES_HOST: ${POSTGRES_HOST}
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      RABBITMQ_HOST: ${RABBITMQ_HOST}
      PYTHONPATH: ${PYTHONPATH}
      OTEL_SERVICE_NAME: ${OTEL_SERVICE_NAME}
      OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT}
      OTEL_EXPORTER_OTLP_PROTOCOL: ${OTEL_EXPORTER_OTLP_PROTOCOL}
      OTEL_TRACES_SAMPLER: ${OTEL_TRACES_SAMPLER}
      OTEL_PROPAGATORS: "tracecontext,baggage"
      DEBUG: ${DEBUG}
      LOG_LEVEL: ${LOG_LEVEL}
      ENVIRONMENT: ${ENVIRONMENT}
    logging:
      driver: "fluentd"
      options:
        fluentd-address: "localhost:24224"
        tag: "docker.{{.Name}}"
        fluentd-async: "true"
    networks:
      - kodekloud-record-store-net

Other compose services include: worker (Celery), db (Postgres), rabbitmq, prometheus, pushgateway, grafana, alertmanager, jaeger, loki, fluent-bit, and blackbox_exporter. The repository already contains Prometheus scrape configs, Grafana provisioning, Loki/fluent-bit logging pipelines, and alert rules to explore.

Testing and demo scripts

Use included helper scripts to generate traffic, produce log lines, and run synthetic probes:

# Generate synthetic business traffic
./test_traffic.sh

# Produce logs for correlation testing
./scripts/generate_logs.sh

# Run a black-box probe / synthetic monitor
./black_box_monitor.sh

Common troubleshooting and tips

If ports conflict, check listeners with netstat -tuln or ss -tuln and update the .env.* files accordingly.
Verify container health with docker-compose ps and inspect logs with docker logs <container>.
Confirm Prometheus scrape targets: http://localhost:9090/api/v1/targets
Verify application metrics are exported:

curl http://localhost:8000/metrics | grep kodekloud_

When deploying to staging or production, copy a template from deploy/environments/templates/, replace sensitive values via CI/CD secrets, and run docker-compose with the chosen .env.* file. Use environment-specific OTEL and Prometheus settings to avoid sending dev telemetry to production systems.

Final notes

This lesson provides the repository and step-by-step instructions so you can explore the app and its observability stack at your own pace. Subsequent modules dive deeper into each component (Prometheus, Grafana, Jaeger, Loki, Fluent Bit), show how to instrument code with OpenTelemetry, create meaningful alerts and dashboards, and correlate traces, logs, and metrics for effective incident analysis. For more on the tools used:

Prometheus: https://prometheus.io/
Grafana: https://grafana.com/
Jaeger: https://www.jaegertracing.io/
OpenTelemetry: https://opentelemetry.io/

Course Introduction

Fundamentals of SRE

Service Level Objectives and Measurements

Managing Complexity, Risk, and Toil

Incident Management

Release Engineering

Observability and Monitoring

Advanced Reliability Engineering

Bringing it All Together

Kodekloud Record Store App Overview and Installation

Overview and architecture

Request flow (how requests propagate and where signals are produced)

Project structure (simplified)

Get started (local development)

Available services (default ports and access)

Inspecting Prometheus and Jaeger

Docker Compose highlights

Testing and demo scripts

Common troubleshooting and tips

Final notes

Watch Video

​Overview and architecture

​Request flow (how requests propagate and where signals are produced)

​Project structure (simplified)

​Get started (local development)

​Available services (default ports and access)

​Inspecting Prometheus and Jaeger

​Docker Compose highlights

​Testing and demo scripts

​Common troubleshooting and tips

​Final notes

Watch Video

Overview and architecture

Request flow (how requests propagate and where signals are produced)

Project structure (simplified)

Get started (local development)

Available services (default ports and access)

Inspecting Prometheus and Jaeger

Docker Compose highlights

Testing and demo scripts

Common troubleshooting and tips

Final notes