> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Kodekloud Record Store App Overview and Installation

> Guide to a demo Kodekloud Record Store app showing architecture, local Docker Compose installation, and observability setup including Prometheus Grafana Jaeger Loki RabbitMQ Celery and PostgreSQL

This lesson uses the small demo application "KodeKloud Record Store Web App" to demonstrate observability, distributed tracing, metrics, and how distributed components interact. The full repository is available on GitHub: [jaked-page91/kodekloud-records-store-web-app](https://github.com/jaked-page91/kodekloud-records-store-web-app). Fork it to experiment with code, compose configurations, and observability settings.

![A screenshot of a GitHub repository page for "kodekloud-records-store-web-app" showing the file and folder list, branches, and recent commits. The right sidebar displays the repo's About section, releases, and package/deployment info.](https://mintcdn.com/kodekloud-c4ac6d9a/8f5ktW2ljIqVrxKx/images/Fundamentals-of-SRE/Course-Introduction/Kodekloud-Record-Store-App-Overview-and-Installation/kodekloud-records-store-repo-screenshot.jpg?fit=max\&auto=format\&n=8f5ktW2ljIqVrxKx\&q=85\&s=91b7527106c29f747c4265b2d36a3611)

The repository README includes a concise introduction and an architecture overview — this is intentionally a small, extendable demo you can treat as a playground for instrumentation and observability experiments.

![A dark-themed README/documentation page for "KodeKloud Records Store" showing an overview and architecture sections. It lists goals and bullet points about observability, monitoring, distributed tracing, and application components.](https://mintcdn.com/kodekloud-c4ac6d9a/8f5ktW2ljIqVrxKx/images/Fundamentals-of-SRE/Course-Introduction/Kodekloud-Record-Store-App-Overview-and-Installation/kodekloud-records-store-observability.jpg?fit=max\&auto=format\&n=8f5ktW2ljIqVrxKx\&q=85\&s=80c505ff72de2d323e1520b39b9478eb)

## Overview and architecture

Key components:

* FastAPI web service: handles HTTP requests and serves the API.
* Celery workers: handle background processing with RabbitMQ as the broker.
* PostgreSQL: persistent storage for products, orders, and other state.
* Observability stack: Prometheus, Grafana, Jaeger, Loki, Fluent Bit, Pushgateway, Blackbox Exporter, and Alertmanager collect metrics, logs, and traces.

The diagram below shows the main components and their interactions.

![A dark-themed system architecture diagram showing an observability stack and a Kodekloud records store. It labels components like Blackbox Exporter, FastAPI web service, RabbitMQ, Celery worker, Prometheus and PostgreSQL with arrows indicating data flows.](https://mintcdn.com/kodekloud-c4ac6d9a/8f5ktW2ljIqVrxKx/images/Fundamentals-of-SRE/Course-Introduction/Kodekloud-Record-Store-App-Overview-and-Installation/observability-kodekloud-architecture-fastapi-rabbitmq-prometheus.jpg?fit=max\&auto=format\&n=8f5ktW2ljIqVrxKx\&q=85\&s=e0b4b8b5bc4062d87990b8edd73dcae8)

## Request flow (how requests propagate and where signals are produced)

* Read-only requests (example: `GET /products`) are handled fully by the FastAPI app querying PostgreSQL.
* Stateful/business requests (example: `POST /checkout`) create orders and emit background tasks to RabbitMQ. Celery workers then process payments, update inventory, and persist results back to PostgreSQL.
* Traces, logs, and metrics are exported so you can follow end-to-end request paths in Jaeger, correlate logs in Loki, and inspect metrics in Prometheus/Grafana.

The sequence diagram below illustrates these interactions and where observability signals are generated for different endpoints.

![A dark-themed sequence diagram titled "Request Flow" showing lanes for Client, FastAPI App, Database, RabbitMQ, and Celery Worker. It diagrams GET /products and POST /checkout interactions (query products, create order, queue background task, process payment/inventory) with notes about Jaeger/Loki tracing and Prometheus metrics.](https://mintcdn.com/kodekloud-c4ac6d9a/8f5ktW2ljIqVrxKx/images/Fundamentals-of-SRE/Course-Introduction/Kodekloud-Record-Store-App-Overview-and-Installation/request-flow-fastapi-celery-rabbitmq-tracing.jpg?fit=max\&auto=format\&n=8f5ktW2ljIqVrxKx\&q=85\&s=f37d2cc1812eb283f2bcef13d1f6a5e3)

## Project structure (simplified)

Below is a simplified tree of the files and directories you’ll interact with in the course:

```text theme={null}
├── main.py                # FastAPI application entry point
├── routes.py              # API endpoints (products, orders, checkout)
├── models.py              # Database models (Product, Order)
├── database.py            # Database connection and session management
├── worker.py              # Celery background tasks
├── telemetry.py           # OpenTelemetry setup
├── metrics.py             # Prometheus metrics definitions (BEST PRACTICES)
├── requirements.txt       # Python dependencies
config/
└── monitoring/            # Observability configuration
    ├── prometheus.yml
    ├── alertmanager.yml
    ├── alert_rules.yml
    ├── sli_rules.yml
    └── grafana-provisioning/
deploy/
└── environments/
    ├── setup-local-env.sh # Environment setup script (creates .env.dev)
    └── templates/
scripts/
├── generate_logs.sh
└── demo_request_correlation.sh
docker-compose.yaml
Dockerfile
test_traffic.sh
black_box_monitor.sh
```

## Get started (local development)

1. Clone the repository and prepare a development environment file:

```bash theme={null}
git clone <your-repo-url>
cd kodekloud-records-store-web-app

# Create a .env.dev file with safe defaults:
./deploy/environments/setup-local-env.sh

# Inspect the created file:
cat deploy/environments/.env.dev
```

Example: `deploy/environments/.env.dev`

```bash theme={null}
# Application Settings
DEBUG=true
LOG_LEVEL=DEBUG
WEB_PORT=8000

# Message Queue
RABBITMQ_HOST=rabbitmq

# Monitoring & Observability
GRAFANA_ADMIN_PASSWORD=dev_admin_123
PROMETHEUS_RETENTION_TIME=7d

# OpenTelemetry
OTEL_SERVICE_NAME=kodekloud-record-store-api-dev
OTEL_EXPORTER_OTLP_ENDPOINT=jaeger:4317
OTEL_EXPORTER_OTLP_PROTOCOL=grpc
OTEL_TRACES_SAMPLER=parentbased_always_on

# Prometheus Pushgateway
PROMETHEUS_PUSHGATEWAY=pushgateway:9091

# Python Path
PYTHONPATH=/app/src

# Environment Tag
ENVIRONMENT=development
```

2. Start the full stack with Docker Compose using the `.env.dev` file:

```bash theme={null}
docker-compose --env-file deploy/environments/.env.dev up -d
```

3. Verify containers are running:

```bash theme={null}
docker-compose ps
```

4. Test key endpoints and generate traces/metrics:

```bash theme={null}
# Root / health
curl http://localhost:8000/
curl http://localhost:8000/health

# Prometheus metrics exposed by the app
curl http://localhost:8000/metrics

# Endpoints that create traces/errors for testing
curl http://localhost:8000/trace-test
curl http://localhost:8000/error-test
```

## Available services (default ports and access)

| Service | URL | Notes / Credentials |
| - | -: | - |
| Records Store API | [http://localhost:8000](http://localhost:8000) | API root |
| API Docs | [http://localhost:8000/docs](http://localhost:8000/docs) | Swagger / OpenAPI UI |
| Grafana | [http://localhost:3000](http://localhost:3000) | `admin` / `dev_admin_123` |
| Prometheus | [http://localhost:9090](http://localhost:9090) | Prometheus UI |
| Jaeger | [http://localhost:16686](http://localhost:16686) | Traces search and timeline |
| Loki | [http://localhost:3100](http://localhost:3100) | Log ingestion/query API |
| Alertmanager | [http://localhost:9093](http://localhost:9093) | Alerting UI |
| RabbitMQ Management | [http://localhost:15672](http://localhost:15672) | `guest` / `guest` |

## Inspecting Prometheus and Jaeger

After the stack is up, verify metrics and traces in the Prometheus and Jaeger UIs.

![A screenshot of the Prometheus web UI (localhost:9090) showing the Query page with an empty expression input, an "Execute" button, and the message "No data queried yet." The interface also shows tabs for Table/Graph/Explain and a visible "Add query" button.](https://mintcdn.com/kodekloud-c4ac6d9a/8f5ktW2ljIqVrxKx/images/Fundamentals-of-SRE/Course-Introduction/Kodekloud-Record-Store-App-Overview-and-Installation/prometheus-query-empty-input.jpg?fit=max\&auto=format\&n=8f5ktW2ljIqVrxKx\&q=85\&s=d0ef578e42bb907abfc0f6776ea66800)

![A screenshot of the Jaeger UI search page with a left-side search form. On the right is a large cartoon blue gopher wearing a green hat peeking up from the bottom of the screen.](https://mintcdn.com/kodekloud-c4ac6d9a/8f5ktW2ljIqVrxKx/images/Fundamentals-of-SRE/Course-Introduction/Kodekloud-Record-Store-App-Overview-and-Installation/jaeger-ui-search-gopher-green-hat.jpg?fit=max\&auto=format\&n=8f5ktW2ljIqVrxKx\&q=85\&s=68bc00555b401a7320bb2d6175efdfb1)

## Docker Compose highlights

The repository provides a single `docker-compose.yaml` describing the application and the observability stack. The `api` service reads many values via environment variables from the external `.env.*` file. Example excerpt showing environment injection:

```yaml theme={null}
services:
  api:
    build:
      context: .
      dockerfile: Dockerfile
    container_name: kodekloud-record-store-api
    restart: always
    depends_on:
      - db
      - rabbitmq
      - jaeger
      - fluent-bit
    ports:
      - "${WEB_PORT:-8000}:8000"
    environment:
      POSTGRES_HOST: ${POSTGRES_HOST}
      POSTGRES_DB: ${POSTGRES_DB}
      POSTGRES_USER: ${POSTGRES_USER}
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
      RABBITMQ_HOST: ${RABBITMQ_HOST}
      PYTHONPATH: ${PYTHONPATH}
      OTEL_SERVICE_NAME: ${OTEL_SERVICE_NAME}
      OTEL_EXPORTER_OTLP_ENDPOINT: ${OTEL_EXPORTER_OTLP_ENDPOINT}
      OTEL_EXPORTER_OTLP_PROTOCOL: ${OTEL_EXPORTER_OTLP_PROTOCOL}
      OTEL_TRACES_SAMPLER: ${OTEL_TRACES_SAMPLER}
      OTEL_PROPAGATORS: "tracecontext,baggage"
      DEBUG: ${DEBUG}
      LOG_LEVEL: ${LOG_LEVEL}
      ENVIRONMENT: ${ENVIRONMENT}
    logging:
      driver: "fluentd"
      options:
        fluentd-address: "localhost:24224"
        tag: "docker.{{.Name}}"
        fluentd-async: "true"
    networks:
      - kodekloud-record-store-net
```

Other compose services include: `worker` (Celery), `db` (Postgres), `rabbitmq`, `prometheus`, `pushgateway`, `grafana`, `alertmanager`, `jaeger`, `loki`, `fluent-bit`, and `blackbox_exporter`. The repository already contains Prometheus scrape configs, Grafana provisioning, Loki/fluent-bit logging pipelines, and alert rules to explore.

## Testing and demo scripts

Use included helper scripts to generate traffic, produce log lines, and run synthetic probes:

```bash theme={null}
# Generate synthetic business traffic
./test_traffic.sh

# Produce logs for correlation testing
./scripts/generate_logs.sh

# Run a black-box probe / synthetic monitor
./black_box_monitor.sh
```

## Common troubleshooting and tips

* If ports conflict, check listeners with `netstat -tuln` or `ss -tuln` and update the `.env.*` files accordingly.
* Verify container health with `docker-compose ps` and inspect logs with `docker logs <container>`.
* Confirm Prometheus scrape targets: [http://localhost:9090/api/v1/targets](http://localhost:9090/api/v1/targets)
* Verify application metrics are exported:

```bash theme={null}
curl http://localhost:8000/metrics | grep kodekloud_
```

When deploying to staging or production, copy a template from `deploy/environments/templates/`, replace sensitive values via CI/CD secrets, and run docker-compose with the chosen `.env.*` file. Use environment-specific OTEL and Prometheus settings to avoid sending dev telemetry to production systems.

## Final notes

This lesson provides the repository and step-by-step instructions so you can explore the app and its observability stack at your own pace. Subsequent modules dive deeper into each component (Prometheus, Grafana, Jaeger, Loki, Fluent Bit), show how to instrument code with OpenTelemetry, create meaningful alerts and dashboards, and correlate traces, logs, and metrics for effective incident analysis. For more on the tools used:

* Prometheus: [https://prometheus.io/](https://prometheus.io/)
* Grafana: [https://grafana.com/](https://grafana.com/)
* Jaeger: [https://www.jaegertracing.io/](https://www.jaegertracing.io/)
* OpenTelemetry: [https://opentelemetry.io/](https://opentelemetry.io/)

> **[Watch Video](https://learn.kodekloud.com/user/courses/fundamentals-of-sre/module/2ed55f19-4a35-40be-9509-e57187ab4866/lesson/4ee9d625-ec0c-421f-80e2-d2ec4cd4dac2)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.