> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Processing Logs With Alloy Pipelines

> Explains using Alloy Loki pipelines to extract structured fields from logs, promote low cardinality values to labels for efficient querying, and avoid indexing high-cardinality fields.

Picture a clerk at a sorting center who opens each parcel as it arrives, reads the few details that matter—the destination and the priority—and writes them clearly on the outside.

A little work up front makes downstream sorting instant because the important details are right there on the label, not hidden inside the box.

Application logs are those unopened boxes. If useful values are embedded inside the free-form message text, queries that filter by those values must scan every line. Extracting and promoting those values into indexed labels makes queries fast and efficient.

Example: logs arriving as JSON lines with method and status code inside the message:

```json theme={null}
{"method":"POST","code":"500","path":"/api/orders","msg":"db timeout"}
{"method":"GET","code":"200","path":"/api/users","msg":"ok"}
{"method":"GET","code":"500","path":"/api/cart","msg":"upstream error"}
{"method":"POST","code":"201","path":"/api/login","msg":"created"}
```

We can search the message text, but filtering by a field (for example, every request that returned a 500) requires scanning the contents of every line. The processing pipeline extracts those values and makes them available as structured fields and, when appropriate, as indexed labels.

In Alloy, the `loki.process` component sits between the source and the writer and runs each log line through an ordered series of stages before storage. A typical pipeline for the example above:

* `match` stage: select only logs from the application we care about.
* `json` stage: parse log lines and extract fields such as `method` and `code`.
* `labels` stage: convert selected extracted fields into indexed labels for fast querying.

Order matters: extract the structured fields first, then promote them to labels.

Always run extraction stages (for example `json`) before `labels`. If you label first, there is nothing to extract from the newly indexed fields.

A small caution before you start labeling everything — labels must remain low-cardinality. Status codes (200, 404, 500) are suitable labels; unique request IDs or user IDs are not. Indexing high-cardinality values harms Loki's performance and increases storage costs.

Keep labels low-cardinality. Use labels for fields with a small, bounded set of possible values (for example, HTTP status codes or `environment`). For high-cardinality values (for example, request IDs or user IDs), keep them as structured metadata so you can filter them after the selector with pipes instead of promoting them to labels.

The loki.process pipeline might be represented like this:

```text theme={null}
loki.source.kubernetes
loki.process
loki.write

1 stage.match    `{app="api"}`
2 stage.json     `{ }` method code
3 stage.labels   method code
```

Pipeline stages at a glance:

| Stage | Purpose | Example |
| - | - | - |
| match | Filter logs to the subset you care about using a selector | `match: '{app="api"}'` |
| json | Parse the log line and extract structured fields | `json: fields: ['method','code','path']` |
| labels | Promote selected structured fields into indexed labels (low-cardinality only) | `labels: ['method','code']` |

For values that should not be indexed, use structured metadata. Structured metadata is attached to each log line but not indexed; you can still filter or query these fields after the selector with a pipe without the costs of making them labels.

Further reading and references:

* [Grafana Loki: LogQL and labels](https://grafana.com/docs/loki/latest/logql/)
* [Loki Pipeline Stages](https://grafana.com/docs/loki/latest/pipelines/)

![A slide titled "Keep labels low-cardinality" comparing two cases: the left shows low-cardinality "code" values (200, 404, 500) aggregated into an "index" with a check mark. The right shows high-cardinality "request\_id" with many unique IDs mapped to an "index" and flagged as problematic.](https://mintcdn.com/kodekloud-c4ac6d9a/dEcIUoAxuol-3ZBK/images/Grafana-Loki/Grafana-Loki-on-Kubernetes/Processing-Logs-With-Alloy-Pipelines/low-cardinality-code-vs-request-id.jpg?fit=max\&auto=format\&n=dEcIUoAxuol-3ZBK\&q=85\&s=280bc954761c5f607c73f777bb73971f)

> **[Watch Video](https://learn.kodekloud.com/user/courses/grafana-loki/module/595f27e5-031e-46ea-bffc-ee1a15f73837/lesson/af297b8c-7009-47a2-a32c-0b27a0818c2f)**

> **[Practice Lab](https://learn.kodekloud.com/user/courses/grafana-loki/module/595f27e5-031e-46ea-bffc-ee1a15f73837/lesson/ccf81c33-8b07-4873-974e-33f488f1abd5)**


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.