> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Allocations and the Task Lifecycle
> How to debug Nomad jobs by inspecting allocations, reading allocation events and logs, using nomad alloc status/logs/exec, and understanding restart and reschedule behavior
When a job shows as "running" but one of its copies has restarted multiple times, both statements can be true. The job is simply what your application asked [Nomad](https://www.nomadproject.io/docs) to create — for example, three replicas of an API. Nomad keeps that promise by ensuring three copies exist; when one dies it creates another to preserve the requested count. What's failing in this scenario is a single copy on a single machine.
That single copy has its own name, its own record, and its own logs. That record is the allocation. Most practical debugging in [Nomad](https://www.nomadproject.io/docs) happens at the allocation level, not at the job level. If you find yourself staring at job status wondering why it says everything is fine, you're likely looking at the wrong screen.
Look at allocations first when a job behaves strangely. The job represents intent (three copies); the allocation represents one concrete instance of that intent with its own events, logs, and runtime environment.
So how does Maya get from a job to the failing allocation?
The job status lists allocations under it, each with a short identifier and a state. One of them may repeatedly say "pending" or "restarting" while the others say "running". Use that allocation identifier with `nomad alloc status` — that one screen often answers most questions.
Reading the allocation status
The allocation status shows the node the allocation runs on, actual CPU and memory usage versus what it reserved (useful to catch incorrect reservations), and — at the bottom — a list of recent events for each task with human-readable timestamps.
The Recent Events list is a timeline. Reading it top-to-bottom often gives the answer: "Task started", "Task exited with code 1", "Restarting in 25 seconds", and so on. Sometimes events only say a task exited, not why — then read the task's logs.
Example output (abbreviated):
```bash theme={null}
$ nomad alloc status alb2c3
Node: box-2 CPU 240/500 MHz
Recent Events: Terminated — exit code 1
```
Stream allocation logs
Nomad's allocation logs show the task's stdout. To stream logs live use the follow option (`-f` or `--follow`). If the application writes errors to stderr instead of stdout, include the `--stderr` flag.
Example: stream both stdout and stderr as they are produced:
```bash theme={null}
$ nomad alloc logs -f --stderr alb2c3
```
Often the first lines of task output reveal the cause. In our example, the API printed an authentication error before dying: the rendered template produced an empty password because the [Vault](https://www.vaultproject.io/docs) role allowed reading a path one level away from where the database credentials actually live. The application read an empty password and refused to start.
Open a shell inside the allocation
If logs aren't enough, use `nomad alloc exec` to open a shell in the allocation's runtime environment. This runs inside the allocation so you can inspect rendered files, configuration, and the filesystem without needing to find the physical machine.
```bash theme={null}
$ nomad alloc exec -i -t alb2c3 /bin/sh
# a shell inside the running allocation
```
Recovery strategies Nomad uses
When a task fails, Nomad uses two recovery strategies governed by the job spec:
* Restart on the same node: Controlled by the task's `restart` stanza: allowed attempts, restart window (interval), and retry delay.
* Reschedule to another node: If restart attempts are exhausted or the task keeps failing, Nomad will reschedule the allocation to a different node. This is controlled by the `reschedule` stanza.
Defaults are sensible: service jobs keep retrying with exponential/backoff delays (e.g., 30s, then progressively longer up to an hour) because long-running services should recover over time. Batch jobs usually get fewer retries because repeated attempts often indicate a job-level failure that retries won't fix.
Be cautious about changing default restart or reschedule policies. They are designed for common service and batch patterns — overriding them without understanding the implications can delay recovery or mask systemic failures.
Quick reference — commands and their purpose
| Command | Purpose | Example |
| -: | - | - |
| `nomad job status ` | View job-level state and list allocations | `nomad job status api-job` |
| `nomad alloc status ` | Inspect allocation events, resources, and node | `nomad alloc status alb2c3` |
| `nomad alloc logs -f --stderr ` | Stream task stdout and stderr from an allocation | `nomad alloc logs -f --stderr alb2c3` |
| `nomad alloc exec -i -t /bin/sh` | Open a shell inside the allocation runtime | `nomad alloc exec -i -t alb2c3 /bin/sh` |
The debugging loop
A concise debugging loop when replicas misbehave:
1. Read the job status to identify the unhappy allocation.
2. Inspect the allocation status (`nomad alloc status`) for events, node, and resource usage.
3. Read allocation logs (`nomad alloc logs`) to see what the application reported.
4. If needed, open a shell in the allocation (`nomad alloc exec`) to inspect files and the environment directly.
After you fix the underlying issue (for example, adjust the [Vault](https://www.vaultproject.io/docs) role so the rendered template contains the correct password), Nomad resubmits and the replicas settle. In our example two copies happened to end up on the same node, both running the same image and both listening on port 8080. It may feel surprising, but Nomad did exactly what the job requested.
Further reading
* [Nomad Documentation](https://www.nomadproject.io/docs)
* [Vault Documentation](https://www.vaultproject.io/docs)
This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.