Skip to main content
When a job shows as “running” but one of its copies has restarted multiple times, both statements can be true. The job is simply what your application asked Nomad to create — for example, three replicas of an API. Nomad keeps that promise by ensuring three copies exist; when one dies it creates another to preserve the requested count. What’s failing in this scenario is a single copy on a single machine. That single copy has its own name, its own record, and its own logs. That record is the allocation. Most practical debugging in Nomad happens at the allocation level, not at the job level. If you find yourself staring at job status wondering why it says everything is fine, you’re likely looking at the wrong screen.
Look at allocations first when a job behaves strangely. The job represents intent (three copies); the allocation represents one concrete instance of that intent with its own events, logs, and runtime environment.
So how does Maya get from a job to the failing allocation? The job status lists allocations under it, each with a short identifier and a state. One of them may repeatedly say “pending” or “restarting” while the others say “running”. Use that allocation identifier with nomad alloc status — that one screen often answers most questions.
A presentation slide titled "Running and Failing at Once" showing Docker and server icons with the message that a job "restarted 11 times in 4 minutes" yet "says running." Four boxes labeled box-1 through box-4 are displayed along the bottom.
Reading the allocation status The allocation status shows the node the allocation runs on, actual CPU and memory usage versus what it reserved (useful to catch incorrect reservations), and — at the bottom — a list of recent events for each task with human-readable timestamps. The Recent Events list is a timeline. Reading it top-to-bottom often gives the answer: “Task started”, “Task exited with code 1”, “Restarting in 25 seconds”, and so on. Sometimes events only say a task exited, not why — then read the task’s logs. Example output (abbreviated):
Stream allocation logs Nomad’s allocation logs show the task’s stdout. To stream logs live use the follow option (-f or --follow). If the application writes errors to stderr instead of stdout, include the --stderr flag. Example: stream both stdout and stderr as they are produced:
Often the first lines of task output reveal the cause. In our example, the API printed an authentication error before dying: the rendered template produced an empty password because the Vault role allowed reading a path one level away from where the database credentials actually live. The application read an empty password and refused to start. Open a shell inside the allocation If logs aren’t enough, use nomad alloc exec to open a shell in the allocation’s runtime environment. This runs inside the allocation so you can inspect rendered files, configuration, and the filesystem without needing to find the physical machine.
Recovery strategies Nomad uses When a task fails, Nomad uses two recovery strategies governed by the job spec:
  • Restart on the same node: Controlled by the task’s restart stanza: allowed attempts, restart window (interval), and retry delay.
  • Reschedule to another node: If restart attempts are exhausted or the task keeps failing, Nomad will reschedule the allocation to a different node. This is controlled by the reschedule stanza.
Defaults are sensible: service jobs keep retrying with exponential/backoff delays (e.g., 30s, then progressively longer up to an hour) because long-running services should recover over time. Batch jobs usually get fewer retries because repeated attempts often indicate a job-level failure that retries won’t fix.
A presentation slide titled "The Defaults Are Sensible" showing a batch retry policy ("Batch · one attempt in 24h") with icons for a gear, hourglass and stopwatch. Below are Docker whale icons above four checked server boxes labeled box-1 through box-4.
Be cautious about changing default restart or reschedule policies. They are designed for common service and batch patterns — overriding them without understanding the implications can delay recovery or mask systemic failures.
Quick reference — commands and their purpose The debugging loop A concise debugging loop when replicas misbehave:
  1. Read the job status to identify the unhappy allocation.
  2. Inspect the allocation status (nomad alloc status) for events, node, and resource usage.
  3. Read allocation logs (nomad alloc logs) to see what the application reported.
  4. If needed, open a shell in the allocation (nomad alloc exec) to inspect files and the environment directly.
After you fix the underlying issue (for example, adjust the Vault role so the rendered template contains the correct password), Nomad resubmits and the replicas settle. In our example two copies happened to end up on the same node, both running the same image and both listening on port 8080. It may feel surprising, but Nomad did exactly what the job requested. Further reading

Watch Video