Skip to main content
One Tuesday morning, a customer uploaded a 2 GB spreadsheet. For a few minutes, every other Tidepool customer watched their dashboards spin. Nothing crashed and nothing was rescheduled — there was simply one copy of the API doing one large job, and everyone else waited in line behind it. Maya fixed this by increasing the number of API instances in the job group from one to three and re-submitting the job:
A few seconds later there were three copies of the API running. That’s all scaling a count change does — but only if the resources block beneath it is accurate. Recall the two key resource values used in job specifications: CPU (in megahertz) and memory (in megabytes). These numbers represent the size of the “box” the app is asking the scheduler to reserve. The scheduler uses these values to filter nodes and pack allocations tightly.
A presentation slide titled "Megahertz and Megabytes" showing icons for CPU (500 MHz), memory (256 MB) and a reservation scale. On the right it says "The size of box you reserve" with Docker whale and server icons labeled box-1 through box-4.
Getting reservation values wrong harms density and reliability in two ways:
  • If Maya declares 512 MB when the API actually uses only 128 MB, every machine looks fuller than it really is. The cluster fits fewer copies than it could, and she pays for wasted capacity.
  • If she declares 128 MB when the API actually needs 512 MB, the allocation is placed but the process may be killed by the kernel for exceeding memory (OOM). Because this is a service job, Nomad restarts it and it can enter a crash loop that looks like an application bug but is actually a bad memory estimate.
How to find the right numbers:
  • Start with conservative (slightly larger) reservations.
  • Run the service under real traffic.
  • Inspect the allocation’s actual CPU and memory usage versus the reservation.
  • After observing steady-state usage for a day or more, tighten reservations toward the measured values to increase density safely.
Start with conservative resource reservations, measure actual CPU and memory usage from allocations, then reduce reservations to the observed steady-state requirements to improve density safely.
What happens when you request more copies than the cluster can hold? If Maya asks for 20 replicas, Nomad will place as many as fit and leave the remainder in pending until capacity frees up. This is not an error — the job is still considered running, but it has fewer allocations than requested.
A slide titled "pending Is Not an Error" showing an hourglass labeled "pending" and a magnifying glass labeled "no machine fits" with the note "The job still says running." Below are illustrations of Docker container and server icons with checked labels for box-1 through box-4.
Inspect the placement summary to see how many allocations are pending and why:
  • nomad job status <job> (or the UI) shows a placement summary with counts for running and pending allocations.
  • If you scale up and traffic doesn’t improve, check that placement/summary before assuming something else is wrong.
There’s a faster way to change count than editing the job file: use nomad job scale. It accepts a job name, a group name, and a target count — and applies it immediately.
Important workflow note: the job file in version control still says count = 3, while the cluster now runs 5 copies. If someone later re-submits the job file from git, the cluster will be scaled back to 3 and that change may cause an outage. Best practice:
  • Use nomad job scale for quick, operational scaling during spikes.
  • Immediately update the job file in version control to match the desired state so future deploys are consistent.
You can declare autoscaling intent directly in a job with a scaling block (minimum, maximum, and policy). However, Nomad only stores that declaration — it does not automatically act on it. To have policies enforced automatically, run the Nomad Autoscaler (a separate program).
The scaling block in a job file is a declaration only. Install and run the Nomad Autoscaler to make scaling policies act automatically: https://github.com/hashicorp/nomad-autoscaler
Finally, remember that replicas run on nodes Nomad chose and bind to ports Nomad assigned. If your clients (for example, Tidepool’s web frontend) still point at a single fixed backend address, scaling won’t help — requests may still go to a copy that no longer exists. Ensure your service discovery or load balancing follows your Nomad deployments. Common solutions include the Nomad service registry, Consul, or an external load balancer so clients route to current allocations.

​
Quick reference: common commands and checks

Watch Video

Practice Lab