Summary of migrating a small production setup from manual SSH deployments to a resilient Nomad cluster, covering architecture, declarative jobs, scheduling flow, self healing, and scaling challenges.
Let’s step back and look at how Tidepool’s platform changed.A few weeks earlier, Tidepool ran on four machines that Maya managed manually. She deployed to each machine in sequence, and the only authoritative record of what should be running lived in her head.When one machine’s disk filled at two in the morning, the software on that host stayed down and Tidepool didn’t notice until a customer emailed.
Today those same four machines are the client half of a resilient Nomad cluster, backed by three small servers that provide the control plane. Maya chose Nomad over Kubernetes for pragmatic reasons she can justify: a six-person company, no dedicated platform team, and a mixed workload that is not only containers. She understands the trade-offs too — a smaller ecosystem and fewer third-party tools when she needs community answers.
Choosing a scheduler is a trade-off. Nomad’s simplicity reduces operational burden for small teams and mixed workloads; Kubernetes offers a larger ecosystem if you need many platform integrations and managed services.
The cluster was created from just two small configuration files and one important number. Architecturally:
Three servers hold cluster state and elect a leader.
Four clients run workloads.
Nomad uses three ports to carry cluster traffic: 4646, 4647, and 4648.
What you hand Nomad is no longer an imperative sequence of SSH commands — it’s a declarative job file that states the desired state. That file describes a job, which contains one or more groups, which contain tasks. These concepts matter when designing and operating workloads.Job, group, task — quick reference:
Resource
Use
Job
The top-level unit you submit to Nomad; describes the workload and scheduling constraints.
Group
A set of tasks that should be colocated or share lifecycle settings (e.g., sidecars).
Task
The actual unit that runs (a process, container, or script).
When you submit a job, you don’t choose a specific machine. Nomad compares the desired state (the job spec) with the cluster’s current state and generates an evaluation. The scheduler converts that evaluation into a placement plan. The elected server leader turns the plan into allocations — concrete assignments of work to particular clients. An allocation represents the actionable unit that runs on a real machine.Scheduling flow (high level):
Submit a job (declarative file).
Nomad produces an evaluation comparing desired vs actual state.
The scheduler generates a placement plan.
The leader commits the plan and creates allocations.
Clients receive allocations and start tasks.
Because this loop runs continuously, the cluster self-heals. When Maya intentionally stopped one host, Nomad re-ran the process and rescheduled the work to other clients, keeping Tidepool available.
Declarative scheduling reduces manual toil, but you should still plan for capacity limits and service discovery — Nomad reschedules tasks, but you must decide how many replicas and how they are discovered.
Tidepool moved from four manually managed machines to a self-healing cluster. That leaves Maya with two important next questions as the platform grows:
What happens when a single API instance isn’t sufficient and she needs more replicas than her four machines can host?
Once multiple copies of the API run (possibly on ephemeral ports), how do other services discover and route traffic to them?
These are the next problems to solve: capacity planning, scaling policies, and service discovery. For more on Nomad concepts and job specifications, see the official documentation: https://www.nomadproject.io/docs.