> ## Documentation Index
> Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Secrets With Vault and Workload Identity

> Using Nomad workload identity and Vault to provide short‑lived, scoped secrets to tasks, removing static credentials from repos and enabling rotation

Two years and a text file.

Every engineer who has ever cloned Tidepool's repository had that database password on their laptop — including the contractor who worked there for only six weeks in the spring.

Rotating that secret meant changing it in the database, updating the file, committing that change, and redeploying every service that reads it, which

<Frame>
  <img src="https://mintcdn.com/kodekloud-c4ac6d9a/l4I9ipyyKXTd2i-D/images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/two-years-env-left-in-repo.jpg?fit=max&auto=format&n=l4I9ipyyKXTd2i-D&q=85&s=1c5ec7c05ea7474115557fbe0640b9eb" alt="A presentation slide titled &#x22;Two Years in a Text File&#x22; showing a .env file left in a repo — &#x22;every laptop has it&#x22; and &#x22;two years old,&#x22; including the contractor's. The slide uses icons (file, briefcase, padlock, Docker whales and server boxes) and labels box-1 through box-4." width="1920" height="1080" data-path="images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/two-years-env-left-in-repo.jpg" />
</Frame>

is why it has never been rotated.

What would have to be true for that password to not be in the repository at all?

* Someone has to provide the credential to the application at startup (or when needed).
* The application must be able to prove its identity without holding a long‑lived secret on disk.
* The proof must be short‑lived and scoped so that rotating credentials becomes a back‑end operation rather than a deploy.

Nomad solves this using workload identities and Vault integration.

When a task starts, Nomad creates a small, signed document describing the task — which job and group it belongs to, and which cluster it's running on. Nomad signs that document with a key only the Nomad servers hold. The task receives this signed document and cannot forge or edit it because any change breaks the signature. This signed description is the task's workload identity.

Vault is configured to trust Nomad's signatures. The task presents its workload identity to Vault; Vault validates the signature, inspects which job the task belongs to, and looks up the permissions for that job. If the configured role allows the job to read database credentials, Vault issues a short‑lived token scoped to those credentials. If the same identity asks for payroll secrets, Vault refuses because the job name in the signed document doesn't match the policy.

What does that cost Maya in the job file? One block: a `vault` block that specifies the role (the name of the rule set in Vault) that applies to this job. There is no static Vault token in the job file, in the agent config, or on any machine.

<Frame>
  <img src="https://mintcdn.com/kodekloud-c4ac6d9a/l4I9ipyyKXTd2i-D/images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/one-block-one-role-expiring-token.jpg?fit=max&auto=format&n=l4I9ipyyKXTd2i-D&q=85&s=a8329f3d65a782bbcff94213f03fb71f" alt="A presentation slide titled &#x22;One Block, One Role&#x22; showing a vault icon, a key and an hourglass to represent a scoped, expiring token, and rows of Docker/container and server icons labeled box-1 through box-4. Large text on the right reads &#x22;A token scoped to that job, and it expires.&#x22;" width="1920" height="1080" data-path="images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/one-block-one-role-expiring-token.jpg" />
</Frame>

Be aware that many older tutorials and blog posts show a Vault token written into the Nomad agent configuration or passed in when the job is submitted. That was the old approach: it was deprecated in Nomad 1.7 and removed completely in 1.10. If you follow one of those guides today, it will fail, and the error won't tell you the guide is four years old.

<Callout icon="lightbulb" color="#1CB2FE">
  Do not put a static Vault token in the agent config or job submission. Use workload identity — it's the supported, secure mechanism since Nomad 1.10.
</Callout>

How does the secret actually reach the application?

Through a template block. A template is a small file Nomad renders for the task: it contains placeholders that Nomad fills before the task starts. In practice, you render a template whose data source pulls the password from Vault at a configured path and set `destination` to a file inside the task's secrets directory. That directory is private to the task; no other task can read it, and Nomad prevents normal file-system introspection into it. Where possible, Nomad backs that directory with an in‑memory filesystem so the secret never touches disk.

Note the words "where possible." Treat in‑memory storage as the normal case, but not an absolute guarantee you'd stake a security review on.

<Frame>
  <img src="https://mintcdn.com/kodekloud-c4ac6d9a/l4I9ipyyKXTd2i-D/images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/template-secrets-per-container-inmemory.jpg?fit=max&auto=format&n=l4I9ipyyKXTd2i-D&q=85&s=962d49b136984816f49c9989413affd5" alt="A slide titled &#x22;The Template and the Secrets Directory&#x22; showing icons for a template file, a secrets folder, and a lock with the caption &#x22;Private to the task, in memory.&#x22; Below are Docker whale and server icons labeled box-1 through box-4, illustrating per-container secret handling." width="1920" height="1080" data-path="images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/template-secrets-per-container-inmemory.jpg" />
</Frame>

What if the secret changes while the application is running?

Nomad's `change_mode` controls that behavior:

| `change_mode` | Behavior |
| - | - |
| `restart` | Nomad restarts the task so the process reads the new value on startup. Use when the app only reads config at startup. |
| `signal` | Nomad sends the configured signal to the process (allowing graceful reloads without dropping connections). |
| `noop` | Nomad rewrites the file and takes no further action. The application must detect and reload the change itself. |

Maya picks `restart` because Tidepool's API reads its configuration only at startup.

<Frame>
  <img src="https://mintcdn.com/kodekloud-c4ac6d9a/l4I9ipyyKXTd2i-D/images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/change-mode-restart-signal-noop-boxes-memory.jpg?fit=max&auto=format&n=l4I9ipyyKXTd2i-D&q=85&s=ab04915b07ac8f4bcc359e2086a01f5e" alt="A slide titled &#x22;change_mode&#x22; showing three mode icons—restart (gear), signal (antenna), and noop (hourglass)—above a row of four boxes labeled box-1 to box-4 with container and server icons. On the right is the caption &#x22;Private to the task, in memory.&#x22;" width="1920" height="1080" data-path="images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/change-mode-restart-signal-noop-boxes-memory.jpg" />
</Frame>

Example: a minimal job fragment showing a `vault` block and a `template` that renders a Vault secret into the task's private secrets directory.

```hcl theme={null}
job "tidepool-api" {
  group "api" {
    task "server" {
      driver = "docker"

      vault {
        # The Vault role name for this job — Nomad uses workload identity to get tokens.
        role = "tidepool-api-role"
      }

      template {
        # Render the DB password into a file the task can read.
        data = <<EOH
{{- with secret "secret/data/tidepool/db" -}}
{{ .Data.data.password }}
{{- end -}}
EOH
        destination  = "secrets/db_password"   # inside the task's private dir
        change_mode  = "restart"
      }

      config {
        image = "tidepool/api:latest"
      }
    }
  }
}
```

After Maya submits this job, the password comes out of the repository and out of the container image. Rotation becomes a Vault operation on a schedule — no deploy required. Maya no longer knows Tidepool's database password, and neither does anyone else at Tidepool. It feels wrong for about a day, then it becomes the norm.

<Frame>
  <img src="https://mintcdn.com/kodekloud-c4ac6d9a/l4I9ipyyKXTd2i-D/images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/vault-rotation-nobody-knows-password.jpg?fit=max&auto=format&n=l4I9ipyyKXTd2i-D&q=85&s=da85d1869bb8b6760a9dabca4e046f4f" alt="A presentation slide that reads &#x22;Nobody Knows the Password&#x22; and &#x22;Rotation is a Vault operation now,&#x22; illustrated with a yellow repo icon, a green shield checkmark, and Docker/server icons labeled box-1 through box-4." width="1920" height="1080" data-path="images/Learn-By-Doing-HashiCorp-Nomad/Scaling-and-Connecting-Workloads/Secrets-With-Vault-and-Workload-Identity/vault-rotation-nobody-knows-password.jpg" />
</Frame>

Although, two minutes after she submitted that file, one of the three API copies started, stopped, and started again, and again.

References and further reading

* [Nomad Workload Identity and Vault Integration](https://www.nomadproject.io/docs/integrations/vault)
* [HashiCorp Vault: Short-lived credentials and dynamic secrets](https://www.vaultproject.io/docs/secrets)
* [Nomad Templates: Render files and query secret engines](https://www.nomadproject.io/docs/job-specification/template)

<CardGroup>
  <Card title="Watch Video" icon="video" cta="Learn more" href="https://learn.kodekloud.com/user/courses/learn-by-doing-hashicorp-nomad/module/d6f53825-444d-461b-adca-bce23263dba7/lesson/aeae934a-a239-4fa3-a3bd-c5ccd380c4ae" />

  <Card title="Practice Lab" icon="flask-conical" cta="Learn more" href="https://learn.kodekloud.com/user/courses/learn-by-doing-hashicorp-nomad/module/d6f53825-444d-461b-adca-bce23263dba7/lesson/2b7098dd-1db3-4f7b-aee5-7533c2435d2f" />
</CardGroup>


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.