Demo Encrypting Secret Data at Rest

This guide demonstrates how to enable encryption-at-rest for Kubernetes Secret objects stored in etcd on a kubeadm single-node control plane. You’ll see how Secrets are stored by default, how to enable API server encryption, and how to ensure existing Secrets become encrypted. Prerequisites

A single-node kubeadm cluster (control plane) with terminal access on the control plane node.
Root or sudo access to edit static pod manifests and move files on the host.

Keywords: Kubernetes secrets encryption, etcd encryption, kube-apiserver encryption-provider-config, EncryptionConfiguration, aescbc.

1) Create a Secret and inspect how it is stored

Create a Secret from a literal value:

kubectl create secret generic my-secret --from-literal=key1=supersecret

Confirm the Secret exists:

kubectl get secret my-secret
# NAME        TYPE    DATA   AGE
# my-secret   Opaque  1      10s

Describe the Secret metadata and data size:

kubectl describe secret my-secret
# Name:         my-secret
# Namespace:    default
# Type:  Opaque
# Data
# ====
# key1: 11 bytes

View the Secret manifest (note: Secret data is Base64-encoded, not encrypted):

kubectl get secret my-secret -o yaml
# apiVersion: v1
# data:
#   key1: c3VwZXJzZWNyZXQ=
# kind: Secret
# metadata:
#   name: my-secret
# ...

Decode to verify the original value:

echo 'c3VwZXJzZWNyZXQ=' | base64 --decode
# supersecret

Base64 encoding is not encryption. Anyone with read access to etcd or Secret manifests can decode the values. Never store raw Secret YAMLs with encoded data in public repositories.

2) Inspect how the Secret is stored in etcd (raw view)

etcd stores Kubernetes objects under keys such as /registry/secrets/<namespace>/<name>. To read that raw key you need etcdctl configured to use the v3 API and valid etcd client TLS certificates. Install etcdctl (Debian/Ubuntu example):

sudo apt-get update
sudo apt-get install -y etcd-client

Set ETCDCTL_API=3 and read the raw key (adjust paths and secret name). On kubeadm clusters, the apiserver-etcd-client certs are commonly accepted by etcd:

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  get /registry/secrets/default/my-secret | hexdump -C

If encryption at rest is not enabled, the hexdump or raw output will include the plaintext value (supersecret), demonstrating that direct etcd access with valid certs reveals Secret contents. Reference: etcdctl v3 usage.

3) Check whether the API server is configured for encryption

The kube-apiserver enables encryption-at-rest using the --encryption-provider-config flag. On kubeadm single-node control planes the API server runs as a static pod; inspect the static pod manifest:

/etc/kubernetes/manifests/kube-apiserver.yaml

Look for --encryption-provider-config=/path/to/enc.yaml in the command args. If it’s absent, encryption is not enabled. Tip: You can also inspect the running process args after kube-apiserver is started:

ps aux | grep kube-apiserver | grep -v grep
# Look for: --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

4) Create an EncryptionConfiguration file

Create an EncryptionConfiguration file (example enc.yaml). Only resources listed under resources will be affected—here we list secrets. Provider order matters: the first provider is used for encrypting new writes; later providers are used for decryption of older records. Example enc.yaml using aescbc plus an identity fallback. Replace the example base64 key with a securely generated 32-byte key encoded in base64:

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: y0xTt+U6xgRdNxe4nDYYsij0GgRDoUYC+wAwOKeNfPs=
      - identity: {}

Notes:

identity: {} means “no encryption” as a fallback for decryption. If identity is listed first, data will not be encrypted.
aescbc requires a 32-byte key; provide the key base64-encoded.

Generate a secure 32-byte key example:

head -c 32 /dev/urandom | base64
# e.g. y0xTt+U6xgRdNxe4nDYYsij0GgRDoUYC+wAwOKeNfPs=

Save enc.yaml locally and move it to a directory the kube-apiserver static pod can mount (example /etc/kubernetes/enc):

sudo mkdir -p /etc/kubernetes/enc
sudo mv enc.yaml /etc/kubernetes/enc/enc.yaml
sudo chown root:root /etc/kubernetes/enc/enc.yaml
sudo chmod 600 /etc/kubernetes/enc/enc.yaml

Reference: Kubernetes Encryption at Rest docs.

5) Configure the kube-apiserver static pod to use the configuration

Edit /etc/kubernetes/manifests/kube-apiserver.yaml and make three changes:

Add the --encryption-provider-config flag in the command args:

    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

Mount the host directory into the kube-apiserver container by adding a volumeMount under containers[0].volumeMounts:

    - name: enc
      mountPath: /etc/kubernetes/enc
      readOnly: true

Add a matching volumes entry at the pod spec level:

  - name: enc
    hostPath:
      path: /etc/kubernetes/enc
      type: DirectoryOrCreate

When you save the manifest, the kubelet will recreate the static pod and restart the kube-apiserver process with the new configuration.

Modifying the kube-apiserver manifest will restart the API server. On single-node clusters this causes a brief outage. Plan for downtime or perform changes during a maintenance window.

6) Verify the API server restarted with the encryption flag

After saving the manifest, verify the API server is running and that the --encryption-provider-config flag is present:

ps aux | grep kube-apiserver | grep -v grep
# Ensure the command line contains: --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

Alternatively, watch the static pod status using the container runtime or kubelet tooling (e.g., crictl ps, docker ps) and kubectl -n kube-system get pods after the API server is ready.

7) Create a new Secret (will be encrypted) and inspect etcd

Create a new Secret after the API server is configured for encryption:

kubectl create secret generic my-secret-2 --from-literal=key2=topsecret
kubectl get secret my-secret-2
# NAME           TYPE     DATA   AGE
# my-secret-2    Opaque   1      10s

Read the raw etcd key for this new Secret (use the same etcdctl invocation as before):

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/apiserver-etcd-client.crt \
  --key=/etc/kubernetes/pki/apiserver-etcd-client.key \
  get /registry/secrets/default/my-secret-2 | hexdump -C

You should now observe an encrypted blob in etcd. The stored key prefix may indicate something like k8s:enc:aescbc:v1:key1: followed by ciphertext bytes. The plaintext topsecret will not appear in the raw etcd output. Example (truncated hexdump):

... /registry/secrets/default/my-secret-2
... k8s:enc:aescbc:v1:key1: <ciphertext bytes> ...

8) Re-encrypt existing Secrets

Enabling encryption affects new writes only. Secrets created or written before enabling encryption remain in their original form in etcd until the resource is written again. Options to re-encrypt existing Secrets:

Reapply or replace a specific Secret to force a rewrite:

kubectl get secret my-secret -o yaml | kubectl replace -f -

Force an update with an annotation (no data change required):

kubectl annotate secret my-secret reencrypt="$(date +%s)" --overwrite

Reapply/replace all Secrets across all namespaces (use carefully — this rewrites all secret objects):

kubectl get secrets --all-namespaces -o json | kubectl replace -f -

After rewriting, verify the etcd entry for the previously unencrypted Secret now contains encrypted data using the etcdctl method shown earlier.

Quick reference table

Topic	Details	Example / Command
Where Secrets are stored in etcd	Keys under `/registry/secrets/<namespace>/<name>`	`ETCDCTL_API=3 etcdctl get /registry/secrets/default/my-secret`
Enable encryption flag	kube-apiserver flag	`--encryption-provider-config=/etc/kubernetes/enc/enc.yaml`
Encryption config resource	Only specified resources are affected	`resources: - secrets` in `enc.yaml`
Generate AES-CBC key	32 random bytes base64 encoded	`head -c 32 /dev/urandom \| base64`
Re-encrypt existing Secrets	Rewrite objects so API server re-saves them encrypted	`kubectl get secret my-secret -o yaml \| kubectl replace -f -`

Summary & best practices

Kubernetes Secrets are Base64-encoded by default (not encrypted). Protect etcd access and avoid committing Secrets into public repositories.
Use the kube-apiserver --encryption-provider-config flag with an EncryptionConfiguration to enable encryption-at-rest for resources such as secrets.
Provider order matters: the first provider is used for encrypting new writes; later providers are used for decryption.
After enabling encryption, re-write existing objects to ensure they are stored encrypted in etcd.
Securely store and rotate encryption keys regularly; follow your organization’s key management policies.

Introduction

Core Concepts

Scheduling

Logging & Monitoring

Application Lifecycle Management

Cluster Maintenance

Security

Storage

Networking

Design and Install a Kubernetes Cluster

Install "Kubernetes the kubeadm way"

Troubleshooting

Helm Basics (2025 Updates)

Kustomize Basics (2025 Updates)

Other Topics

Mock Exams

Demo Encrypting Secret Data at Rest

1) Create a Secret and inspect how it is stored

2) Inspect how the Secret is stored in etcd (raw view)

3) Check whether the API server is configured for encryption

4) Create an EncryptionConfiguration file

5) Configure the kube-apiserver static pod to use the configuration

6) Verify the API server restarted with the encryption flag

7) Create a new Secret (will be encrypted) and inspect etcd

8) Re-encrypt existing Secrets

Quick reference table

Summary & best practices

Watch Video

​1) Create a Secret and inspect how it is stored

​2) Inspect how the Secret is stored in etcd (raw view)

​3) Check whether the API server is configured for encryption

​4) Create an EncryptionConfiguration file

​5) Configure the kube-apiserver static pod to use the configuration

​6) Verify the API server restarted with the encryption flag

​7) Create a new Secret (will be encrypted) and inspect etcd

​8) Re-encrypt existing Secrets

​Quick reference table

​Summary & best practices

Watch Video

1) Create a Secret and inspect how it is stored

2) Inspect how the Secret is stored in etcd (raw view)

3) Check whether the API server is configured for encryption

4) Create an EncryptionConfiguration file

5) Configure the kube-apiserver static pod to use the configuration

6) Verify the API server restarted with the encryption flag

7) Create a new Secret (will be encrypted) and inspect etcd

8) Re-encrypt existing Secrets

Quick reference table

Summary & best practices