Demo Encrypting Secret Data at Rest

In this guide, we explain how to secure secret data in your Kubernetes cluster by enabling encryption at rest. We start by creating secret objects, examine how Kubernetes encodes them in etcd, and then show you how to configure the API server to encrypt these secrets.

Creating a Secret Object

Begin by launching your Kubernetes playground—a single-node cluster running Kubernetes with ContainerD. Kubernetes makes it easy to create secrets from files, literal values, or environment files. Below are some example commands:

# Create a secret from all files within a directory:
kubectl create secret generic my-secret --from-file=path/to/bar

# Create a secret using specified keys from files:
kubectl create secret generic my-secret --from-file=ssh-privatekey=path/to/id_rsa --from-file=ssh-publickey=path/to/id_rsa.pub

# Create a secret from literal key-value pairs:
kubectl create secret generic my-secret --from-literal=key1=supersecret --from-literal=key2=topsecret

# Create a secret combining a file and a literal:
kubectl create secret generic my-secret --from-file=ssh-privatekey=path/to/id_rsa --from-literal=passphrase=topsecret

# Create a secret from environment files:
kubectl create secret generic my-secret --from-env-file=path/to/foo.env --from-env-file=path/to/bar.env

Additional options like --allow-missing-template-keys, --append-hash, and --dry-run can further refine your secret creation process.

After the command executes, verify the secret:

kubectl create secret generic my-secret --from-literal=key1=supersecret
kubectl get secret my-secret

Using the describe command provides detailed metadata, including the base64-encoded data:

kubectl describe secret my-secret

Viewing the Encoded Secret

Kubernetes stores secret values in base64‑encoded format. Retrieve the secret as YAML to inspect its contents:

kubectl get secret my-secret -o yaml

The output might look like:

apiVersion: v1
data:
  key1: c3VwZXJzWmNyZVQ=
kind: Secret
metadata:
  creationTimestamp: "2022-10-24T05:34:13Z"
  name: my-secret
  namespace: default
  resourceVersion: "2111"
  uid: dfe97c62-5aa1-46a8-b71c-ffa0cd4c08ec
type: Opaque

To decode the secret value:

echo "c3VwZXJzWmNyZVQ=" | base64 --decode

This reveals that the stored secret is only encoded, not encrypted, making it potentially accessible to anyone with access to the YAML output or an etcd dump.

Inspecting Secret Data in etcd

etcd is the key-value store where Kubernetes persists cluster data. Without encryption at rest, secret values remain only base64-encoded, allowing anyone with access to etcd to decode them. Use the etcdctl client (API version 3) to query etcd:

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret | hexdump -C

Before running the above command, ensure that the etcdctl client is installed. On Ubuntu, install it using:

apt-get install etcd-client

Verify the installation:

etcdctl

Also, check that your control plane node can access the necessary certificate files:

ls /etc/kubernetes/pki/etcd/ca.crt

The hexdump output will display the raw data, illustrating that without encryption, the secret’s value is visible within etcd.

Enabling Encryption at Rest

To protect sensitive data stored in etcd, Kubernetes offers an encryption at rest mechanism using an encryption provider configuration. First, verify if encryption is enabled by checking the kube-apiserver process:

ps -aux | grep kube-api | grep "encryption-provider-config"

If no configuration is found, follow these steps:

1. Create an Encryption Configuration File

Generate a random 32-byte key (base64-encoded) with:

head -c 32 /dev/urandom | base64

Next, create a YAML file (e.g., enc.yaml) with the following content (replace the sample key with your generated key):

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
  providers:
    - aescbc:
        keys:
          - name: key1
            secret: y0xTt+U6xgRdNxe4nDYYsijOGgRDoUYC+wAwOKeNfPs=  # Replace with your generated key
    - identity: {}

Review the file to ensure accuracy:

cat enc.yaml

2. Mount the Encryption Configuration File into the API Server

Next, incorporate the configuration into the kube-apiserver by performing these steps:

1. Move the encryption configuration file to a secure directory:

   mkdir -p /etc/kubernetes/enc
   mv enc.yaml /etc/kubernetes/enc/
   

2. Modify the kube-apiserver manifest (found at /etc/kubernetes/manifests/kube-apiserver.yaml) by adding the --encryption-provider-config flag. Include a new volume and volume mount for the /etc/kubernetes/enc directory. For example:

   spec:
     containers:
     - command:
       - kube-apiserver
       # ... other flags ...
       - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
       volumeMounts:
         # ... other volume mounts ...
         - name: enc
           mountPath: /etc/kubernetes/enc
           readOnly: true
     volumes:
       # ... other volumes ...
       - name: enc
         hostPath:
           path: /etc/kubernetes/enc
           type: DirectoryOrCreate
   

After saving the changes, the kube-apiserver will restart and begin using the new encryption configuration.

Verifying Encryption

Once encryption is activated, any new secret you create will be encrypted at rest. Create a new secret:

kubectl create secret generic my-secret-2 --from-literal=key2=topsecret

Verify the secret's creation:

kubectl get secret

Then inspect etcd to ensure the secret’s value is encrypted:

ETCDCTL_API=3 etcdctl \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  get /registry/secrets/default/my-secret-2 | hexdump -C

You should observe that the secret’s data no longer appears in plain text. Remember, secrets created before enabling encryption remain unencrypted until updated. To re-encrypt existing secrets, use:

kubectl get secret --all-namespaces -o json | kubectl replace -f -

Then confirm that the secrets in etcd are now encrypted.

Conclusion

This guide demonstrated how Kubernetes handles secret data by showing that, by default, secrets are only base64-encoded—not encrypted—in etcd. We then detailed the process for enabling encryption at rest: creating an encryption configuration file, mounting it into the API server, and verifying that new secrets store their data securely. By following these steps, you can significantly enhance the security posture of your Kubernetes cluster.

Happy encrypting!

Additional Resources

For further reading on securing your Kubernetes environment, consider consulting the Kubernetes Documentation.