DevSecOps - Kubernetes DevOps & Security

DevSecOps Pipeline

Demo Fixing Script and Read Only Root File System

In this tutorial, you’ll learn how to troubleshoot a Deployment script that skips full manifest updates and how to enable a read-only root filesystem in your container without breaking writable paths like /tmp.

Table of Contents


Problem Overview

You’ve added readOnlyRootFilesystem: true to your container’s securityContext, but after deployment, the pod spec doesn’t reflect this change. The Deployment script only updates the image, never reapplies the full YAML, so new securityContext settings are ignored.

Initial Deployment Configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devsecops
  labels:
    app: devsecops
spec:
  replicas: 2
  selector:
    matchLabels:
      app: devsecops
  template:
    metadata:
      labels:
        app: devsecops
    spec:
      serviceAccountName: default
      containers:
        - name: devsecops-container
          image: replace
          securityContext:
            runAsNonRoot: true
            runAsUser: 100
            readOnlyRootFilesystem: true
---
apiVersion: v1
kind: Service
metadata:
  name: devsecops-svc
  labels:
    app: devsecops
spec:
  type: NodePort
  selector:
    app: devsecops
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP

After applying:

kubectl get po devsecops-66cd4b7475-8fn5d -o yaml | grep readOnlyRootFilesystem
# <no output>

Why readOnlyRootFilesystem Isn’t Applied

Because the deployment script checks for an existing Deployment and only runs kubectl set image…, it never reapplies the manifest changes (securityContext, volumes, etc.).

Original Deployment Script Analysis

#!/bin/bash
# Replace image placeholder
sed -i "s|replace|${imageName}|g" k8s_deployment_service.yaml
kubectl get deployment ${deploymentName} > /dev/null

if [[ $? -ne 0 ]]; then
    echo "deployment ${deploymentName} doesn't exist"
    kubectl apply -f k8s_deployment_service.yaml
else
    echo "deployment ${deploymentName} exists, updating image to ${imageName}"
    kubectl -n default set image deployment ${deploymentName} \
      ${containerName}=${imageName} --record=true
fi

This script never picks up any YAML changes besides the image tag.

Quick Workaround: Always Apply Manifest

#!/bin/bash
sed -i "s|replace|${imageName}|g" k8s_deployment_service.yaml
# Always apply full manifest to pick up config changes
kubectl -n default apply -f k8s_deployment_service.yaml

Warning

Always applying the full manifest will restart pods and may cause brief downtime. Plan for rolling updates.

After pushing this change, pods now crash with:

kubectl logs devsecops-6d547ad96b-67x7n
# org.springframework.context.ApplicationContextException: Unable to start web server;
# nested exception is org.springframework.boot.web.server.WebServerException:
# Unable to create tempDir. java.io.tmpdir is set to /tmp

Since /tmp is on a read-only root, the Spring Boot app can’t create its temp directory.

Solution: Mounting an emptyDir Volume

To provide a writable /tmp while keeping the rest of the filesystem read-only, add an emptyDir volume and mount it at /tmp.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: devsecops
  labels:
    app: devsecops
spec:
  replicas: 2
  selector:
    matchLabels:
      app: devsecops
  template:
    metadata:
      labels:
        app: devsecops
    spec:
      serviceAccountName: default
      volumes:
        - name: tmp-vol
          emptyDir: {}
      containers:
        - name: devsecops-container
          image: replace
          volumeMounts:
            - name: tmp-vol
              mountPath: /tmp
          securityContext:
            runAsNonRoot: true
            runAsUser: 100
            readOnlyRootFilesystem: true
---
apiVersion: v1
kind: Service
metadata:
  name: devsecops-svc
  labels:
    app: devsecops
spec:
  type: NodePort
  selector:
    app: devsecops
  ports:
    - port: 8080
      targetPort: 8080
      protocol: TCP

Note

The emptyDir volume is ephemeral and only persists for the pod’s lifetime. Use a PersistentVolume if you need data durability.

Applying the Updated Manifest

kubectl -n default apply -f k8s_deployment_service.yaml

Verification Steps

StepCommandExpected Output
1. Check pods are runningkubectl get podsAll pods in Running state
2. Confirm readOnlyRootFilesystemkubectl get po devsecops-xxx -o yaml | grep readOnlyRootFilesystemreadOnlyRootFilesystem: true
3. Test write to /etckubectl exec -it devsecops-xxx -- touch /etc/deny && echo oktouch: cannot touch '/etc/deny': Read-only file system
4. Test write to /tmpkubectl exec -it devsecops-xxx -- touch /tmp/allow && echo okok
5. Verify application startup logskubectl logs devsecops-xxxTomcat and Spring Boot start messages

Best Practices

ResourcePurposeReference
securityContextEnforce container security policiesKubernetes Docs
emptyDir volumeProvide ephemeral writable storageemptyDir Volume
Rolling UpdatesMinimize downtime when applying new manifestsDeployments

References

Watch Video

Watch video content

Previous
Demo Kubesec