Lab Solution Taints and Toleration Optional

This lab walkthrough demonstrates Kubernetes taints and tolerations with step-by-step commands and explanations. Follow each numbered step to reproduce the behaviour and understand why pods are scheduled or remain Pending when taints are present.

Check how many nodes exist (including the control plane)

Run:

root@controlplane:~# kubectl get nodes
NAME           STATUS   ROLES                    AGE   VERSION
controlplane   Ready    control-plane,master     17m   v1.20.0
node01         Ready    <none>                   16m   v1.20.0
root@controlplane:~#

Summary:

Node	Role	Status	Kubernetes Version
`controlplane`	`control-plane,master`	`Ready`	`v1.20.0`
`node01`	`<none>`	`Ready`	`v1.20.0`

Answer: There are two nodes: controlplane and node01.

Do any taints exist on node01?

Describe node01 to inspect taints and other conditions:

root@controlplane:~# kubectl describe node node01
Name:               node01
Roles:              <none>
Labels:
  beta.kubernetes.io/arch=amd64
  beta.kubernetes.io/os=linux
  kubernetes.io/arch=amd64
  kubernetes.io/hostname=node01
  kubernetes.io/os=linux
Annotations:
  flannel.alpha.coreos.com/backend-data: {"VNI":1,"VtepMAC":"8e:62:74:26:35:47"}
  flannel.alpha.coreos.com/backend-type: vxlan
  flannel.alpha.coreos.com/kube-subnet-manager: true
  flannel.alpha.coreos.com/public-ip: 10.58.27.11
  kubeadm.alpha.kubernetes.io/cri-socket: /var/run/dockershim.sock
  node.alpha.kubernetes.io/ttl: 0
  volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Fri, 15 Apr 2022 22:57:19 +0000
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:   node01
  AcquireTime:      <unset>
  RenewTime:        Fri, 15 Apr 2022 23:14:20 +0000
Conditions:
  Type                 Status    LastHeartbeatTime                 LastTransitionTime                Reason   Message
  ----                 ------    -----------------                 ------------------                ------   -------
  NetworkUnavailable   False     Fri, 15 Apr 2022 22:57:25 +0000     Fri, 15 Apr 2022 22:57:25 +0000   FlannelIsUp   Flannel is running on this node
  MemoryPressure       False     Fri, 15 Apr 2022 23:12:35 +0000     Fri, 15 Apr 2022 22:57:19 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False     Fri, 15 Apr 2022 23:12:35 +0000     Fri, 15 Apr 2022 22:57:19 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False     Fri, 15 Apr 2022 23:12:35 +0000     Fri, 15 Apr 2022 22:57:19 +0000   KubeletHasSufficientPID     kubelet has sufficient PID available

Conclusion: node01 currently has no taints.

Add a taint to node01

Task: add a taint with key spray, value mortein, and effect NoSchedule on node01. Command pattern: kubectl taint nodes <node-name> key=value:Effect Example:

root@controlplane:~# kubectl taint node node01 spray=mortein:NoSchedule
node/node01 tainted
root@controlplane:~#

Confirm the taint is applied:

root@controlplane:~# kubectl describe node node01
Name:               node01
Roles:              <none>
...
Taints:             spray=mortein:NoSchedule
Unschedulable:      false
...

Create a pod named mosquito (nginx) and observe scheduling

Create the pod:

root@controlplane:~# kubectl run mosquito --image=nginx
pod/mosquito created
root@controlplane:~#

Check pod status:

root@controlplane:~# kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
mosquito    0/1     Pending   0          10s
root@controlplane:~#

Describe the pod to find why it’s Pending:

root@controlplane:~# kubectl describe pod mosquito
Name:         mosquito
Namespace:    default
Priority:     0
Node:         `none`
Labels:       run=mosquito
Status:       Pending
IP:
IPs:          `none`
Containers:
  mosquito:
    Image:      nginx
    Port:       `none`
    Host Port:  `none`
    Environment: `none`
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-f6lxf (ro)
Conditions:
  Type           Status
  PodScheduled   False
Volumes:
  default-token-f6lxf:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-f6lxf
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  `none`
Tolerations:
  node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
  node.kubernetes.io/unreachable:NoExecute op=Exists for 300s

Events:
  Type     Reason             Age               From               Message
  ----     ------             ----              ----               -------
  Warning  FailedScheduling   45s (x2 over 45s) default-scheduler  0/2 nodes are available: 1 node(s) had taint `node-role.kubernetes.io/master:NoSchedule` that the pod didn't tolerate, 1 node(s) had taint `spray=mortein:NoSchedule` that the pod didn't tolerate.
root@controlplane:~#

Explanation: mosquito is Pending because both nodes have taints it does not tolerate:

controlplane has node-role.kubernetes.io/master:NoSchedule
node01 has spray=mortein:NoSchedule

Create a pod named bee (nginx) that tolerates the spray=mortein:NoSchedule taint

kubectl run does not provide a flag to add tolerations directly. Use --dry-run=client to generate a manifest, then add the tolerations section manually. Generate the YAML manifest:

root@controlplane:~# kubectl run bee --image=nginx --dry-run=client -o yaml > bee.yaml
root@controlplane:~#

Edit bee.yaml and add the tolerations block under spec: so it matches the taint’s key, value, and effect. Example bee.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: bee
  labels:
    run: bee
spec:
  containers:
  - name: bee
    image: nginx
    resources: {}
  dnsPolicy: ClusterFirst
  restartPolicy: Always
  tolerations:
  - key: "spray"
    operator: "Equal"
    value: "mortein"
    effect: "NoSchedule"

Apply the manifest:

root@controlplane:~# kubectl apply -f bee.yaml
pod/bee created
root@controlplane:~#

Watch pod creation and status:

root@controlplane:~# kubectl get pods --watch
NAME       READY   STATUS              RESTARTS   AGE
bee        0/1     ContainerCreating   0          13s
mosquito   0/1     Pending             0          3m43s
bee        1/1     Running             0          14s
^C
root@controlplane:~#

Check where bee was scheduled:

root@controlplane:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE    IP           NODE     NOMINATED NODE   READINESS GATES
bee         1/1     Running   0          36s    10.244.1.2   node01   <none>           <none>
mosquito    0/1     Pending   0          4m6s   <none>       <none>   <none>           <none>
root@controlplane:~#

Result: bee was scheduled on node01 because it tolerates the spray=mortein:NoSchedule taint.

A screenshot of a web search results page showing Kubernetes documentation for "taints and tolerations" (search box contains the query). A blue banner at the top mentions the Dockershim removal for Kubernetes 1.24.

Toleration examples:

Use case	Example
Match specific key/value/effect	`yaml tolerations: - key: "key1" operator: "Equal" value: "value1" effect: "NoSchedule"`
Match only key existence	`yaml tolerations: - key: "key1" operator: "Exists" effect: "NoSchedule"`

Inspect the control plane node taint and remove it

Inspect the control plane node to confirm its taint:

root@controlplane:~# kubectl describe node controlplane
Name:               controlplane
Roles:              control-plane,master
...
Taints:             node-role.kubernetes.io/master:NoSchedule
Unschedulable:      false
...

Remove the taint by appending a trailing dash (-) to the taint spec:

root@controlplane:~# kubectl taint node controlplane node-role.kubernetes.io/master:NoSchedule-
node/controlplane untainted
root@controlplane:~#

Observe that mosquito gets scheduled once the control plane taint is removed

Check pod statuses after removing the taint:

root@controlplane:~# kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
bee         1/1     Running   0          2m37s
mosquito    1/1     Running   0          6m7s
root@controlplane:~#

Confirm where mosquito is running:

root@controlplane:~# kubectl get pods -o wide
NAME        READY   STATUS    RESTARTS   AGE     IP           NODE            NOMINATED NODE   READINESS GATES
bee         1/1     Running   0          2m52s   10.244.1.2   node01          <none>           <none>
mosquito    1/1     Running   0          6m22s   10.244.0.4   controlplane    <none>           <none>
root@controlplane:~#

Explanation: Initially both nodes had taints preventing mosquito from being scheduled. After adding a toleration, bee could run on node01. Once the controlplane taint (node-role.kubernetes.io/master:NoSchedule) was removed, the scheduler could place mosquito onto the control plane node. Final answer to the likely quiz question (“Which node is the mosquito pod finally placed on?”): controlplane

Taints prevent pods from being scheduled on nodes unless the pod has a matching toleration. A toleration does not “place” a pod on a node by itself; it simply allows scheduling onto nodes with the matching taint. The scheduler still selects the node according to normal scheduling rules.

Introduction

Core Concepts

Scheduling

Logging & Monitoring

Application Lifecycle Management

Cluster Maintenance

Security

Storage

Networking

Design and Install a Kubernetes Cluster

Install "Kubernetes the kubeadm way"

Troubleshooting

Helm Basics (2025 Updates)

Kustomize Basics (2025 Updates)

Other Topics

Mock Exams

Lab Solution Taints and Toleration Optional

Watch Video