Skip to main content
In this lesson we walk through the solution for the lab on viewing Kubernetes certificate details and then troubleshoot two outages where the kube-apiserver lost connectivity to etcd. The investigation covers identifying certificate files, inspecting certificate contents (CN, issuer, SANs, validity), and step-by-step remediation when TLS-related configuration errors break control plane connectivity.

​
1) Identify the certificate file used for the kube-apiserver

The kube-apiserver runs as a static pod; its manifest lives at /etc/kubernetes/manifests/kube-apiserver.yaml. Inspect that manifest to find which TLS assets are mounted and which flags point to certificate files. Typical flags include the API server certificate and its private key:
Therefore, the kube-apiserver server certificate (leaf cert) is:
  • /etc/kubernetes/pki/apiserver.crt
The API server also includes client certificates used to authenticate to other components (etcd and kubelet), for example:
  • Certificate used to authenticate kube-apiserver as a client to etcd:
    /etc/kubernetes/pki/apiserver-etcd-client.crt
  • Key used to authenticate kube-apiserver to kubelet:
    /etc/kubernetes/pki/apiserver-kubelet-client.key
API server certificates (server cert and client certs) are typically under /etc/kubernetes/pki. etcd server certs and the etcd CA live under the etcd subdirectory (/etc/kubernetes/pki/etcd). Always confirm the paths shown in the static pod manifests on each control plane node.

​
2) Identify the etcd server certificate (server cert & CA)

The etcd static pod manifest is at /etc/kubernetes/manifests/etcd.yaml. Inspecting the etcd manifest reveals the server and peer TLS configuration, for example:
  • etcd server certificate used to host etcd:
    /etc/kubernetes/pki/etcd/server.crt
  • etcd server CA (trusted CA for etcd TLS):
    /etc/kubernetes/pki/etcd/ca.crt

​
3) View certificate details (common name, issuer, SANs, validity)

To inspect certificate contents, use OpenSSL:
Key fields to check:
  • Subject Common Name (CN)
  • Issuer (the signing CA)
  • Validity interval (Not Before / Not After)
  • Subject Alternative Names (SANs) — DNS names and IPs
Example details for the API server certificate (sample output):
  • Subject CN: kube-apiserver
  • Issuer: CN = kubernetes
  • Validity (example): Not Before: Apr 17 21:17:20 2022 GMT — Not After : Apr 17 21:17:20 2023 GMT → 1 year
  • SANs (example): DNS:controlplane, DNS:kubernetes, DNS:kubernetes.default, DNS:kubernetes.default.svc, DNS:kubernetes.default.svc.cluster.local, plus cluster IP(s) like 10.46.98.9 or 10.96.0.1
Lab question: which alternate name is NOT configured on the kube-apiserver?
From the sample SANs above, kube-master is not present — so kube-master is not configured as a SAN on that API server cert. To inspect etcd server certificate:
Sample values for etcd certs:
  • Subject CN: controlplane (example)
  • Issuer: CN = etcd-ca (etcd CA separate from Kubernetes CA)
  • etcd CA validity (example): Not Before: Apr 17 2022 — Not After : Apr 14 2032 → ~10 years
Note: In many production setups the Kubernetes cluster root CA (/etc/kubernetes/pki/ca.crt) and the etcd CA (/etc/kubernetes/pki/etcd/ca.crt) are separate; ensure services use the correct CA when validating peers.

​
4) Troubleshooting scenario 1 — kubectl fails: connection refused

Symptom: After a maintenance window, kubectl reports:
This indicates kubectl cannot reach the kube-apiserver. When the API server is a static pod, inspect its container and the etcd container directly using the container runtime (examples use docker, but crictl or ctr may be used depending on environment). Steps to diagnose and fix:
  1. Find the kube-apiserver static pod container:
  1. Check kube-apiserver logs:
Observed error (example):
Interpretation: kube-apiserver cannot connect to etcd on 127.0.0.1:2379.
  1. Inspect the etcd static pod/container and its logs:
Example etcd log line indicating the issue:
Root cause: A manifest edit mistakenly pointed etcd to a non-existent cert file (/etc/kubernetes/pki/etcd/server-certificate.crt), so etcd failed to start and kube-apiserver could not connect. Fix:
  • Edit the etcd static pod manifest and correct the --cert-file path:
  • The kubelet will detect the manifest change and restart the etcd static pod. Wait for etcd to become healthy, then kube-apiserver should be able to connect.
Validation:
  • Confirm etcd and kube-apiserver are up and responding:
  • Inspect logs again to verify the previous connection errors have disappeared.
When editing static pod manifests under /etc/kubernetes/manifests, any syntax or path errors can prevent critical control plane components from starting. Make backups or use caution when modifying these files.

​
5) Troubleshooting scenario 2 — kube-apiserver TLS handshake timeout / unknown CA

Symptom: kube-apiserver becomes unresponsive and kubectl reports:
kube-apiserver logs show repeated TLS auth failures:
etcd logs show rejected TLS connections:
Diagnosis:
  • These messages indicate a trust or certificate mismatch between kube-apiserver (client) and etcd (server).
  • Common cause: kube-apiserver is pointed at the wrong CA file for etcd verification. For example, the kube-apiserver manifest had:
but etcd’s server certificate was signed by the etcd CA at /etc/kubernetes/pki/etcd/ca.crt. Since kube-apiserver used the Kubernetes cluster CA (/etc/kubernetes/pki/ca.crt) it did not trust the etcd server certificate, resulting in “certificate signed by unknown authority”. Correct kube-apiserver etcd flags should look like:
Fix:
  1. Edit the kube-apiserver static pod manifest:
  1. Save the file. The kubelet will restart kube-apiserver automatically. Monitor kube-apiserver logs until it successfully authenticates to etcd.
Validation:
  • Check that the authentication handshake errors are gone in the kube-apiserver logs.
  • Verify cluster API responsiveness:

​
Quick reference: common certificate files and their purposes

​
Summary / Key takeaways

  • kube-apiserver uses multiple certificates: its own server cert (apiserver.crt) and client certs for etcd (apiserver-etcd-client.crt) and kubelet (apiserver-kubelet-client.crt). These live under /etc/kubernetes/pki.
  • etcd typically uses its own certificate hierarchy under /etc/kubernetes/pki/etcd and frequently has a separate etcd CA (/etc/kubernetes/pki/etcd/ca.crt).
  • Two common misconfigurations that break kube-apiserver ↔ etcd communication:
    • etcd fails to start because a manifest references a non-existent certificate file.
    • kube-apiserver cannot validate etcd’s TLS certificate because --etcd-cafile points to the wrong CA.
  • When kubectl cannot reach the API server, perform offline diagnostics by inspecting static pod containers with the container runtime (docker, crictl, etc.) and review logs for TLS/connection errors. Always verify file paths in the static pod manifests.

Watch Video