> ## Documentation Index > Fetch the complete documentation index at: https://notes.kodekloud.com/llms.txt > Use this file to discover all available pages before exploring further. # Lab Solution View Certification Details > Guide to inspecting Kubernetes certificate files and details and troubleshooting kube-apiserver to etcd TLS connectivity issues with step by step diagnosis and remediation In this lesson we walk through the solution for the lab on viewing Kubernetes certificate details and then troubleshoot two outages where the kube-apiserver lost connectivity to etcd. The investigation covers identifying certificate files, inspecting certificate contents (CN, issuer, SANs, validity), and step-by-step remediation when TLS-related configuration errors break control plane connectivity. *** ## 1) Identify the certificate file used for the kube-apiserver The kube-apiserver runs as a static pod; its manifest lives at `/etc/kubernetes/manifests/kube-apiserver.yaml`. Inspect that manifest to find which TLS assets are mounted and which flags point to certificate files. Typical flags include the API server certificate and its private key: ```yaml theme={null} - --tls-cert-file=/etc/kubernetes/pki/apiserver.crt - --tls-private-key-file=/etc/kubernetes/pki/apiserver.key ``` Therefore, the kube-apiserver server certificate (leaf cert) is: * `/etc/kubernetes/pki/apiserver.crt` The API server also includes client certificates used to authenticate to other components (etcd and kubelet), for example: ```yaml theme={null} - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --kubelet-client-certificate=/etc/kubernetes/pki/apiserver-kubelet-client.crt - --kubelet-client-key=/etc/kubernetes/pki/apiserver-kubelet-client.key ``` * Certificate used to authenticate kube-apiserver as a client to etcd:\ `/etc/kubernetes/pki/apiserver-etcd-client.crt` * Key used to authenticate kube-apiserver to kubelet:\ `/etc/kubernetes/pki/apiserver-kubelet-client.key` API server certificates (server cert and client certs) are typically under `/etc/kubernetes/pki`. etcd server certs and the etcd CA live under the `etcd` subdirectory (`/etc/kubernetes/pki/etcd`). Always confirm the paths shown in the static pod manifests on each control plane node. *** ## 2) Identify the etcd server certificate (server cert & CA) The etcd static pod manifest is at `/etc/kubernetes/manifests/etcd.yaml`. Inspecting the etcd manifest reveals the server and peer TLS configuration, for example: ```yaml theme={null} - --cert-file=/etc/kubernetes/pki/etcd/server.crt - --key-file=/etc/kubernetes/pki/etcd/server.key - --trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt - --peer-cert-file=/etc/kubernetes/pki/etcd/peer.crt - --peer-key-file=/etc/kubernetes/pki/etcd/peer.key - --peer-trusted-ca-file=/etc/kubernetes/pki/etcd/ca.crt ``` * etcd server certificate used to host etcd:\ `/etc/kubernetes/pki/etcd/server.crt` * etcd server CA (trusted CA for etcd TLS):\ `/etc/kubernetes/pki/etcd/ca.crt` *** ## 3) View certificate details (common name, issuer, SANs, validity) To inspect certificate contents, use OpenSSL: ```bash theme={null} openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout ``` Key fields to check: * Subject Common Name (CN) * Issuer (the signing CA) * Validity interval (`Not Before` / `Not After`) * Subject Alternative Names (SANs) — DNS names and IPs Example details for the API server certificate (sample output): * Subject CN: `kube-apiserver` * Issuer: `CN = kubernetes` * Validity (example): `Not Before: Apr 17 21:17:20 2022 GMT` — `Not After : Apr 17 21:17:20 2023 GMT` → 1 year * SANs (example): `DNS:controlplane`, `DNS:kubernetes`, `DNS:kubernetes.default`, `DNS:kubernetes.default.svc`, `DNS:kubernetes.default.svc.cluster.local`, plus cluster IP(s) like `10.46.98.9` or `10.96.0.1` Lab question: which alternate name is NOT configured on the kube-apiserver?\ From the sample SANs above, `kube-master` is not present — so `kube-master` is not configured as a SAN on that API server cert. To inspect etcd server certificate: ```bash theme={null} openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout ``` Sample values for etcd certs: * Subject CN: `controlplane` (example) * Issuer: `CN = etcd-ca` (etcd CA separate from Kubernetes CA) * etcd CA validity (example): `Not Before: Apr 17 2022` — `Not After : Apr 14 2032` → \~10 years Note: In many production setups the Kubernetes cluster root CA (`/etc/kubernetes/pki/ca.crt`) and the etcd CA (`/etc/kubernetes/pki/etcd/ca.crt`) are separate; ensure services use the correct CA when validating peers. *** ## 4) Troubleshooting scenario 1 — kubectl fails: connection refused Symptom: After a maintenance window, `kubectl` reports: ```text theme={null} The connection to the server was refused - did you specify the right host or port? ``` This indicates kubectl cannot reach the kube-apiserver. When the API server is a static pod, inspect its container and the etcd container directly using the container runtime (examples use `docker`, but `crictl` or `ctr` may be used depending on environment). Steps to diagnose and fix: 1. Find the kube-apiserver static pod container: ```bash theme={null} docker ps -a | grep kube-apiserver ``` 2. Check kube-apiserver logs: ```bash theme={null} docker logs ``` Observed error (example): ```Go theme={null} connection error: desc = "transport: Error while dialing dial tcp 127.0.0.1:2379: connect: connection refused" ``` Interpretation: kube-apiserver cannot connect to etcd on `127.0.0.1:2379`. 3. Inspect the etcd static pod/container and its logs: ```bash theme={null} docker ps -a | grep etcd docker logs ``` Example etcd log line indicating the issue: ```bash theme={null} C | etcdmain: open /etc/kubernetes/pki/etcd/server-certificate.crt: no such file or directory ``` Root cause: A manifest edit mistakenly pointed etcd to a non-existent cert file (`/etc/kubernetes/pki/etcd/server-certificate.crt`), so etcd failed to start and kube-apiserver could not connect. Fix: * Edit the etcd static pod manifest and correct the `--cert-file` path: ```bash theme={null} vi /etc/kubernetes/manifests/etcd.yaml # change: # - --cert-file=/etc/kubernetes/pki/etcd/server-certificate.crt # to: # - --cert-file=/etc/kubernetes/pki/etcd/server.crt ``` * The kubelet will detect the manifest change and restart the etcd static pod. Wait for etcd to become healthy, then kube-apiserver should be able to connect. Validation: * Confirm etcd and kube-apiserver are up and responding: ```bash theme={null} kubectl get nodes kubectl get pods -n kube-system ``` * Inspect logs again to verify the previous connection errors have disappeared. When editing static pod manifests under `/etc/kubernetes/manifests`, any syntax or path errors can prevent critical control plane components from starting. Make backups or use caution when modifying these files. *** ## 5) Troubleshooting scenario 2 — kube-apiserver TLS handshake timeout / unknown CA Symptom: kube-apiserver becomes unresponsive and `kubectl` reports: ```text theme={null} Unable to connect to the server: net/http: TLS handshake timeout ``` kube-apiserver logs show repeated TLS auth failures: ```text theme={null} transport: authentication handshake failed: x509: certificate signed by unknown authority ``` etcd logs show rejected TLS connections: ```text theme={null} embed: rejected connection from "127.0.0.1:xxxxx" (error "remote error: tls: bad certificate", ServerName "") ``` Diagnosis: * These messages indicate a trust or certificate mismatch between kube-apiserver (client) and etcd (server). * Common cause: kube-apiserver is pointed at the wrong CA file for etcd verification. For example, the kube-apiserver manifest had: ```yaml theme={null} - --etcd-cafile=/etc/kubernetes/pki/ca.crt ``` but etcd's server certificate was signed by the etcd CA at `/etc/kubernetes/pki/etcd/ca.crt`. Since kube-apiserver used the Kubernetes cluster CA (`/etc/kubernetes/pki/ca.crt`) it did not trust the etcd server certificate, resulting in "certificate signed by unknown authority". Correct kube-apiserver etcd flags should look like: ```yaml theme={null} - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt - --etcd-certfile=/etc/kubernetes/pki/apiserver-etcd-client.crt - --etcd-keyfile=/etc/kubernetes/pki/apiserver-etcd-client.key - --etcd-servers=https://127.0.0.1:2379 ``` Fix: 1. Edit the kube-apiserver static pod manifest: ```bash theme={null} vi /etc/kubernetes/manifests/kube-apiserver.yaml # ensure: # - --etcd-cafile=/etc/kubernetes/pki/etcd/ca.crt ``` 2. Save the file. The kubelet will restart kube-apiserver automatically. Monitor kube-apiserver logs until it successfully authenticates to etcd. Validation: * Check that the authentication handshake errors are gone in the kube-apiserver logs. * Verify cluster API responsiveness: ```bash theme={null} kubectl get nodes kubectl get pods -n kube-system ``` *** ## Quick reference: common certificate files and their purposes | File path | Purpose / Use | | - | - | | `/etc/kubernetes/pki/apiserver.crt` | kube-apiserver server TLS certificate (leaf cert) | | `/etc/kubernetes/pki/apiserver.key` | kube-apiserver server private key | | `/etc/kubernetes/pki/apiserver-etcd-client.crt` | Certificate kube-apiserver uses as a client when connecting to etcd | | `/etc/kubernetes/pki/apiserver-etcd-client.key` | Private key for the above etcd-client cert | | `/etc/kubernetes/pki/apiserver-kubelet-client.crt` | Certificate for kube-apiserver to authenticate to kubelet | | `/etc/kubernetes/pki/ca.crt` | Kubernetes cluster root CA (may be different from etcd CA) | | `/etc/kubernetes/pki/etcd/ca.crt` | etcd CA used to sign etcd server certs (trusted by etcd clients) | | `/etc/kubernetes/pki/etcd/server.crt` | etcd server certificate | | `/etc/kubernetes/pki/etcd/server.key` | etcd server private key | *** ## Summary / Key takeaways * kube-apiserver uses multiple certificates: its own server cert (`apiserver.crt`) and client certs for etcd (`apiserver-etcd-client.crt`) and kubelet (`apiserver-kubelet-client.crt`). These live under `/etc/kubernetes/pki`. * etcd typically uses its own certificate hierarchy under `/etc/kubernetes/pki/etcd` and frequently has a separate etcd CA (`/etc/kubernetes/pki/etcd/ca.crt`). * Two common misconfigurations that break kube-apiserver ↔ etcd communication: * etcd fails to start because a manifest references a non-existent certificate file. * kube-apiserver cannot validate etcd's TLS certificate because `--etcd-cafile` points to the wrong CA. * When `kubectl` cannot reach the API server, perform offline diagnostics by inspecting static pod containers with the container runtime (`docker`, `crictl`, etc.) and review logs for TLS/connection errors. Always verify file paths in the static pod manifests. *** ## Links and References * [Kubernetes Certificates and TLS](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/) * [etcd TLS Setup](https://etcd.io/docs/v3.5/op-guide/security/) * [Kubernetes Static Pods](https://kubernetes.io/docs/tasks/configure-pod-container/static-pod/) * [kubectl Reference](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands) This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.