Troubleshoot pods, nodes, the control plane and the network

Dit deel is nog niet vertaald. Je ziet de Engelse tekst.

$ als gewone gebruiker# als root (of met sudo)

Vaardigheden in dit hoofdstuk

Clusters, nodes, componenten, services en netwerken debuggen

Debug a pod

The pod status tells you where to look.

Status	Usual cause	Look at
Pending	No node fits (resources, taints, affinity), PVC not bound	`describe pod` events
ImagePullBackOff / ErrImagePull	Wrong image name/tag, private registry without secret	`describe pod` events
CrashLoopBackOff	The app exits: bad config, missing env, wrong command	`logs --previous`
CreateContainerConfigError	Missing ConfigMap or Secret	`describe pod`
Running but not Ready	Readiness probe fails	`describe pod`, app logs
OOMKilled	Memory limit too low	`describe pod` (Last State)

$kubectl get pods -n shop -o wide
$kubectl describe pod web-7d9f -n shop
$kubectl logs web-7d9f -n shop
$kubectl logs web-7d9f -n shop --previous
$kubectl logs web-7d9f -n shop -c sidecar
$kubectl get events -n shop --sort-by=.lastTimestamp
$kubectl exec -it web-7d9f -n shop -- sh
$kubectl debug -it web-7d9f -n shop --image=busybox:1.36 --target=nginx

12.2

Debug a node

A NotReady node usually has a stopped or broken kubelet or container runtime.

$kubectl get nodes
$kubectl describe node worker1 | grep -A8 Conditions
#ssh worker1
#systemctl status kubelet
#journalctl -u kubelet -n 50 --no-pager
#systemctl status containerd
#crictl ps -a
#crictl logs CONTAINER_ID
#df -h ; free -m
#systemctl enable --now kubelet

Symptom in the kubelet log	Fix
`failed to load kubelet config file`	Wrong path or typo in `/var/lib/kubelet/config.yaml`
`unable to connect to the runtime`	Start containerd; check the socket path
Certificate errors	Wrong CA or expired certs (`kubeadm certs check-expiration`)
Disk or memory pressure	Clean up space; check `Conditions`

12.3

Debug the control plane

The control-plane parts are static pods. When the API server is down, kubectl does not work — use crictl and the log files.

$kubectl get pods -n kube-system
#crictl ps -a | grep -E 'apiserver|etcd|scheduler|controller'
#crictl logs $(crictl ps -a --name kube-apiserver -q | head -1)
#ls /var/log/pods/
#tail -n 30 /var/log/pods/kube-system_kube-apiserver-*/kube-apiserver/*.log
#vim /etc/kubernetes/manifests/kube-apiserver.yaml

Examentip

A typo in a manifest under /etc/kubernetes/manifests/ (a wrong flag, a wrong certificate path) is a favourite exam task. Compare with a working cluster or the docs, fix, wait 30 – 60 s.

12.4

Debug Services and DNS

Follow the path: DNS → Service → endpoints → pod port.

$kubectl get svc web -n shop -o wide
$kubectl get endpointslices -n shop -l kubernetes.io/service-name=web
$kubectl get pods -n shop -l app=web --show-labels
$kubectl run t --rm -it --image=busybox:1.36 --restart=Never -- nslookup web.shop
$kubectl run t --rm -it --image=busybox:1.36 --restart=Never -- wget -qO- -T 3 http://web.shop
$kubectl get pods -n kube-system -l k8s-app=kube-dns
$kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

Finding	Cause
No endpoints	Selector does not match pod labels, or pods not ready
Endpoints OK, connection refused	`targetPort` is not the port the app listens on
Name does not resolve	CoreDNS down, or a NetworkPolicy blocks port 53
Works by IP, not from other namespaces	A NetworkPolicy

12.5

Use logs and metrics of the cluster

Resource use and cluster events help find the reason.

$kubectl top nodes
$kubectl top pods -A --sort-by=memory | head
$kubectl get events -A --field-selector type=Warning
$kubectl get --raw='/readyz?verbose' | tail

Alle commando's in dit hoofdstuk