Kubernetes, as a yard
of shipping containers

Bootstrapping: an empty yard comes up in order

Nothing in Kubernetes can happen before the API server is serving, and the API server cannot serve before etcd is up: that is the order kubeadm init follows. The scheduler and controllers are just clients of the API server, so they start last and can restart any time.

A node joins by starting a kubelet that registers itself with the API server (kubeadm join hands it a bootstrap token and the CA). It is not schedulable until it is Ready, which needs a container runtime and a CNI network plugin running on it. Managed clusters (EKS, GKE, DOKS) do the control-plane half for you; the node half is still this.

kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f cni.yaml                     # nodes go Ready once this runs
kubeadm join 10.0.0.1:6443 --token ... --discovery-token-ca-cert-hash ...
kubectl get nodes --watch                      # NotReady -> Ready
kubectl apply -f app/

A Pod is a container, and a container holds crates

A Pod is the smallest thing Kubernetes schedules. It is one or more containers that always land on the same node, share one network namespace (one IP, one set of ports: the socket) and can share volumes. The tiny pause container holds that namespace open while app containers come and go.

Config and secrets are not baked into the image; they are mounted in from ConfigMaps and Secrets, so the same image runs in every environment. A PersistentVolumeClaim gives a pod storage that outlives the container (the db pod has one).

# what is in a pod
kubectl get pod web-6b8fd-xk2p9 -o yaml
kubectl describe pod web-6b8fd-xk2p9
kubectl logs web-6b8fd-xk2p9 -c nginx

Try: click a container for its name, node, container count and restarts.

A Node is a dock with a booth

A node is a machine, virtual or real. The kubelet on it watches the API server for pods bound to it, asks the container runtime (containerd, CRI-O) to pull images and start containers, runs the probes, and reports status back. kube-proxy programs the node's networking so Service addresses reach pods.

Nodes have capacity: CPU, memory, pod count. The scheduler only places a pod where its requests fit, which is why the gauge matters and why a full yard leaves pods Pending.

kubectl get nodes -o wide
kubectl describe node node-a      # capacity, allocated, conditions
kubectl top node                   # live usage (metrics-server)

The control plane is the harbour tower

Everything goes through the API server: kubectl, the kubelets, the controllers. It validates requests and stores the result in etcd, the only stateful part. Nothing else talks to etcd directly.

Controllers are loops: each watches one kind of object and nudges the world toward what the object says. The Deployment controller makes ReplicaSets; the ReplicaSet controller makes Pods; the scheduler picks a node for each Pod that has none. None of them call each other. They all just watch and write, which is why the cluster keeps working while any one of them restarts.

kubectl get --raw /healthz
kubectl get events --sort-by=.lastTimestamp
kubectl get rs                      # the ReplicaSets a Deployment made

The control room: what the tower actually does

The API server is a pipeline, and nothing reaches etcd by any other door: authentication (client certificates, tokens, OIDC), authorization (RBAC), mutating admission (defaults, sidecar injection), validating admission (policies), schema validation against the OpenAPI types, then the write. Every object stored gets a new resourceVersion, and every watcher sees the changes in that order.

etcd is a Raft group, usually three members. One is the leader for the current term. A write is appended to the leader's log, replicated to the followers, committed once a quorum (two of three) has acknowledged it, then applied to the key-value store, which is where watches stream from. Lose a follower and nothing changes: the leader still has its quorum. Lose the leader and the followers' election timers fire, one asks for votes, the term goes up by one, and for those seconds the API server can serve reads from its watch cache but cannot commit a write; the controllers simply retry.

Controllers never call each other. Each keeps an informer cache fed by a watch, puts the key of anything that changed on a work queue, and reconciles: compare what the object asks for with what exists, then write the difference back through the API server. The Deployment controller writes ReplicaSets, the ReplicaSet controller writes Pods, the EndpointSlice controller writes the list of Ready pod addresses that kube-proxy turns into rules. The scheduler is the same shape with a cycle per Pending pod: filter out nodes that cannot take it (room, GPUs, cordons and taints, and for a PodGroup the gang's admission), score the rest (spread), bind by writing spec.nodeName. The kubelet on that node sees the binding, asks the container runtime to pull and start, runs the probes, and writes the status back; nothing in the control plane ever starts a container itself.

kubectl get --raw /healthz?verbose            # the API server's own checks
kubectl -v=8 scale deploy/web --replicas=5    # the request, the response, its resourceVersion
kubectl get leases -n kube-system             # controller-manager and scheduler leader election
kubectl get events --field-selector reason=Scheduled
ETCDCTL_API=3 etcdctl endpoint status --cluster -w table   # leader, term, raft index per member
ETCDCTL_API=3 etcdctl member list

Try: scale to 8, then stop the etcd leader while the ReplicaSet is still creating pods, and watch the writes queue at the API server until the new leader is elected.

kubectl apply: wanting something

You never tell Kubernetes to start a pod. You write down what should exist (a Deployment with two replicas of redis:7) and apply it. The API server stores it; from there it is controllers all the way down: Deployment to ReplicaSet to two Pending Pods, scheduler binds each to a node, that node's kubelet pulls the image and starts it.

This is the reason apply is idempotent: applying the same file twice changes nothing, because the world already matches.

# cache.yaml
apiVersion: apps/v1
kind: Deployment
metadata: { name: cache }
spec:
  replicas: 2
  selector: { matchLabels: { app: cache } }
  template:
    metadata: { labels: { app: cache } }
    spec:
      containers: [{ name: redis, image: redis:7 }]

kubectl apply -f cache.yaml
kubectl rollout status deploy/cache
kubectl delete -f cache.yaml        # and the pods go with it

Scale: change the number on the board

Scaling only edits spec.replicas. The ReplicaSet controller compares that number with the live pods it owns and creates or deletes the difference. Which pods go first when scaling down is a rule, not chance: unscheduled ones, then not-ready ones, then the youngest.

A HorizontalPodAutoscaler is just something that edits the same number on your behalf, based on CPU or a custom metric.

kubectl scale deploy/web --replicas=8
kubectl get pods -l app=web -o wide --watch
kubectl autoscale deploy/web --min=3 --max=10 --cpu-percent=70

Try: scale to 8, then Drain a node and watch where the evicted pods land.

Rolling update: one in, one out

Changing the pod template (a new image tag) makes the Deployment create a second ReplicaSet and shift replicas from old to new under two limits: maxSurge (how many extra pods may exist, here 1) and maxUnavailable (how many may be missing, here 0). The old ReplicaSet is kept at zero so a rollback is just the same dance in reverse.

kubectl set image deploy/web nginx=nginx:1.28
kubectl rollout status deploy/web
kubectl rollout history deploy/web
kubectl rollout undo deploy/web      # back to the previous ReplicaSet

Try: Roll out ▸ web back to v1 to watch the undo.

Self-healing: the count is the contract

Pods are cattle: a deleted pod is gone, with its IP and its name. What persists is the ReplicaSet's promise of three. This is why you almost never create bare pods, and why anything that must find pods (a Service, a DNS name) looks them up by label rather than by name.

kubectl delete pod web-6b8fd-xk2p9
kubectl get pods -l app=web          # a new name appears
kubectl get events | grep -i replicaset

Crash versus not-ready: two different lamps

Probes also chain. In the demo api's readiness check talks to db and web's talks to api, so Break ▸ delete the db pod turns the api lamps amber, then the web lamps, and requests get a 503 until the StatefulSet brings db back and the lamps go green again in order. Nothing restarted: readiness just told the truth about who could serve.

A liveness probe failing means "restart this container". A readiness probe failing means "do not send me traffic", and the Service removes the pod from its endpoints until it passes. Mixing them up is the classic mistake: a liveness probe on a slow dependency turns a brownout into a restart storm.

readinessProbe: { httpGet: { path: /healthz, port: 8080 }, periodSeconds: 5 }
livenessProbe:  { httpGet: { path: /livez,   port: 8080 }, failureThreshold: 3 }

kubectl get pods                      # READY 0/1 but STATUS Running: readiness
kubectl get pods                      # RESTARTS climbing: liveness or a crash

GPUs: resources are real things on real slabs

A pod asks for resources (requests) and the scheduler only binds it where they fit: CPU and memory on every node, and extended resources such as nvidia.com/gpu only where a device plugin has advertised them. GPUs cannot be shared or overcommitted the way CPU can, so a GPU request is all-or-nothing per device. This is why "Pending" is usually a capacity question, not a bug.

resources:
  limits: { nvidia.com/gpu: 1 }          # requests == limits for extended resources

kubectl describe node node-c | grep -A3 Allocatable
kubectl get pods -l app=train            # 2 Running, 1 Pending
kubectl describe pod train-...           # 0/3 nodes are available: 3 Insufficient nvidia.com/gpu

Gang scheduling: all of us or none of us

The default scheduler places pods one at a time, which is wrong for a distributed training or Spark job whose workers wait on each other: three workers placed and one Pending waste three slots forever. A gang scheduler such as Volcano (or the scheduler plugins' coscheduling) adds a PodGroup with minMember: the group is admitted only when every member can be bound, then all are bound in one go.

apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata: { name: spark }
spec: { minMember: 4 }
# in the Job's pod template:
schedulerName: volcano
annotations: { scheduling.k8s.io/group-name: spark }

kubectl get podgroup spark        # Pending -> Running once all 4 fit

Drain a node: tape it off and move everything

Drain is two steps: cordon (mark the node unschedulable) then evict every pod on it, respecting PodDisruptionBudgets so you cannot take out the last replica of something. It is how you patch a kernel or resize a node pool without an outage. DaemonSet pods are ignored because they belong on every node.

kubectl cordon node-a
kubectl drain node-a --ignore-daemonsets --delete-emptydir-data
# ... reboot / replace the machine ...
kubectl uncordon node-a

Services and Ingress: how a request finds a pod

Pods come and go, so nothing talks to a pod's IP. A Service is a stable virtual IP and DNS name (web.default.svc) with a label selector; its endpoints are the Ready pods that match, kept current by a controller and wired into every node by kube-proxy. An Ingress (or Gateway) is the HTTP front door: hostnames and paths routed to Services, usually terminating TLS.

kubectl get svc web                   # ClusterIP 10.96.0.12  80/TCP
kubectl get endpointslices -l kubernetes.io/service-name=web
kubectl get ingress                   # shop.example.com -> web:80
kubectl port-forward svc/web 8080:80  # try it from your laptop

Point it at a real cluster

One snapshot, no server

kubectl get nodes,pods,deploy,sts,ds,svc,ingress -A -o json > cluster.json
# open figure/longshore3d.html, Cluster ▸ "open a kubectl JSON file"

Live, refreshed every few seconds

python3 tools/longshore.py serve --every 5      # opens localhost:8788
python3 tools/longshore.py serve -n shop --context prod
# in another terminal:
kubectl scale deploy/web --replicas=5          # and watch the crane

The helper only ever runs kubectl get. Each node gets a slab with eighteen slots per three tiers and stacks higher if it must; more than three nodes adds rows of slabs behind. The app colour comes from the app or app.kubernetes.io/name label, or the ReplicaSet name. The newest version of an app wears the pale band.