Problem statement
Namespace wedge-watch runs two web services, checkout-api and orders-api. About twelve seconds after it starts, checkout-api deadlocks: the listener stays open, the process stays up, the pod stays Running - and /healthz stops answering. Nothing notices, so it sits like that for hours. A container restart clears the fault, but nobody is there to press the button. orders-api has never faulted and must not be disturbed.
About this exercise
A liveness probe is the kubelet's repeated question to a container: are you still working? When the answer stops coming, the kubelet kills the container and the pod's restart policy brings it back. That is the whole point - it is not a monitor that pages someone, it is the thing that presses the button.
Without a probe the kubelet only knows whether the process is alive. A deadlocked process is alive. So is one holding an open socket it will never read from again. This is the gap the probe closes, and the handler you pick decides how wide the gap stays:
- tcpSocket - the port accepts a connection. A wedged process whose listener is still bound passes this happily.
- httpGet - a GET against a path and port; any 2xx or 3xx is a pass, anything else is a failure. The request has to travel through the code that is actually stuck, which is why it sees what the socket check cannot.
- exec - a command run inside the container; exit status 0 is a pass.
Timing is the rest of the design, and it is a trade. Every probe carries initialDelaySeconds (how long to wait before the first check), periodSeconds (how often to check), timeoutSeconds (how long one check may take), failureThreshold (how many consecutive failures kill the container) and successThreshold. Slack in these fields is time a wedged service keeps taking traffic; too little of it restarts healthy containers over a hiccup. Detection time is roughly periodSeconds x failureThreshold after the fault.
Probes are part of a container's spec, and a pod's containers cannot be rewritten in place - a pod is replaced, not edited. kubectl explain pod.spec.containers.livenessProbe lists every field.
How this project is graded
The grader reads each probe back off the live pod and compares every field against the specification below. Then it watches the cluster: it waits for the wedge to be caught, reads the restart count and the events the kubelet recorded, checks from inside the container that serving resumed, and confirms the healthy service ran through many probe periods without a single restart. Both pods are re-seeded without probes if they are missing, so a half-finished replacement is graded as unprobed.
What you will build
Both pods in wedge-watch keep the containers, images, commands and volume they have now, and each gains one liveness probe. manifest.yaml holds them as they run today.
handler HTTP GET on path /healthz, port 80 initial delay 3 seconds after the container starts period one check every 3 seconds timeout 1 second for a single check failure threshold restart after 2 consecutive failures
The same probe, with the same numbers, on checkout-api and on orders-api.
Step 1 - a check that sees more than an open port
Give both containers the probe described above. The commands, images and the state volume stay exactly as they are.
liveness_probe_declared_with_the_required_check and liveness_thresholds_match_the_required_timing should pass
Step 2 - the wedge gets caught
With the probe in place the deadlock has a deadline. Watch checkout-api: its restart count must move on its own, the kubelet must record why, and the container that comes back must be serving /healthz again.
wedged_container_is_restarted_by_the_probe, probe_failure_and_kill_are_recorded_in_events and restart_clears_the_wedge_and_serving_resumes should pass
Step 3 - the healthy service is left alone
orders-api carries the same probe and must run through it untouched - a probe that restarts a working container is worse than no probe at all.
healthy_workload_is_not_restarted_by_the_same_probe should pass
When every test passes
You have:
- A health check that asks the application, not the socket, whether it is working
- A deadlock that ends in seconds instead of hours, with no human involved
- The kubelet's own record of what failed and why the container was killed
- Thresholds tight enough to catch the fault and loose enough to leave a healthy service alone
You do not have yet:
- Any control over traffic. A liveness probe restarts a container; it never takes a still-starting one out of rotation. That is the next exercise.
Verify
Click Test after every step.
