StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 10 · Health Probes

Liveness Probes

mediumLiveness ProbesProbe Timing & Thresholds—No data yet

Problem statement

Namespace wedge-watch runs two web services, checkout-api and orders-api. About twelve seconds after it starts, checkout-api deadlocks: the listener stays open, the process stays up, the pod stays Running - and /healthz stops answering. Nothing notices, so it sits like that for hours. A container restart clears the fault, but nobody is there to press the button. orders-api has never faulted and must not be disturbed.

About this exercise

A liveness probe is the kubelet's repeated question to a container: are you still working? When the answer stops coming, the kubelet kills the container and the pod's restart policy brings it back. That is the whole point - it is not a monitor that pages someone, it is the thing that presses the button.

Without a probe the kubelet only knows whether the process is alive. A deadlocked process is alive. So is one holding an open socket it will never read from again. This is the gap the probe closes, and the handler you pick decides how wide the gap stays:

  • tcpSocket - the port accepts a connection. A wedged process whose listener is still bound passes this happily.
  • httpGet - a GET against a path and port; any 2xx or 3xx is a pass, anything else is a failure. The request has to travel through the code that is actually stuck, which is why it sees what the socket check cannot.
  • exec - a command run inside the container; exit status 0 is a pass.

Timing is the rest of the design, and it is a trade. Every probe carries initialDelaySeconds (how long to wait before the first check), periodSeconds (how often to check), timeoutSeconds (how long one check may take), failureThreshold (how many consecutive failures kill the container) and successThreshold. Slack in these fields is time a wedged service keeps taking traffic; too little of it restarts healthy containers over a hiccup. Detection time is roughly periodSeconds x failureThreshold after the fault.

Probes are part of a container's spec, and a pod's containers cannot be rewritten in place - a pod is replaced, not edited. kubectl explain pod.spec.containers.livenessProbe lists every field.

How this project is graded

The grader reads each probe back off the live pod and compares every field against the specification below. Then it watches the cluster: it waits for the wedge to be caught, reads the restart count and the events the kubelet recorded, checks from inside the container that serving resumed, and confirms the healthy service ran through many probe periods without a single restart. Both pods are re-seeded without probes if they are missing, so a half-finished replacement is graded as unprobed.

What you will build

Both pods in wedge-watch keep the containers, images, commands and volume they have now, and each gains one liveness probe. manifest.yaml holds them as they run today.

handler             HTTP GET on path /healthz, port 80
initial delay       3 seconds after the container starts
period              one check every 3 seconds
timeout             1 second for a single check
failure threshold   restart after 2 consecutive failures

The same probe, with the same numbers, on checkout-api and on orders-api.

Step 1 - a check that sees more than an open port

Give both containers the probe described above. The commands, images and the state volume stay exactly as they are.

liveness_probe_declared_with_the_required_check and liveness_thresholds_match_the_required_timing should pass

Step 2 - the wedge gets caught

With the probe in place the deadlock has a deadline. Watch checkout-api: its restart count must move on its own, the kubelet must record why, and the container that comes back must be serving /healthz again.

wedged_container_is_restarted_by_the_probe, probe_failure_and_kill_are_recorded_in_events and restart_clears_the_wedge_and_serving_resumes should pass

Step 3 - the healthy service is left alone

orders-api carries the same probe and must run through it untouched - a probe that restarts a working container is worse than no probe at all.

healthy_workload_is_not_restarted_by_the_same_probe should pass

When every test passes

You have:

  • A health check that asks the application, not the socket, whether it is working
  • A deadlock that ends in seconds instead of hours, with no human involved
  • The kubelet's own record of what failed and why the container was killed
  • Thresholds tight enough to catch the fault and loose enough to leave a healthy service alone

You do not have yet:

  • Any control over traffic. A liveness probe restarts a container; it never takes a still-starting one out of rotation. That is the next exercise.

Verify

Click Test after every step.

More in Health Probes

See the module
Readiness ProbesGate a cold cache out of rotation with a readiness probe, and prove the Ready condition tracks the warm-up at startup, through a mid-life failure and on recovery without a single restart.—No data yetmediumStartup ProbesProtect a slow boot with a startup probe carrying its own budget so a tight liveness probe can take over afterwards, and prove both the over-budget kill and the post-boot wedge land on their own timing.—No data yetmediumProbed Service FleetProbe a fleet of four misbehaving services so each self-reports and self-heals honestly - tight liveness for the wedger, warm-up readiness for the cache, a startup budget in front of liveness for the slow booters - and audit conditions, restarts and probe failures across the namespace.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.