Problem statement
Namespace probe-fleet runs four workloads and not one of them tells the truth about itself. session-gateway serves for twelve seconds and then deadlocks with its listener still open, and only a restart clears it. catalog-cache reports itself up the instant it starts, ten seconds before its cache is warm, and drops out again while an upstream refresh runs. ledger-legacy needs twenty-five seconds to boot and gets killed by anything tight enough to catch the gateway. archive-worker claims to be another slow booter and takes ninety seconds - which is not a boot, it is a failure. Make the fleet self-report and self-heal, and then produce the audit that proves it.
About this exercise
This brings together the topic's three exercises:
- Liveness - a check that restarts a container the application itself can no longer answer for (Liveness Probes)
- Readiness - a check that takes an instance out of rotation without restarting it, and puts it back (Readiness Probes)
- Startup - a boot budget that holds the other two off until the process is actually up (Startup Probes)
One idea ties them together: each probe answers a different question, and using the wrong one is worse than using none. Ask "is it working?" of a container that is still warming and you restart it forever. Ask "can it serve?" of a deadlocked process and you quietly route around it while it never recovers. Ask either during a ninety-second boot with no budget and you never find out which of the two you had.
The three questions, and the single answer each drives: liveness -> restart the container. readiness -> take it out of rotation, leave it running. startup -> hold the other two off until this succeeds, and kill the container if it never does. kubectl explain pod.spec.containers lists the fields of all three. Probes are part of a container's spec, and a pod's containers cannot be rewritten in place - a pod is replaced, not edited.
How this project is graded
The grader reads every probe back off every live pod and compares it against the specification below. Then it watches the fleet run its drill: it waits for the gateway's wedge to be caught and reads how long the killed container was actually allowed to live, checks the events the kubelet recorded for each probe kind, measures how long the slow boot took and confirms nothing restarted it, and waits for the over-budget worker to be killed with the startup probe named. To see the cache's whole life cycle from the first second, it rebuilds that container with your readiness probe in a scratch namespace of its own and samples the pod every two seconds. Finally it runs your audit twice: once against probe-fleet, where every line must match what the cluster reports at that moment, and once against a second scratch namespace holding the same four containers with no probes at all. Both scratch namespaces are removed afterwards, and nothing in probe-fleet is created, patched or deleted by the grader.
What you will build
All four pods keep the containers, images, commands and volume they have now - manifest.yaml holds them as they run today - and each gains exactly the probes below and no others. Every handler is an HTTP GET on port 80.
session-gateway liveness path /healthz, initial delay 3s, period 3s,
timeout 1s, failure threshold 2
and no readiness probe and no startup probe
catalog-cache readiness path /ready, initial delay 2s, period 2s,
timeout 1s, failure threshold 2
and no liveness probe and no startup probe
ledger-legacy startup path /healthz, period 3s, timeout 1s,
failure threshold 15 (a 45 second boot budget)
liveness path /healthz, period 3s, timeout 1s,
failure threshold 2, and no initial delay
and no readiness probe
archive-worker the same two probes as ledger-legacy, with the same numbers
And probe-audit.sh, a read-only scan that takes an optional namespace argument and defaults to probe-fleet:
probe-audit.sh [namespace]
It prints one line per pod, sorted by pod name, and then one count:
AUDIT <pod> <ready> <restarts> <probes> FAILURES <n>ready the pod's Ready condition, lowercase: true or false restarts the restart count of the pod's container probes the probe kinds declared on that container, comma-separated and always in the order liveness,readiness,startup - or a single - when the container carries no probe at all n how many pods in the namespace the cluster has recorded at least one probe failure event for (events with reason Unhealthy)
So a namespace of unprobed containers audits as four - lines and FAILURES 0.
The tests
- Liveness_wedged_gateway_is_restarted_on_the_tight_timing - the killed container's own lifetime proves the timing
- Liveness_probe_failures_are_recorded_in_the_events - the kubelet names the liveness probe as the cause
- Readiness_warmup_gates_the_ready_condition_at_start - not ready until the cache is warm, then ready
- Readiness_midlife_cycle_unreadies_without_restarting - out of rotation and back, restart count untouched
- Startup_slow_boot_survives_with_liveness_deferred - twenty-five seconds of boot, zero restarts, no liveness failure on record
- Startup_over_budget_boot_is_restarted - ninety seconds does not fit in the budget, and the kubelet says so
- Composition_probe_parameters_match_the_fleet_spec - every probe on every pod, and none that should not be there
- Composition_audit_reports_conditions_restarts_and_probes - the audit matches the live cluster line for line
- Composition_audit_run_against_an_unprobed_copy - the same audit, run on a namespace the grader builds, reports what is really there
When every test passes
You have:
- A deadlock that ends in seconds, on a probe tight enough to be worth having
- A cold cache that stays out of rotation until it can serve, and rejoins without losing what it warmed
- A slow boot with a budget instead of a grace period, and a tight liveness probe waiting behind it
- A workload that cannot boot failing loudly instead of hiding among the slow ones
- One command that reports the condition, restart count and declared probes of a whole namespace, and works on a namespace it has never seen
- The next topic unlocked
Verify
Click Test.
