StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 10 · Health Probes

Startup Probes

mediumStartup ProbesBoot Budgets & Liveness Handover—No data yet

Problem statement

Namespace slow-start runs three workloads. ledger-legacy needs about twenty-five seconds to come up before /healthz answers - that is simply how long it takes. pricing-api is up in a moment and then deadlocks ten seconds later, listener still open, and only a restart clears it. archive-worker claims it is just slow too, and takes ninety seconds. Any liveness probe tight enough for pricing-api kills ledger-legacy mid-boot, over and over; any liveness probe loose enough to let ledger-legacy boot lets pricing-api sit wedged for minutes.

About this exercise

That trade is real, and a liveness probe cannot resolve it - it has exactly one schedule, and it has to serve both the slowest boot and the fastest fault. initialDelaySeconds is the usual attempt at a fix, and it is a bad one: it is a fixed grace period that also applies after every restart, and it has to be sized for the worst boot you have ever seen.

A startup probe splits the problem in two. While it is defined and has not yet succeeded:

  • the liveness and readiness probes are not run at all - the container cannot be killed for being unhealthy, because nothing is asking
  • the container is not marked ready
  • the startup probe runs on its own schedule, with its own failureThreshold

The moment it succeeds it never runs again, and liveness and readiness take over on their schedules. If it never succeeds - if periodSeconds x failureThreshold runs out first - the kubelet kills the container exactly as a liveness failure would, and records that the startup probe was what failed. So the startup probe is a boot budget: this much time to come up, and after that the tight rules apply.

That is what lets the liveness probe be tight. It no longer has to accommodate the boot, so it can be sized for the only thing it is actually for: how long a wedged container is allowed to keep taking traffic. With the boot covered, initialDelaySeconds on liveness stops being needed at all.

kubectl explain pod.spec.containers.startupProbe lists every field, and kubectl describe pod shows the probe lines and the events the kubelet recorded for each one. Probes are part of a container's spec, and a pod's containers cannot be rewritten in place - a pod is replaced, not edited.

How this project is graded

The grader reads both probes back off each live pod and compares every field against the specification below. Then it watches: it measures how long ledger-legacy takes to become ready and confirms it got there without a single restart and without one liveness failure on record; it waits for archive-worker to be killed and checks that the kubelet blamed the startup probe; and it reads how long the wedged incarnation of pricing-api actually lived, which is what proves the liveness timing is tight rather than merely declared. All three pods are re-seeded without probes if they are missing.

What you will build

All three pods in slow-start keep the containers, images, commands and volume they have now - manifest.yaml holds them as they run today - and each gains the same two probes.

startup probe
  handler             HTTP GET on path /healthz, port 80
  period              one check every 3 seconds
  timeout             1 second for a single check
  failure threshold   15 consecutive failures    (a 45 second boot budget)

liveness probe handler HTTP GET on path /healthz, port 80 period one check every 3 seconds timeout 1 second for a single check failure threshold restart after 2 consecutive failures initial delay none - the boot window is the startup probe's job

No readiness probe on any of the three.

Step 1 - a budget for the boot, tight rules after it

Give every container both probes exactly as specified, with no initial delay on the liveness probe.

startup_and_liveness_probes_declared_on_every_workload and probe_budgets_match_the_required_timing should pass

Step 2 - the slow boot survives

ledger-legacy must reach ready on its own schedule, twenty-five seconds in, having never been restarted - and the events must show the startup probe was the one doing the checking while it booted, never the liveness probe.

slow_boot_completes_inside_the_startup_budget and liveness_holds_off_until_startup_succeeds should pass

Step 3 - the boot that never ends, and the wedge that comes after

archive-worker runs out of budget and must be restarted with the startup probe named as the cause. pricing-api boots instantly, wedges ten seconds later, and must be restarted on the tight liveness timing - the lifetime of the killed container is the proof.

over_budget_boot_is_restarted_with_the_startup_failure_recorded and post_startup_wedge_is_caught_on_the_tight_liveness_timing should pass

When every test passes

You have:

  • A slow boot protected by a budget instead of by a guess at initialDelaySeconds
  • A liveness probe sized for faults alone, because it no longer has to survive a boot
  • A workload that cannot come up in its budget failing loudly, with the kubelet naming the startup probe
  • A post-boot deadlock caught in seconds by the same tight probe that would have killed the slow booter without the startup gate

Verify

Click Test after every step.

More in Health Probes

See the module
Liveness ProbesCatch a deadlocked container that keeps its port open by giving it an HTTP liveness probe with tight thresholds, and prove the restart clears the wedge while a healthy service is left alone.—No data yetmediumReadiness ProbesGate a cold cache out of rotation with a readiness probe, and prove the Ready condition tracks the warm-up at startup, through a mid-life failure and on recovery without a single restart.—No data yetmediumProbed Service FleetProbe a fleet of four misbehaving services so each self-reports and self-heals honestly - tight liveness for the wedger, warm-up readiness for the cache, a startup budget in front of liveness for the slow booters - and audit conditions, restarts and probe failures across the namespace.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.