Problem statement
Namespace deploy-gate runs checkout-api, two replicas behind a ClusterIP Service that already selects them. The container binds its port the instant it starts, but for the first ten seconds it answers every request with a cold-start page; then it serves properly for twelve seconds, drops its upstream - reporting that it can no longer serve, and falling back to that same cold-start page a few seconds later - and recovers on its own about twenty seconds after that. All of that time it is a backend of the Service, so users are routed to it throughout - during the warm-up and during the outage. Make membership of the Service track what the instance can actually serve.
About this exercise
The endpoint list behind a Service is not just "the pods the selector matched". It is the pods the selector matched and that are ready. Every pod carries a Ready condition, and only ready pods are published as endpoints; the rest are held back, in the endpoint records, as not-ready addresses.
Without a readiness probe the container's ready flag is set as soon as the container is running - which is a statement about the process, not about the application. A process that has bound its port and is still loading is running and useless, and the endpoint list cannot tell the difference.
A readiness probe gives the application the vote. It runs on a schedule with the usual handlers (httpGet, tcpSocket, exec), and its result drives the container's ready flag and nothing else:
- Fail while the pod is starting, and the pod never enters the endpoint list - traffic waits instead of arriving early.
- Fail mid-life, and the address is withdrawn from the endpoint list. The container is not restarted: it keeps running, keeps its restart count, keeps whatever it had warmed.
- Pass again, and the address is published again, with no interruption to the process.
That is the whole mechanism behind a safe deploy: the Service stops sending work to an instance that says it cannot do it, and resumes when it says it can. initialDelaySeconds and periodSeconds decide how fast a warmed instance joins; failureThreshold decides how many bad checks it takes to withdraw one. kubectl explain deployment.spec.template.spec.containers.readinessProbe lists every field.
A Deployment's pod template can be changed in place - the Deployment rolls the pods for you.
Your shell here sits on the cluster's own network, so the Service's cluster IP answers from it exactly as it would from a pod. A Service with no ready endpoints does not queue anything: the connection is refused immediately, which is the correct behaviour and looks nothing like a cold-start error page.
How this project is graded
The grader reads your probe back off the live Deployment and compares every field against the specification below. Then, to see the whole cycle from the first second, it rebuilds the same Deployment carrying your probe - plus a copy of the Service - in a scratch namespace of its own, and watches it from the first second: it records how many ready endpoints that Service has and what its address answers, and reads the cluster's own timestamps for when each container started and when it went ready. It checks the sequence and the durations: no endpoints through the warm-up, both pods published once they are good, the addresses withdrawn again for the whole outage when the upstream drops, published once more on recovery, with the restart count still zero and never a single cold-start page in any answered request. Nothing in deploy-gate is created, patched or deleted by the grader, and the scratch namespace is removed afterwards.
What you will build
checkout-api in deploy-gate keeps the container, image, port and command it has now - deployment.yaml holds it as it runs today - and its pod template gains one readiness probe:
handler HTTP GET on path /ready, port 8080 initial delay 2 seconds after the container starts period one check every 2 seconds timeout 1 second for a single check failure threshold 2 consecutive failures withdraw the pod
That is the only probe on the container. It carries no liveness probe and no startup probe.
Step 1 - the application gets a vote
Give the container the probe described above, and nothing else.
readiness_probe_gates_the_checkout_pods should pass
Step 2 - nobody arrives early
From the moment the containers start until they can serve, the Service must publish no endpoints at all.
endpoints_stay_empty_while_the_pods_warm_up should pass
Step 3 - out of rotation, still alive
When the upstream drops, the addresses must leave the endpoint list, and the containers must keep running - restart counts stay at zero. When it comes back, the addresses return.
a_readiness_failure_drops_the_pod_from_the_endpoints, the_dropped_pod_keeps_running_without_a_restart and recovery_readmits_the_pods_to_the_endpoints should pass
Step 4 - no cold pages, ever
Across the whole cycle, every request the Service answers must come from an instance that was ready. Requests that are refused because there is nothing ready to serve them are correct; a cold-start page is not.
every_answered_probe_came_from_a_ready_backend should pass
When every test passes
You have:
- A Service whose backend list is the set of instances that can actually serve
- Deploys where traffic waits for warm-up instead of landing on it
- A mid-life failure that costs rotation and nothing else - no restart, no lost state
- Automatic re-admission the moment the instance recovers
You do not have yet:
- Any way in for traffic from outside the cluster. A cluster IP is reachable on the cluster network and nowhere else. That is the next exercise.
Verify
Click Test after every step.
