StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 12 · Services (ClusterIP & NodePort)

Endpoints & Readiness Gating

mediumReadiness GatingService Endpoints—No data yet

Problem statement

Namespace deploy-gate runs checkout-api, two replicas behind a ClusterIP Service that already selects them. The container binds its port the instant it starts, but for the first ten seconds it answers every request with a cold-start page; then it serves properly for twelve seconds, drops its upstream - reporting that it can no longer serve, and falling back to that same cold-start page a few seconds later - and recovers on its own about twenty seconds after that. All of that time it is a backend of the Service, so users are routed to it throughout - during the warm-up and during the outage. Make membership of the Service track what the instance can actually serve.

About this exercise

The endpoint list behind a Service is not just "the pods the selector matched". It is the pods the selector matched and that are ready. Every pod carries a Ready condition, and only ready pods are published as endpoints; the rest are held back, in the endpoint records, as not-ready addresses.

Without a readiness probe the container's ready flag is set as soon as the container is running - which is a statement about the process, not about the application. A process that has bound its port and is still loading is running and useless, and the endpoint list cannot tell the difference.

A readiness probe gives the application the vote. It runs on a schedule with the usual handlers (httpGet, tcpSocket, exec), and its result drives the container's ready flag and nothing else:

  • Fail while the pod is starting, and the pod never enters the endpoint list - traffic waits instead of arriving early.
  • Fail mid-life, and the address is withdrawn from the endpoint list. The container is not restarted: it keeps running, keeps its restart count, keeps whatever it had warmed.
  • Pass again, and the address is published again, with no interruption to the process.

That is the whole mechanism behind a safe deploy: the Service stops sending work to an instance that says it cannot do it, and resumes when it says it can. initialDelaySeconds and periodSeconds decide how fast a warmed instance joins; failureThreshold decides how many bad checks it takes to withdraw one. kubectl explain deployment.spec.template.spec.containers.readinessProbe lists every field.

A Deployment's pod template can be changed in place - the Deployment rolls the pods for you.

Your shell here sits on the cluster's own network, so the Service's cluster IP answers from it exactly as it would from a pod. A Service with no ready endpoints does not queue anything: the connection is refused immediately, which is the correct behaviour and looks nothing like a cold-start error page.

How this project is graded

The grader reads your probe back off the live Deployment and compares every field against the specification below. Then, to see the whole cycle from the first second, it rebuilds the same Deployment carrying your probe - plus a copy of the Service - in a scratch namespace of its own, and watches it from the first second: it records how many ready endpoints that Service has and what its address answers, and reads the cluster's own timestamps for when each container started and when it went ready. It checks the sequence and the durations: no endpoints through the warm-up, both pods published once they are good, the addresses withdrawn again for the whole outage when the upstream drops, published once more on recovery, with the restart count still zero and never a single cold-start page in any answered request. Nothing in deploy-gate is created, patched or deleted by the grader, and the scratch namespace is removed afterwards.

What you will build

checkout-api in deploy-gate keeps the container, image, port and command it has now - deployment.yaml holds it as it runs today - and its pod template gains one readiness probe:

handler             HTTP GET on path /ready, port 8080
initial delay       2 seconds after the container starts
period              one check every 2 seconds
timeout             1 second for a single check
failure threshold   2 consecutive failures withdraw the pod

That is the only probe on the container. It carries no liveness probe and no startup probe.

Step 1 - the application gets a vote

Give the container the probe described above, and nothing else.

readiness_probe_gates_the_checkout_pods should pass

Step 2 - nobody arrives early

From the moment the containers start until they can serve, the Service must publish no endpoints at all.

endpoints_stay_empty_while_the_pods_warm_up should pass

Step 3 - out of rotation, still alive

When the upstream drops, the addresses must leave the endpoint list, and the containers must keep running - restart counts stay at zero. When it comes back, the addresses return.

a_readiness_failure_drops_the_pod_from_the_endpoints, the_dropped_pod_keeps_running_without_a_restart and recovery_readmits_the_pods_to_the_endpoints should pass

Step 4 - no cold pages, ever

Across the whole cycle, every request the Service answers must come from an instance that was ready. Requests that are refused because there is nothing ready to serve them are correct; a cold-start page is not.

every_answered_probe_came_from_a_ready_backend should pass

When every test passes

You have:

  • A Service whose backend list is the set of instances that can actually serve
  • Deploys where traffic waits for warm-up instead of landing on it
  • A mid-life failure that costs rotation and nothing else - no restart, no lost state
  • Automatic re-admission the moment the instance recovers

You do not have yet:

  • Any way in for traffic from outside the cluster. A cluster IP is reachable on the cluster network and nowhere else. That is the next exercise.

Verify

Click Test after every step.

More in Services (ClusterIP & NodePort)

See the module
ClusterIP BasicsGive a churning web tier one stable virtual address with a ClusterIP Service - selector, port and targetPort - and prove the endpoints track the ready pods while the address does not move.—No data yetmediumNodePort & Named PortsOpen a storefront to the outside with NodePort Services - one pinned, one allocated, one rejected out of range - and decouple the Service from the container port with a named target port that survives a renumber.—No data yetmediumStorefront Traffic LayerBuild a storefront's whole traffic layer - a stable internal address, endpoints gated on readiness, a node port for the outside world and both Services written against a named port that survives a renumber - and audit the layer of any namespace.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.