StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 8 · Deployments & ReplicaSets

Self-Healing Web Tier

mediumDeployment OperationsOwnership & Status Auditing—No data yet

Problem statement

The chaos drill is on Thursday. Somebody will delete pods without warning, somebody will ask for the tier to be twice its size and then a quarter of it, and somebody will try to push a manifest whose selector points at the cache tier sharing the namespace. The web tier has to survive all of it: hold its declared size, heal what it loses, resize on command, and never claim a pod that is not its own. Namespace web-tier already runs cache-store, the neighbour you must not disturb. Build the tier, run the drill, and leave an audit behind that proves what happened.

About this exercise

This brings together the topic's three exercises:

  • Declared size - a Deployment's replica count and pod template, the ReplicaSet it generates, and the owner references that make that ReplicaSet responsible for those pods (Declarative Replicas)
  • Scaling - changing the count on the live object and in the manifest, and reading desired, current, ready and available while they disagree (Scaling)
  • The label contract - a selector that must cover its own template, and what happens to a pod labelled out from under its owner (Selector-Template Contract)

One idea ties them together: the tier's size is a fact the cluster enforces, and membership in it is decided by labels, continuously. Everything in the drill is a consequence. Pods vanish and the count comes back because a controller is comparing, not because anyone acted. A resize is one number and the same ReplicaSet keeps serving, because the template did not change. A selector that does not cover its own template is refused before it can adopt anything, and a pod relabelled out of the selector is released - still running, owned by nobody, counted by nothing.

That last case is why the audit exists. A tier that reports four ready pods can still be wrong: there can be a fifth pod beside it that nothing manages, and the only way to see it is to read ownership off the objects rather than trusting the count.

How this project is graded

The grader reads the live Deployment, its ReplicaSet and the owner reference on every pod, re-runs your rejected manifest against the API server in a way that stores nothing, and compares your saved answers with what the cluster actually assigned. It then runs your audit twice: once against the namespace as you left it, and once after it has put a pod of its own into the namespace, which the second report must find. That grader-owned pod is created and removed by the grader; nothing else in the namespace is touched. The namespace is re-seeded before each run, so the cache tier comes back if you remove it.

What you will build

Everything lives in namespace web-tier.

1. Deployment web-frontend, described in manifest.yaml:

replicas      4
selector      app=web, tier=frontend
pod labels    app=web, tier=frontend, role=edge
container     name web, image nginx:alpine

2. broken.yaml - a Deployment named web-frontend-broken that must never reach the cluster:

replicas      2
selector      app=cache, tier=store        the cache tier's labels
pod labels    app=web, tier=frontend
container     name web, image nginx:alpine

3. The drill, run in this order:

build        the tier reaches its declared size of 4
heal         two tier pods are deleted; the count returns to 4
contract     broken.yaml is applied and refused; then one tier pod has its
             tier label changed to quarantine and is left running
scale-up     the live tier is taken to 7
scale-down   the live tier is taken to 2
restore      the count declared in manifest.yaml is applied, returning it to 4

4. Six answers in answers/:

killed.txt      the two pod names you deleted in the heal phase, one per line
rejection.txt   what the API server said when broken.yaml was applied
orphan.txt      the name of the pod you relabelled, exactly that name
pods-at-7.txt   the seven pod names once the tier has settled at seven, one per line
pods-at-2.txt   the two pod names once it has settled at two, one per line
timeline.txt    one line per phase above, in that order:
                <phase> <desired> <replicas> <ready> <available>

Each timeline.txt line is read once that phase has converged, using the phase names exactly as spelled above.

5. tier-audit.sh - a read-only report on namespace web-tier, two sections, each introduced by its name on a line of its own:

OWNERSHIP
<pod> <replicaset>       one line per pod in the namespace, sorted by pod name;
                         a pod no ReplicaSet controls prints - as its second field

STATUS <deployment> <desired> <replicas> <ready> <available> one line per Deployment in the namespace, sorted by name

The audit reports the namespace, not the tier: the cache tier's pods and Deployment belong in it too, and so does anything else that turns up.

The tests

  • Replicas_declared_size_is_ready_and_available - the tier ends at four, reported by the Deployment itself
  • Replicas_pods_are_owned_by_the_tier_replicaset - one ReplicaSet, owned by the Deployment, owning all four pods
  • Healing_killed_pods_are_gone_and_the_count_restored - the deleted pods are gone and replacements took their place
  • Scaling_up_and_down_converge_at_each_target - seven, then two, then four from the manifest
  • Scaling_surplus_removal_keeps_the_survivors - shrinking removed the surplus and left the rest running
  • Scaling_status_timeline_tracks_every_phase - all four counters recorded for every phase, and the last still true
  • Contract_mismatched_selector_is_rejected - the selector aimed at the cache tier never became an object
  • Contract_relabelled_pod_is_orphaned_and_replaced - the relabelled pod was released and a replacement built
  • Contract_orphan_keeps_running_unmanaged - the orphan is still Running and owned by nothing
  • Composition_audit_matches_live_ownership_and_status - the report agrees with the live cluster
  • Composition_audit_reads_the_cluster_on_every_run - a pod that appears later shows up in the next report

When every test passes

You have:

  • A web tier whose size is enforced by the cluster and restored without anybody being paged
  • Both resize paths exercised on one ReplicaSet, with the four status counters recorded at every phase
  • A mismatched selector refused before it could adopt the neighbour's pods
  • An orphan produced deliberately, still running, and visible as unowned in your own audit
  • One report that answers "who owns what, and is the tier the size it claims" for the whole namespace
  • The next topic unlocked

Verify

Click Test.

More in Deployments & ReplicaSets

See the module
Declarative ReplicasReplace a hand-built pod fleet with a Deployment, read the ReplicaSet and owner references it generates, and prove the count heals itself when a pod is deleted.—No data yetmediumScalingGrow and shrink a running Deployment in place and then from its manifest, tracking desired, ready and available counts through every step on one ReplicaSet.—No data yetmediumSelector-Template ContractProve the selector-template contract on a live Deployment: a mismatched selector is refused at admission, and a pod labelled out of the selector is released, replaced and left running unmanaged.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.