Problem statement
The chaos drill is on Thursday. Somebody will delete pods without warning, somebody will ask for the tier to be twice its size and then a quarter of it, and somebody will try to push a manifest whose selector points at the cache tier sharing the namespace. The web tier has to survive all of it: hold its declared size, heal what it loses, resize on command, and never claim a pod that is not its own. Namespace web-tier already runs cache-store, the neighbour you must not disturb. Build the tier, run the drill, and leave an audit behind that proves what happened.
About this exercise
This brings together the topic's three exercises:
- Declared size - a Deployment's replica count and pod template, the ReplicaSet it generates, and the owner references that make that ReplicaSet responsible for those pods (Declarative Replicas)
- Scaling - changing the count on the live object and in the manifest, and reading desired, current, ready and available while they disagree (Scaling)
- The label contract - a selector that must cover its own template, and what happens to a pod labelled out from under its owner (Selector-Template Contract)
One idea ties them together: the tier's size is a fact the cluster enforces, and membership in it is decided by labels, continuously. Everything in the drill is a consequence. Pods vanish and the count comes back because a controller is comparing, not because anyone acted. A resize is one number and the same ReplicaSet keeps serving, because the template did not change. A selector that does not cover its own template is refused before it can adopt anything, and a pod relabelled out of the selector is released - still running, owned by nobody, counted by nothing.
That last case is why the audit exists. A tier that reports four ready pods can still be wrong: there can be a fifth pod beside it that nothing manages, and the only way to see it is to read ownership off the objects rather than trusting the count.
How this project is graded
The grader reads the live Deployment, its ReplicaSet and the owner reference on every pod, re-runs your rejected manifest against the API server in a way that stores nothing, and compares your saved answers with what the cluster actually assigned. It then runs your audit twice: once against the namespace as you left it, and once after it has put a pod of its own into the namespace, which the second report must find. That grader-owned pod is created and removed by the grader; nothing else in the namespace is touched. The namespace is re-seeded before each run, so the cache tier comes back if you remove it.
What you will build
Everything lives in namespace web-tier.
1. Deployment web-frontend, described in manifest.yaml:
replicas 4 selector app=web, tier=frontend pod labels app=web, tier=frontend, role=edge container name web, image nginx:alpine
2. broken.yaml - a Deployment named web-frontend-broken that must never reach the cluster:
replicas 2 selector app=cache, tier=store the cache tier's labels pod labels app=web, tier=frontend container name web, image nginx:alpine
3. The drill, run in this order:
build the tier reaches its declared size of 4
heal two tier pods are deleted; the count returns to 4
contract broken.yaml is applied and refused; then one tier pod has its
tier label changed to quarantine and is left running
scale-up the live tier is taken to 7
scale-down the live tier is taken to 2
restore the count declared in manifest.yaml is applied, returning it to 4
4. Six answers in answers/:
killed.txt the two pod names you deleted in the heal phase, one per line
rejection.txt what the API server said when broken.yaml was applied
orphan.txt the name of the pod you relabelled, exactly that name
pods-at-7.txt the seven pod names once the tier has settled at seven, one per line
pods-at-2.txt the two pod names once it has settled at two, one per line
timeline.txt one line per phase above, in that order:
<phase> <desired> <replicas> <ready> <available>
Each timeline.txt line is read once that phase has converged, using the phase names exactly as spelled above.
5. tier-audit.sh - a read-only report on namespace web-tier, two sections, each introduced by its name on a line of its own:
OWNERSHIP
<pod> <replicaset> one line per pod in the namespace, sorted by pod name;
a pod no ReplicaSet controls prints - as its second field
STATUS
<deployment> <desired> <replicas> <ready> <available>
one line per Deployment in the namespace, sorted by name
The audit reports the namespace, not the tier: the cache tier's pods and Deployment belong in it too, and so does anything else that turns up.
The tests
- Replicas_declared_size_is_ready_and_available - the tier ends at four, reported by the Deployment itself
- Replicas_pods_are_owned_by_the_tier_replicaset - one ReplicaSet, owned by the Deployment, owning all four pods
- Healing_killed_pods_are_gone_and_the_count_restored - the deleted pods are gone and replacements took their place
- Scaling_up_and_down_converge_at_each_target - seven, then two, then four from the manifest
- Scaling_surplus_removal_keeps_the_survivors - shrinking removed the surplus and left the rest running
- Scaling_status_timeline_tracks_every_phase - all four counters recorded for every phase, and the last still true
- Contract_mismatched_selector_is_rejected - the selector aimed at the cache tier never became an object
- Contract_relabelled_pod_is_orphaned_and_replaced - the relabelled pod was released and a replacement built
- Contract_orphan_keeps_running_unmanaged - the orphan is still Running and owned by nothing
- Composition_audit_matches_live_ownership_and_status - the report agrees with the live cluster
- Composition_audit_reads_the_cluster_on_every_run - a pod that appears later shows up in the next report
When every test passes
You have:
- A web tier whose size is enforced by the cluster and restored without anybody being paged
- Both resize paths exercised on one ReplicaSet, with the four status counters recorded at every phase
- A mismatched selector refused before it could adopt the neighbour's pods
- An orphan produced deliberately, still running, and visible as unowned in your own audit
- One report that answers "who owns what, and is the tier the size it claims" for the whole namespace
- The next topic unlocked
Verify
Click Test.
