Problem statement
Eight workloads landed in fleet-audit over two years and nobody agreed on anything. One is unlabeled entirely, one spells its environment the old way, one is filed under the wrong tier, two still carry a key that was retired last spring. Every operational question - what is in the web tier, what can be cleaned up, what is still unaccounted for - is currently answered by someone reading a list of names off a wiki page. Bring the fleet under management, then prove it by running the operations against the cluster instead of the wiki.
About this exercise
This brings together the topic's three exercises:
- Labels - one scheme carried by every object, applied at creation and by correcting live objects, with the retired key removed (Labeling Scheme)
- Selectors - equality and set-based queries, in, notin and existence, answering questions no name list can keep up with (Selector Queries)
- Annotations - the machine metadata that no selector can see and no label could hold (Annotations for Machinery)
One rule governs the whole thing: the operations may not know any pod's name. A selector describes a set by what its members are, so it keeps answering correctly when membership changes; a name list is a snapshot of yesterday that nothing updates. The grader proves the difference by adding an object of its own to the namespace, re-running your script, and requiring the report to have noticed.
The scheme every object must carry - exactly these three keys, no others:
app the service's own name tier the role it plays one of: web, worker, batch env where it runs one of: prod, staging, dev
squad is retired. It must not survive on any object.
How this project is graded
The grader reads the labels and annotations off the live objects, runs bash fleet.sh and checks every section against the same query run against the cluster. It then runs the script a second time, and finally creates a throwaway object of its own, runs the script again, and requires the report to include it - which only happens if the sections are selector results rather than lists you typed. The grader's object is its own; it removes it afterwards and never touches yours.
What you will build
The migration. Every pod in fleet-audit ends up labeled exactly like this, with no squad key anywhere:
checkout-web app=checkout tier=web env=prod checkout-worker app=checkout tier=worker env=prod ledger-worker app=ledger tier=worker env=prod search-indexer app=search tier=worker env=staging invoice-batch app=invoice tier=batch env=staging report-batch app=report tier=batch env=dev pricing-api app=pricing tier=web env=dev mailer-relay app=mailer tier=worker env=dev
The deploy record, on checkout-web, exactly these keys and values:
deploy.fleet/ticket OPS-4502
deploy.fleet/owner payments-platform
deploy.fleet/build {"commit":"9f2c1ab","pipeline":41,"artifact":"checkout-web-1.8.3"}
fleet.sh, printing one report with six sections, each introduced by its name alone on a line. Every section holds matching pod names, one per line, sorted ascending, and is empty when nothing matches:
WEB the web tier WORKER the worker tier BATCH the batch tier CLEANUP cleanup candidates: everything running in dev NONPROD everything whose env is not prod LEGACY everything still carrying the retired squad key
No pod name may appear anywhere in fleet.sh. Every section is a query the cluster answers.
Step 1 - bring the fleet to the scheme
Fix the unlabeled object, the wrong tier, the old environment spelling and the missing key, and retire squad wherever it survives. When you are done every pod carries exactly app, tier and env.
Labels_every_object_carries_the_scheme and Labels_retired_key_gone_from_the_fleet should pass
Step 2 - the deploy record
Attach the three deploy annotations to checkout-web. The build value is a JSON document and must be stored exactly as written - which is why it is an annotation and not a label.
Annotations_machine_metadata_reads_back_exactly should pass
Step 3 - the tier listings
Write the WEB, WORKER and BATCH sections of fleet.sh. Three sections, three selectors, no names.
Selectors_per_tier_listing_matches_each_tier should pass
Step 4 - the cleanup scope
Write CLEANUP: everything running in the dev environment, the fleet's candidates for deletion. Getting this wrong is how a cleanup job takes production with it.
Selectors_cleanup_candidates_match_the_env_scope should pass
Step 5 - the audits
Write NONPROD and LEGACY. NONPROD is everything whose env is not prod; LEGACY is everything that still carries the squad key at all, whatever its value - and after Step 1 that is nothing, which is the point of running it. Neither is an equality query, and the two use different requirements.
Selectors_audits_use_notin_and_exists_semantics and Composition_report_follows_a_grader_relabel should pass
Step 6 - the report is live
Nothing in the report may be remembered from an earlier run, and nothing you annotated may change any section: the annotations are invisible to every selector in the file.
Annotations_change_no_selector_result and Composition_second_run_reads_the_cluster_again should pass
The tests
- Labels_every_object_carries_the_scheme - every pod carries exactly app, tier and env with the assigned values
- Labels_retired_key_gone_from_the_fleet - the squad key survives on nothing
- Annotations_machine_metadata_reads_back_exactly - the ticket, owner and build JSON read back character for character
- Annotations_change_no_selector_result - a selector for an annotation key matches nothing, and the tier sections are unmoved by them
- Selectors_per_tier_listing_matches_each_tier - WEB, WORKER and BATCH match the same queries run against the cluster
- Selectors_cleanup_candidates_match_the_env_scope - CLEANUP is exactly the dev environment
- Selectors_audits_use_notin_and_exists_semantics - NONPROD excludes prod and nothing else, LEGACY comes back empty
- Composition_second_run_reads_the_cluster_again - a second run produces the same live report
- Composition_report_follows_a_grader_relabel - an object the grader adds appears in the sections it belongs to, which no typed-out list could do
When every test passes
You have:
- A fleet where every object is described the same way, and nothing carries the retired key
- Machine metadata living on the objects it describes, in a form no label could hold
- Every operational question answered by a query rather than a list of names off a wiki
- Proof that the answers follow the cluster: the grader changed the fleet and the report changed with it
Verify
Click Test.
