Problem statement
Namespace batch-suite is one team's whole batch estate, and every piece of it is run the wrong way. The catalog import is six shards and import-shard is one of them, started from a terminal and restarted by the kubelet every time it finishes, because a pod does not believe work can end. feed-doomed reads a column that was dropped in the last migration: it cannot succeed, and on the default budget it will spend the rest of the week proving it. feed-hung is waiting on a partner file that never arrived, so it never fails and never finishes and holds its slot forever. purge-window is a CronJob that fires every minute and must not fire during tomorrow's migration - and must still exist the day after. Nobody can say what any of it did last night. Run the estate properly, and then write the one command that reports on it.
About this exercise
This brings together the topic's three exercises:
- Completions - work run by a controller that counts finished pods and records that they finished (Run-To-Completion Jobs)
- Bounds - a retry budget for work that cannot succeed and a wall-clock deadline for work that will not end (Failure Bounds & Deadlines)
- Width and schedule - several pods in flight at once, and a schedule with policies for concurrency, lateness and history (Parallelism & CronJobs)
One idea ties them together: a batch object's value is the terminal fact it leaves behind. status.succeeded and status.failed are counts the controller wrote; a condition of type Complete or Failed, and the reason on it, is the verdict; and the pods and Jobs still in the namespace are the evidence, kept exactly as long as your history limits and your absent time-to-live say they are. Every one of those can be read hours later by something that was not watching - which is the only kind of answer worth having at six in the morning.
Two facts about changing batch objects. A Job's spec is fixed once it is created: a bound cannot be added to a running Job, it is replaced. A CronJob's suspend is the opposite - a field you can flip on the live object, which is what makes pausing a schedule different from deleting one. kubectl explain job.spec, kubectl explain cronjob.spec and kubectl create job --help are the reference.
How this project is graded
The grader reads every declared field back off the live cluster, then judges what each object actually did. For the import it reconstructs concurrency from the pods themselves - each container's own start and finish times, counted at the busiest moment - and comes back afterwards to confirm the finished Job and its pods are still readable. For the two bounded feeds it counts attempts, reads the terminal condition and its reason, and reads the exit codes and logs of the pods left behind. For the schedules it checks the policy fields, that a due tick produced a Job that finished, how many Jobs each CronJob still owns against the limit declared, and that the paused one has started nothing for longer than its own schedule period. Then it runs your audit twice: once against batch-suite, where every line must match what the cluster reports at that moment, and once against a scratch namespace it builds itself holding two Jobs it chose and no CronJobs at all. The scratch namespace is removed afterwards, and nothing in batch-suite is created, patched or deleted by the grader. Any object you delete is re-seeded in its original unrun form.
What you will build
In namespace batch-suite. manifest.yaml holds the hand-run shard pod, the two unbounded feeds and the purge CronJob exactly as they are now; every container, image, command, exit code, grace period and Never restart policy stays as written there.
Job catalog-import completions 6, parallelism 3, backoff limit 2
an active deadline of 120 seconds
container name shard - the task import-shard runs today,
which prints SHARD IMPORTED
Job feed-doomed backoff limit 1 and no deadline
Job feed-hung backoff limit 1 and an active deadline of 20 seconds
CronJob ledger-tick schedule every minute
concurrency policy Forbid
starting deadline 30 seconds
successful history 1
failed history 1
job template backoff limit 2, pod restart policy Never
container name tick, image busybox:1.36
the task print LEDGER TICKED, then take two seconds
CronJob purge-window paused, and otherwise exactly as it is now
Job purge-window-manual the purge run once, now, from purge-window's own job template
No ttlSecondsAfterFinished on anything: the finished Jobs and their pods are the audit trail, and history limits are what removes them here.
And batch-audit.sh, a read-only scan that takes an optional namespace argument and defaults to batch-suite:
batch-audit.sh [namespace]
It prints one line per Job sorted by name, then one line per CronJob sorted by name, then one count line:
JOB <name> <outcome> <succeeded> <failed> <pods> CRON <name> <paused> <retained> TOTALS <complete> <failed>outcome the Job's terminal condition, lowercase: complete or failed - and active when it carries neither succeeded the succeeded count the controller wrote, 0 when it has written none failed the failed count the controller wrote, 0 when it has written none pods how many pods the Job still owns paused whether the CronJob is suspended, lowercase: true or false retained how many Jobs the CronJob still owns complete how many Jobs in the namespace ended complete failed how many ended failed
So a namespace holding one finished Job and no CronJobs audits as one JOB line, no CRON line, and one TOTALS.
The tests
- Completions_import_meets_its_completions_with_the_declared_width - six shards, three at a time inside its deadline, proven by the pods' own timestamps
- Completions_finished_import_and_its_pods_remain_readable - the Job and all six pods still there after it completed, succeeded and un-restarted, with no time-to-live
- Bounds_doomed_feed_fails_after_exactly_the_bounded_retries - two attempts and a terminal Failed naming the budget
- Bounds_failed_pods_record_their_exit_evidence - both attempts kept, with the task's real exit code and log
- Bounds_hung_feed_dies_at_its_deadline - killed by the clock, with the deadline named as the reason
- Schedule_ticks_create_jobs_under_the_declared_policies - every policy read back, and a due tick that produced a finished Job
- Schedule_history_retains_exactly_the_declared_runs - two ticks came due and exactly one Job of them is kept
- Schedule_paused_window_creates_no_jobs - nothing started for longer than a full schedule period, definition intact
- Schedule_manual_trigger_from_the_template_completes - the paused schedule's own template, run once on purpose
- Composition_audit_matches_conditions_counts_and_retained_objects - the audit matches the live namespace line for line
- Composition_audit_run_against_a_scratch_namespace - the same audit, on two Jobs the grader chose, reports what is really there
When every test passes
You have:
- Six shards imported in two waves by an object that records that they finished, under the same kind of clock that killed the hung feed
- A doomed feed that gives up after two attempts, and its exit codes kept as the reason
- A hung feed killed by a clock, because nothing else was ever going to end it
- A minute-by-minute schedule with policies for a late tick, an overlapping tick and an accumulating history
- A schedule paused for a window instead of deleted, and one run of it done by hand anyway
- One command that reports the verdict, the counts and the retained objects of a whole batch namespace, and works on a namespace it has never seen
- The next topic unlocked
Verify
Click Test.
