StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 13 · Jobs & CronJobs

Batch Operations Suite

mediumBatch CompositionBatch Auditing—No data yet

Problem statement

Namespace batch-suite is one team's whole batch estate, and every piece of it is run the wrong way. The catalog import is six shards and import-shard is one of them, started from a terminal and restarted by the kubelet every time it finishes, because a pod does not believe work can end. feed-doomed reads a column that was dropped in the last migration: it cannot succeed, and on the default budget it will spend the rest of the week proving it. feed-hung is waiting on a partner file that never arrived, so it never fails and never finishes and holds its slot forever. purge-window is a CronJob that fires every minute and must not fire during tomorrow's migration - and must still exist the day after. Nobody can say what any of it did last night. Run the estate properly, and then write the one command that reports on it.

About this exercise

This brings together the topic's three exercises:

  • Completions - work run by a controller that counts finished pods and records that they finished (Run-To-Completion Jobs)
  • Bounds - a retry budget for work that cannot succeed and a wall-clock deadline for work that will not end (Failure Bounds & Deadlines)
  • Width and schedule - several pods in flight at once, and a schedule with policies for concurrency, lateness and history (Parallelism & CronJobs)

One idea ties them together: a batch object's value is the terminal fact it leaves behind. status.succeeded and status.failed are counts the controller wrote; a condition of type Complete or Failed, and the reason on it, is the verdict; and the pods and Jobs still in the namespace are the evidence, kept exactly as long as your history limits and your absent time-to-live say they are. Every one of those can be read hours later by something that was not watching - which is the only kind of answer worth having at six in the morning.

Two facts about changing batch objects. A Job's spec is fixed once it is created: a bound cannot be added to a running Job, it is replaced. A CronJob's suspend is the opposite - a field you can flip on the live object, which is what makes pausing a schedule different from deleting one. kubectl explain job.spec, kubectl explain cronjob.spec and kubectl create job --help are the reference.

How this project is graded

The grader reads every declared field back off the live cluster, then judges what each object actually did. For the import it reconstructs concurrency from the pods themselves - each container's own start and finish times, counted at the busiest moment - and comes back afterwards to confirm the finished Job and its pods are still readable. For the two bounded feeds it counts attempts, reads the terminal condition and its reason, and reads the exit codes and logs of the pods left behind. For the schedules it checks the policy fields, that a due tick produced a Job that finished, how many Jobs each CronJob still owns against the limit declared, and that the paused one has started nothing for longer than its own schedule period. Then it runs your audit twice: once against batch-suite, where every line must match what the cluster reports at that moment, and once against a scratch namespace it builds itself holding two Jobs it chose and no CronJobs at all. The scratch namespace is removed afterwards, and nothing in batch-suite is created, patched or deleted by the grader. Any object you delete is re-seeded in its original unrun form.

What you will build

In namespace batch-suite. manifest.yaml holds the hand-run shard pod, the two unbounded feeds and the purge CronJob exactly as they are now; every container, image, command, exit code, grace period and Never restart policy stays as written there.

Job catalog-import          completions 6, parallelism 3, backoff limit 2
                            an active deadline of 120 seconds
                            container name shard - the task import-shard runs today,
                            which prints SHARD IMPORTED

Job feed-doomed backoff limit 1 and no deadline Job feed-hung backoff limit 1 and an active deadline of 20 seconds

CronJob ledger-tick schedule every minute concurrency policy Forbid starting deadline 30 seconds successful history 1 failed history 1 job template backoff limit 2, pod restart policy Never container name tick, image busybox:1.36 the task print LEDGER TICKED, then take two seconds

CronJob purge-window paused, and otherwise exactly as it is now

Job purge-window-manual the purge run once, now, from purge-window's own job template

No ttlSecondsAfterFinished on anything: the finished Jobs and their pods are the audit trail, and history limits are what removes them here.

And batch-audit.sh, a read-only scan that takes an optional namespace argument and defaults to batch-suite:

batch-audit.sh [namespace]

It prints one line per Job sorted by name, then one line per CronJob sorted by name, then one count line:

JOB <name> <outcome> <succeeded> <failed> <pods>
CRON <name> <paused> <retained>
TOTALS <complete> <failed>

outcome the Job's terminal condition, lowercase: complete or failed - and active when it carries neither succeeded the succeeded count the controller wrote, 0 when it has written none failed the failed count the controller wrote, 0 when it has written none pods how many pods the Job still owns paused whether the CronJob is suspended, lowercase: true or false retained how many Jobs the CronJob still owns complete how many Jobs in the namespace ended complete failed how many ended failed

So a namespace holding one finished Job and no CronJobs audits as one JOB line, no CRON line, and one TOTALS.

The tests

  • Completions_import_meets_its_completions_with_the_declared_width - six shards, three at a time inside its deadline, proven by the pods' own timestamps
  • Completions_finished_import_and_its_pods_remain_readable - the Job and all six pods still there after it completed, succeeded and un-restarted, with no time-to-live
  • Bounds_doomed_feed_fails_after_exactly_the_bounded_retries - two attempts and a terminal Failed naming the budget
  • Bounds_failed_pods_record_their_exit_evidence - both attempts kept, with the task's real exit code and log
  • Bounds_hung_feed_dies_at_its_deadline - killed by the clock, with the deadline named as the reason
  • Schedule_ticks_create_jobs_under_the_declared_policies - every policy read back, and a due tick that produced a finished Job
  • Schedule_history_retains_exactly_the_declared_runs - two ticks came due and exactly one Job of them is kept
  • Schedule_paused_window_creates_no_jobs - nothing started for longer than a full schedule period, definition intact
  • Schedule_manual_trigger_from_the_template_completes - the paused schedule's own template, run once on purpose
  • Composition_audit_matches_conditions_counts_and_retained_objects - the audit matches the live namespace line for line
  • Composition_audit_run_against_a_scratch_namespace - the same audit, on two Jobs the grader chose, reports what is really there

When every test passes

You have:

  • Six shards imported in two waves by an object that records that they finished, under the same kind of clock that killed the hung feed
  • A doomed feed that gives up after two attempts, and its exit codes kept as the reason
  • A hung feed killed by a clock, because nothing else was ever going to end it
  • A minute-by-minute schedule with policies for a late tick, an overlapping tick and an accumulating history
  • A schedule paused for a window instead of deleted, and one run of it done by hand anyway
  • One command that reports the verdict, the counts and the retained objects of a whole batch namespace, and works on a namespace it has never seen
  • The next topic unlocked

Verify

Click Test.

More in Jobs & CronJobs

See the module
Run-To-Completion JobsTurn a hand-run batch pod that the kubelet keeps restarting into a Job with declared completions and a retry budget, and read the succeeded count, the retained pods and the Complete condition back off the cluster.—No data yetmediumFailure Bounds & DeadlinesBound a job that can never succeed with a retry budget and a job that never finishes with a wall-clock deadline, then read the terminal conditions, reasons and exit evidence they leave behind.—No data yetmediumParallelism & CronJobsGive a six-segment reindex a width instead of another night, put the mirror sync on a one-minute schedule with concurrency, lateness and history policies, pause a live schedule for a migration window and still run it once by hand.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.