Problem statement
Namespace batch-bounds holds two jobs nobody bounded. reconcile-doomed reads a table whose schema changed: it cannot succeed, and it spent the whole weekend proving it - fail, wait, fail, wait, on the default retry budget, filling the namespace with dead pods. archive-hang is worse in a quieter way: it is waiting on an upstream feed that never arrived, so it never fails at all. It held its slot for nine hours and would still be holding it now. The nightly ledger job, which does work and finishes, has never been given a bound either - nobody has thought about the question.
About this exercise
A Job that cannot succeed and a Job that will not finish are two different failures, and they need two different bounds.
backoffLimit bounds attempts. When a pod of the Job fails, the Job creates another one, and it keeps count: once the number of failed pods passes the limit, the Job stops trying and writes a terminal condition of type Failed on itself with the reason BackoffLimitExceeded. The waiting between attempts is exponential - roughly ten seconds, then twenty, then forty, doubling up to six minutes - which is why the default budget of six retries stretches over the better part of an hour before anyone is told. The number you choose is a statement about the work: a task whose failure is a data problem will not be fixed by a seventh try.
activeDeadlineSeconds bounds wall-clock. It is measured from the moment the Job starts, across every attempt, and it does not care whether the pods are failing or sitting still. When the deadline passes, the Job terminates its running pods and takes a terminal Failed condition with the reason DeadlineExceeded. A pod being told to stop is not the same as it stopping - the container gets a termination signal and a grace period, and the Job's condition follows once the pod is really gone. This is the only bound that catches a task that hangs, because a task that hangs never fails.
Either one ends the Job for good: a terminal condition is final, and nothing retries afterwards. What is left behind is the evidence - the failed pods, their exit codes and their logs stay in the namespace to be read.
One practical fact shapes the work: a Job's spec is fixed once it is created. Neither of these fields can be edited on a running Job - a Job is replaced, not amended. kubectl explain job.spec lists the fields.
How this project is graded
The grader reads each Job's declared bounds off the live cluster, the failure and success counts the controller wrote, and the terminal condition and reason on each Job. It measures how long the hanging job was actually allowed to stay active, from its start time to the moment its failure was recorded, and it reads the exit codes and logs of the pods the doomed job left behind. Any of the three jobs that is missing is re-seeded in its original unbounded form, so a half-finished replacement is graded as unbounded.
What you will build
Three Jobs in namespace batch-bounds. manifest.yaml holds all three as they are - the first two are running now, the third has never been run - and every container, image, command, grace period and the Never restart policy stays exactly as written there.
reconcile-doomed backoff limit 2 and no deadline archive-hang backoff limit 2 and an active deadline of 20 seconds ledger-nightly backoff limit 2 and an active deadline of 20 seconds
Nothing else changes, and no other field is added.
Step 1 - bound the doomed one
reconcile-doomed must stop on its own after the budget above, and say so in its own conditions. Three attempts is what a budget of two retries buys.
failing_job_is_bounded_by_the_retry_budget and failing_job_ends_with_the_backoff_limit_condition should pass
Step 2 - what it leaves behind
Every attempt it made is still in the namespace, and each one recorded the exit status the task really returned. That is the difference between a bounded failure and a silent one.
failed_pods_record_their_exit_evidence should pass
Step 3 - put a clock on the hang
archive-hang never fails, so no retry budget will ever end it. Give it the deadline above and let the cluster kill it, with the reason the deadline wrote.
hanging_job_is_killed_at_its_deadline should pass
Step 4 - the same bounds on work that works
ledger-nightly carries both bounds and must still reach Complete. A bound tight enough to break the healthy run is not a bound, it is an outage.
healthy_job_completes_under_the_same_bounds should pass
When every test passes
You have:
- A doomed job that gives up after three attempts instead of retrying into Monday
- Its failed pods, exit codes and logs kept as the evidence of why
- A hung job killed by the clock, with DeadlineExceeded naming what killed it
- Proof that the same bounds leave a healthy run untouched
You do not have yet:
- Any way to run the work faster, or to run it at all without a person. Several pods at once, and a schedule that starts them, are the next exercise.
Verify
Click Test after every step.
