StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 13 · Jobs & CronJobs

Failure Bounds & Deadlines

mediumBackoff LimitsActive Deadlines—No data yet

Problem statement

Namespace batch-bounds holds two jobs nobody bounded. reconcile-doomed reads a table whose schema changed: it cannot succeed, and it spent the whole weekend proving it - fail, wait, fail, wait, on the default retry budget, filling the namespace with dead pods. archive-hang is worse in a quieter way: it is waiting on an upstream feed that never arrived, so it never fails at all. It held its slot for nine hours and would still be holding it now. The nightly ledger job, which does work and finishes, has never been given a bound either - nobody has thought about the question.

About this exercise

A Job that cannot succeed and a Job that will not finish are two different failures, and they need two different bounds.

backoffLimit bounds attempts. When a pod of the Job fails, the Job creates another one, and it keeps count: once the number of failed pods passes the limit, the Job stops trying and writes a terminal condition of type Failed on itself with the reason BackoffLimitExceeded. The waiting between attempts is exponential - roughly ten seconds, then twenty, then forty, doubling up to six minutes - which is why the default budget of six retries stretches over the better part of an hour before anyone is told. The number you choose is a statement about the work: a task whose failure is a data problem will not be fixed by a seventh try.

activeDeadlineSeconds bounds wall-clock. It is measured from the moment the Job starts, across every attempt, and it does not care whether the pods are failing or sitting still. When the deadline passes, the Job terminates its running pods and takes a terminal Failed condition with the reason DeadlineExceeded. A pod being told to stop is not the same as it stopping - the container gets a termination signal and a grace period, and the Job's condition follows once the pod is really gone. This is the only bound that catches a task that hangs, because a task that hangs never fails.

Either one ends the Job for good: a terminal condition is final, and nothing retries afterwards. What is left behind is the evidence - the failed pods, their exit codes and their logs stay in the namespace to be read.

One practical fact shapes the work: a Job's spec is fixed once it is created. Neither of these fields can be edited on a running Job - a Job is replaced, not amended. kubectl explain job.spec lists the fields.

How this project is graded

The grader reads each Job's declared bounds off the live cluster, the failure and success counts the controller wrote, and the terminal condition and reason on each Job. It measures how long the hanging job was actually allowed to stay active, from its start time to the moment its failure was recorded, and it reads the exit codes and logs of the pods the doomed job left behind. Any of the three jobs that is missing is re-seeded in its original unbounded form, so a half-finished replacement is graded as unbounded.

What you will build

Three Jobs in namespace batch-bounds. manifest.yaml holds all three as they are - the first two are running now, the third has never been run - and every container, image, command, grace period and the Never restart policy stays exactly as written there.

reconcile-doomed   backoff limit 2       and no deadline
archive-hang       backoff limit 2       and an active deadline of 20 seconds
ledger-nightly     backoff limit 2       and an active deadline of 20 seconds

Nothing else changes, and no other field is added.

Step 1 - bound the doomed one

reconcile-doomed must stop on its own after the budget above, and say so in its own conditions. Three attempts is what a budget of two retries buys.

failing_job_is_bounded_by_the_retry_budget and failing_job_ends_with_the_backoff_limit_condition should pass

Step 2 - what it leaves behind

Every attempt it made is still in the namespace, and each one recorded the exit status the task really returned. That is the difference between a bounded failure and a silent one.

failed_pods_record_their_exit_evidence should pass

Step 3 - put a clock on the hang

archive-hang never fails, so no retry budget will ever end it. Give it the deadline above and let the cluster kill it, with the reason the deadline wrote.

hanging_job_is_killed_at_its_deadline should pass

Step 4 - the same bounds on work that works

ledger-nightly carries both bounds and must still reach Complete. A bound tight enough to break the healthy run is not a bound, it is an outage.

healthy_job_completes_under_the_same_bounds should pass

When every test passes

You have:

  • A doomed job that gives up after three attempts instead of retrying into Monday
  • Its failed pods, exit codes and logs kept as the evidence of why
  • A hung job killed by the clock, with DeadlineExceeded naming what killed it
  • Proof that the same bounds leave a healthy run untouched

You do not have yet:

  • Any way to run the work faster, or to run it at all without a person. Several pods at once, and a schedule that starts them, are the next exercise.

Verify

Click Test after every step.

More in Jobs & CronJobs

See the module
Run-To-Completion JobsTurn a hand-run batch pod that the kubelet keeps restarting into a Job with declared completions and a retry budget, and read the succeeded count, the retained pods and the Complete condition back off the cluster.—No data yetmediumParallelism & CronJobsGive a six-segment reindex a width instead of another night, put the mirror sync on a one-minute schedule with concurrency, lateness and history policies, pause a live schedule for a migration window and still run it once by hand.—No data yetmediumBatch Operations SuiteRun a whole batch estate properly - a six-shard import at a declared width, a retry budget for the feed that cannot succeed and a deadline for the one that never ends, a scheduled ledger tick with concurrency, lateness and history policies, a paused window still run once by hand - then audit conditions, counts and retained objects across the namespace.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.