StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 13 · Jobs & CronJobs

Parallelism & CronJobs

mediumJob ParallelismCronJob Scheduling—No data yet

Problem statement

Namespace batch-schedule runs the catalog night shift by hand. The reindex is six independent segments and each one takes about eight seconds; reindex-segment is one of them, started from a terminal, and the other five are started the same way, one after the next, by whoever is awake. The archive purge is the opposite problem: purge-archive is a CronJob that fires every minute and it must not fire during tomorrow's index migration - but it is needed again the day after, so deleting it is not the answer. And the mirror sync, which the runbook says to run every minute, is not in the cluster at all; someone types it in when they remember.

About this exercise

A Job counts completions, and by default it runs them one pod at a time. parallelism is the second number: how many pods the Job is allowed to have running at the same moment. With completions set to the amount of work and parallelism to the width, the Job keeps that many pods in flight and starts a replacement each time one succeeds, until the completions are met. Six segments at a width of three is two waves instead of six, and it costs nothing but saying so - the segments were always independent.

A CronJob is a controller whose whole job is creating Jobs. Its schedule is a cron expression in five fields, and one minute is the finest granularity a schedule has. Everything else on it is policy about what happens when reality does not fit the schedule:

  • concurrencyPolicy - what to do when a tick comes due and the previous run is still going: Allow starts it anyway, Forbid skips this tick, Replace kills the old run and starts the new one.
  • startingDeadlineSeconds - how late a missed tick may still be started. Past that it is counted as missed and skipped rather than run at the wrong hour.
  • successfulJobsHistoryLimit and failedJobsHistoryLimit - how many finished Jobs of each outcome to keep. Every tick leaves a Job behind, and a CronJob on a one-minute schedule would otherwise fill the namespace by lunchtime. When the limit is exceeded the controller deletes the oldest finished Jobs, and their pods go with them, so the number you pick is exactly how far back your audit trail goes.

Two facts about stopping one. suspend is a field on the CronJob, and unlike most of a batch spec it can be flipped on a live object: setting it stops new Jobs being created, leaves the Jobs already there alone, and leaves the definition in place for when the window is over. And a CronJob's job template can be instantiated on demand, which is how a paused schedule still gets run once by hand - the Job that comes out is an ordinary Job like any other. kubectl explain cronjob.spec lists the fields, and kubectl create job --help covers the second.

How this project is graded

The grader reads the reindex Job's declared width and its completions, then reconstructs concurrency from the pods themselves: it takes each pod's container start and finish times and counts how many were running at the busiest moment. It reads every policy field off each CronJob, checks that a due tick actually produced a Job that finished, counts how many Jobs each CronJob still owns against the limit it declares, and confirms that the paused one has not started anything for longer than its own schedule period. It reads nothing from a file. reindex-segment and the purge CronJob are re-seeded if you delete them, and the hand-run segment pod is graded as nothing - it is the control group.

What you will build

In namespace batch-schedule:

Job catalog-reindex        completions 6, parallelism 3, backoff limit 2
                           pod restart policy Never
                           container name segment, image busybox:1.36
                           the task reindex-segment runs today - it prints SEGMENT INDEXED

CronJob mirror-sync schedule every minute concurrency policy Forbid starting deadline 30 seconds successful history 1 failed history 1 job template backoff limit 2, pod restart policy Never container name sync, image busybox:1.36 the task print MIRROR SYNCED, then take two seconds

CronJob purge-archive paused, and otherwise exactly as it is now

Job purge-archive-manual the purge run once, now, from purge-archive's own job template

Set no ttlSecondsAfterFinished anywhere, and change nothing about the purge CronJob other than pausing it.

Step 1 - width, not more nights

Replace the hand-run segments with catalog-reindex, and let it finish. Six segments, three at a time.

parallel_job_declares_the_completions_and_parallelism and parallel_job_runs_three_pods_at_once_to_all_completions should pass

Step 2 - a schedule instead of a person

Declare mirror-sync with every policy above and let it come due at least twice. Its history limit is the audit trail you are choosing: after the second tick exactly one finished Job of its own may remain.

cronjob_schedule_and_policies_read_back, a_due_tick_creates_a_job and history_retention_keeps_the_declared_count should pass

Step 3 - pause the window

purge-archive must stop firing without losing its definition or the runs it already did.

suspended_cronjob_creates_no_new_jobs should pass

Step 4 - run it anyway, once

The migration still needs one purge before it starts. Produce purge-archive-manual from the paused CronJob's template and let it complete.

manual_trigger_from_the_template_completes should pass

When every test passes

You have:

  • Six segments finished in two waves, proven by the pods' own timestamps rather than a stopwatch
  • A minute-by-minute schedule with a policy for a tick that arrives while the last one is still running, and one for a tick that arrives late
  • A history limit that keeps the namespace from filling up, and says how far back you can audit
  • A schedule paused for a window instead of deleted and rebuilt from memory
  • One run of a paused schedule, done on purpose

You do not have yet:

  • Any of it in one place. Completions, bounds, width, schedules and the audit that reads all of them back is the next exercise.

Verify

Click Test after every step.

More in Jobs & CronJobs

See the module
Run-To-Completion JobsTurn a hand-run batch pod that the kubelet keeps restarting into a Job with declared completions and a retry budget, and read the succeeded count, the retained pods and the Complete condition back off the cluster.—No data yetmediumFailure Bounds & DeadlinesBound a job that can never succeed with a retry budget and a job that never finishes with a wall-clock deadline, then read the terminal conditions, reasons and exit evidence they leave behind.—No data yetmediumBatch Operations SuiteRun a whole batch estate properly - a six-shard import at a declared width, a retry budget for the feed that cannot succeed and a deadline for the one that never ends, a scheduled ledger tick with concurrency, lateness and history policies, a paused window still run once by hand - then audit conditions, counts and retained objects across the namespace.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.