StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 9 · Rollouts & Rollbacks

Pause, Resume & Failed Rollouts

mediumRollout ControlFailed Rollout Diagnosis—No data yet

Problem statement

Two habits are costing the checkout team. The first: a release is two edits - the image and the release marker - typed one after the other, so checkout-api in namespace release-gate rolls twice for every release, doubling the churn and leaving a half-finished version in the history. The second: last week's typo in an image tag was not noticed for ten minutes, because the Deployment sat there retrying and the cluster said nothing. Fix both.

About this exercise

A Deployment reacts to its pod template the moment the template changes, which is why two edits mean two rollouts. A Deployment can be paused: it still accepts edits, but it stops acting on them - no new ReplicaSet, no replacement, nothing. Edits accumulate on the template, and resuming compares the template against the current ReplicaSet once and rolls the difference as a single release. One revision, one churn, one entry in the history.

The other half is what happens when a release cannot finish. A pod whose image tag does not resolve never becomes ready, so the new ReplicaSet never scales up and the Deployment never completes - and by itself it would retry forever. progressDeadlineSeconds is the clock that turns that into a verdict: if the Deployment makes no progress for that long, the controller sets its Progressing condition to false with the reason ProgressDeadlineExceeded, and rollout status reports failure instead of waiting. Its default is 600 seconds, which is why nobody heard about the typo.

A failed rollout is not an outage, and the reason is the strategy. maxUnavailable is the floor under your capacity: while the new ReplicaSet is stuck at zero ready, the old one is not allowed to shed replicas below that floor, so the previous version keeps serving the whole time. That is what makes a stuck release survivable, and what makes rolling back from it uneventful.

The pods themselves say what is wrong: a container that cannot be started sits in a waiting state, with a reason recorded on its status. kubectl rollout --help and kubectl explain deployment.spec are the reference.

How this project is graded

The grader reads the live Deployment and every ReplicaSet it owns - a ReplicaSet that was never created is as visible as one that was, so a release that rolled twice cannot look like a release that rolled once. A failed rollout, on the other hand, is transient: once you roll back, the condition and the failing pods are gone, so what you recorded while it was failing is graded as your evidence, alongside the wreckage the cluster keeps. The namespace is re-seeded only when the Deployment is missing.

What you will build

Everything lives in namespace release-gate, deployment checkout-api, container api.

1. The Deployment keeps replicas: 3 and declares:

strategy type          RollingUpdate
maxSurge               1
maxUnavailable         0
progressDeadlineSeconds  30

2. Release v2, batched. With the Deployment not acting on edits, make both changes:

image        nginx:alpine
env RELEASE  v2

While it is still paused and both edits are in, save the ReplicaSet listing. Then let it roll. The two edits must land as one revision - no ReplicaSet may exist anywhere in this namespace that carries only one of them.

3. Release v3, broken. Put the container on the image nginx:1.99-does-not-exist, leaving RELEASE at v2, and let it run out of its deadline. While it is failing, record:

answers/rollout-failure.txt        the failure the cluster recorded on the deployment,
                                   naming the reason it gives
answers/pull-reason.txt            the reason on the waiting container of a pod from
                                   the broken release, exactly that word and nothing else
answers/serving-during-failure.txt how many replicas were still available while that
                                   rollout was failing, exactly that number

4. The recovery. Roll back to the previous revision, so checkout-api is serving nginx:alpine with RELEASE=v2 again on all 3 replicas.

Step 1 - set the bounds and the clock

Give the Deployment the strategy, the surge and unavailable bounds and the progress deadline above. Nothing rolls: none of it is part of the pod template.

deployment_declares_the_progress_deadline_and_disruption_bounds should pass

Step 2 - one release, not two

Stop the Deployment acting on edits, make both changes to the template, save the ReplicaSet listing with images while it is still paused, then let it roll.

paused_edits_produced_no_new_replicaset and resumed_batch_landed_as_a_single_revision should pass

Step 3 - make the failure show up

Release the broken image and let the deadline expire. Record the cluster's verdict, the reason on the stuck pods, and how many replicas were still serving while all this was happening.

failed_rollout_reported_the_progress_deadline_verdict, broken_release_pods_reported_the_image_pull_failure and old_replicas_kept_serving_through_the_failed_rollout should pass

Step 4 - back out

Return the Deployment to the revision that was working.

rollback_recovered_the_previous_healthy_revision should pass

When every test passes

You have:

  • A release that lands as one rollout no matter how many edits it took to describe
  • A deadline that turns a stuck release into a reported failure instead of a silence
  • The pods' own account of why the release could not start
  • Proof that the previous version served the whole time the broken one was failing
  • A recovery that is just the previous revision, rolled again

You do not have yet:

  • All of it in one piece. Running a real sequence of releases - bounded, batched, annotated, and reversed when one of them is a dud - is the next exercise.

Verify

Click Test after every step.

More in Rollouts & Rollbacks

See the module
Rolling UpdatesReplace an all-at-once release with a rolling one: declare surge and unavailable bounds, roll a new image through the Deployment and prove from sampled counts that the replacement stayed inside them.—No data yetmediumHistory & RollbackRecover a lost manifest from a Deployment's revision history, record change causes that survive later releases, and roll back both to the previous revision and to a named one.—No data yetmediumRelease Train DisciplineRun three releases of a web tier end to end: a bounded rolling update, a batched pause-and-resume release, and a broken one that must report itself against the progress deadline and be reversed, with a change-cause audit trail throughout.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.