Problem statement
One leaky process took the node down and every neighbour with it. cache-warmer in namespace limit-enforcement allocates until it runs out of room, payments-api sits next to it, and neither declares a ceiling of any kind - so when the warmer started climbing, the kernel picked its victim out of everything on the machine, and payments went with it. Put a ceiling on what a container may use, and find out what happens when it reaches it.
About this exercise
A request is what the scheduler reserves. A limit is what the kubelet enforces at runtime, and the two resources are enforced in completely different ways:
- Memory is incompressible. You cannot give a process less memory than it is holding - you can only refuse the next allocation. So a memory limit becomes a hard wall: when the container's usage reaches it, the kernel's out-of-memory killer terminates the process inside that container. The container's termination is recorded with the reason OOMKilled and exit code 137, the restart policy restarts it, and the restart count climbs.
- CPU is compressible. A process that wants more CPU than its limit is simply given less: the scheduler throttles it. Nothing is killed and no restart is recorded - the workload just runs slower.
That difference is the whole point of a limit. The blast radius of a leak stops at the container that leaked: the kill lands on the offender, and a neighbour under the same ceiling that never approaches it is untouched.
Two more rules govern how requests and limits relate:
- A limit may never be lower than the request for the same resource; the API server rejects that outright.
- If a container declares only limits, Kubernetes fills the requests in for you - each request defaults to the matching limit. The reverse is not true: requests alone leave the container uncapped.
kubectl explain pod.spec.containers.resources and kubectl explain pod.status.containerStatuses.lastState are the reference for the fields and for what the cluster records about a termination.
How this project is graded
The grader reads the live cluster: the limits and requests each container declares, what the cluster recorded about the last time a container was terminated, how many times it has restarted, and whether the neighbouring workload was affected. Saved answers are compared against what the cluster reports right now. The namespace is re-seeded before every run, so a pod you never built cannot pass by accident.
What you will build
Everything lives in namespace limit-enforcement. cache-warmer and payments-api are already running with no ceilings; keep their images, commands and arguments exactly as they are and give them the resources below. A pod's resources are immutable, so these are replacements, not edits.
1. cache-warmer, container warmer - the leak, capped:
limits cpu 200m memory 32Mi
The warmer allocates far more than 32Mi, so it will not survive its own ceiling. Leave it that way: the kill and the restarts are the evidence.
2. payments-api, container api - the neighbour, under the same ceiling:
limits cpu 200m memory 32Mi
It must stay Running, having never been killed and never restarted.
3. Pod ledger-sync, container sync, image busybox:1.36, staying alive, Running - declaring only limits and letting the platform fill in the rest:
limits cpu 150m memory 96Mi
4. Two answers in answers/:
limits.txt the limits the cluster reports for the warmer container of
cache-warmer, exactly as it reports them:
cpu <value>
memory <value>
oom.txt what the cluster recorded about that container's last termination:
reason <value>
exitCode <value>
Step 1 - a ceiling on the leak
Cap cache-warmer. It allocates past 32Mi within seconds of starting, so once the cap is in place the container is terminated at the wall, restarted by its policy, and terminated again - the cluster records the reason and the exit code each time.
memory_and_cpu_limits_declared_on_the_hog, hog_is_oom_killed_at_its_limit and hog_restarts_after_each_kill should pass
Step 2 - the neighbour survives
Give payments-api the same ceiling. It uses a fraction of it, so nothing happens to it at all - which is exactly what the incident was missing.
compliant_workload_survives_the_same_limit should pass
Step 3 - limits alone
Declare ledger-sync with limits and nothing else, then look at what the object holds once the API server has accepted it. Its requests are not empty.
requests_default_from_the_declared_limits should pass
Step 4 - read the enforcement back
Save the warmer's limits as the cluster reports them, and the record of its last termination.
limits_read_back_from_the_pod and oom_termination_evidence_saved should pass
When every test passes
You have:
- A leak whose blast radius is one container instead of one node
- The cluster's own record of a memory limit being enforced: the reason, the exit code and the restart count
- A neighbour running untouched under the very same ceiling
- A workload whose requests were filled in from its limits, and the readback that proves it
You do not have yet:
- Any ranking between workloads. Requests and limits together decide which pods the node sacrifices first when it runs short; that ranking is the next exercise.
Verify
Click Test after every step.
