StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 11 · Resource Requests & Limits

Limits & Enforcement

mediumResource LimitsOOM Kills & Throttling—No data yet

Problem statement

One leaky process took the node down and every neighbour with it. cache-warmer in namespace limit-enforcement allocates until it runs out of room, payments-api sits next to it, and neither declares a ceiling of any kind - so when the warmer started climbing, the kernel picked its victim out of everything on the machine, and payments went with it. Put a ceiling on what a container may use, and find out what happens when it reaches it.

About this exercise

A request is what the scheduler reserves. A limit is what the kubelet enforces at runtime, and the two resources are enforced in completely different ways:

  • Memory is incompressible. You cannot give a process less memory than it is holding - you can only refuse the next allocation. So a memory limit becomes a hard wall: when the container's usage reaches it, the kernel's out-of-memory killer terminates the process inside that container. The container's termination is recorded with the reason OOMKilled and exit code 137, the restart policy restarts it, and the restart count climbs.
  • CPU is compressible. A process that wants more CPU than its limit is simply given less: the scheduler throttles it. Nothing is killed and no restart is recorded - the workload just runs slower.

That difference is the whole point of a limit. The blast radius of a leak stops at the container that leaked: the kill lands on the offender, and a neighbour under the same ceiling that never approaches it is untouched.

Two more rules govern how requests and limits relate:

  • A limit may never be lower than the request for the same resource; the API server rejects that outright.
  • If a container declares only limits, Kubernetes fills the requests in for you - each request defaults to the matching limit. The reverse is not true: requests alone leave the container uncapped.

kubectl explain pod.spec.containers.resources and kubectl explain pod.status.containerStatuses.lastState are the reference for the fields and for what the cluster records about a termination.

How this project is graded

The grader reads the live cluster: the limits and requests each container declares, what the cluster recorded about the last time a container was terminated, how many times it has restarted, and whether the neighbouring workload was affected. Saved answers are compared against what the cluster reports right now. The namespace is re-seeded before every run, so a pod you never built cannot pass by accident.

What you will build

Everything lives in namespace limit-enforcement. cache-warmer and payments-api are already running with no ceilings; keep their images, commands and arguments exactly as they are and give them the resources below. A pod's resources are immutable, so these are replacements, not edits.

1. cache-warmer, container warmer - the leak, capped:

limits     cpu 200m    memory 32Mi

The warmer allocates far more than 32Mi, so it will not survive its own ceiling. Leave it that way: the kill and the restarts are the evidence.

2. payments-api, container api - the neighbour, under the same ceiling:

limits     cpu 200m    memory 32Mi

It must stay Running, having never been killed and never restarted.

3. Pod ledger-sync, container sync, image busybox:1.36, staying alive, Running - declaring only limits and letting the platform fill in the rest:

limits     cpu 150m    memory 96Mi

4. Two answers in answers/:

limits.txt   the limits the cluster reports for the warmer container of
             cache-warmer, exactly as it reports them:
                 cpu <value>
                 memory <value>
oom.txt      what the cluster recorded about that container's last termination:
                 reason <value>
                 exitCode <value>

Step 1 - a ceiling on the leak

Cap cache-warmer. It allocates past 32Mi within seconds of starting, so once the cap is in place the container is terminated at the wall, restarted by its policy, and terminated again - the cluster records the reason and the exit code each time.

memory_and_cpu_limits_declared_on_the_hog, hog_is_oom_killed_at_its_limit and hog_restarts_after_each_kill should pass

Step 2 - the neighbour survives

Give payments-api the same ceiling. It uses a fraction of it, so nothing happens to it at all - which is exactly what the incident was missing.

compliant_workload_survives_the_same_limit should pass

Step 3 - limits alone

Declare ledger-sync with limits and nothing else, then look at what the object holds once the API server has accepted it. Its requests are not empty.

requests_default_from_the_declared_limits should pass

Step 4 - read the enforcement back

Save the warmer's limits as the cluster reports them, and the record of its last termination.

limits_read_back_from_the_pod and oom_termination_evidence_saved should pass

When every test passes

You have:

  • A leak whose blast radius is one container instead of one node
  • The cluster's own record of a memory limit being enforced: the reason, the exit code and the restart count
  • A neighbour running untouched under the very same ceiling
  • A workload whose requests were filled in from its limits, and the readback that proves it

You do not have yet:

  • Any ranking between workloads. Requests and limits together decide which pods the node sacrifices first when it runs short; that ranking is the next exercise.

Verify

Click Test after every step.

More in Resource Requests & Limits

See the module
Requests & SchedulingGive workloads the cpu and memory requests the scheduler does its arithmetic with, prove that a request above the node's allocatable memory can never be placed, and read the node's own allocatable and allocated accounting.—No data yetmediumQoS ClassesCompose pods that land in Guaranteed, Burstable and BestEffort from the request and limit rules alone, including the case where one under-declared container costs the whole pod its class, and read the assigned classes back from status.—No data yetmediumNode Resource GovernanceRight-size a namespace of ungoverned tenants on one node: honest requests, a ceiling that keeps a leak inside its own container, a deliberate eviction ranking, and a ledger reconciling allocatable against requested.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.