StaffCoder
Kubernetes
15 modules · 60 challengesSign up
Module 11 · Resource Requests & Limits

Node Resource Governance

mediumNode Resource GovernanceCapacity Reconciliation—No data yet

Problem statement

One node, six tenants, zero discipline. In namespace node-governance four workloads declare nothing at all, so the node's ledger swears it is empty while the machine is full; cache-warmer leaks until something dies and it is never the warmer that dies alone; metrics-scraper is pinned harder than the payment service; and analytics-import has been Pending since it was applied, because it asks for more memory than this node has. Right-size all of it, and produce the ledger that proves it.

About this exercise

This brings together the topic's three exercises:

  • Requests - what the scheduler reserves, and the arithmetic that decides whether a pod fits on a node at all (Requests & Scheduling)
  • Limits - what the kubelet enforces at runtime, and what happens to a container that crosses its memory ceiling (Limits & Enforcement)
  • Classes - the quality of service class the API server composes out of those two fields, which is the order the node evicts in (QoS Classes)

One idea ties them together: a node's resources are a ledger, and every pod is an entry in it. A workload with no request is an entry written as zero - the node looks free, placement becomes luck, and when memory runs short the kubelet chooses its victims from a ranking nobody set on purpose. Governing a node means every tenant carries an honest entry: a request the scheduler can do arithmetic with, a limit that keeps one tenant's failure inside that tenant, and a class that matches how much the business would actually miss it.

analytics-import is the entry that stays red. Its request is larger than the node's allocatable memory, so no amount of tidying makes it fit, and pretending otherwise by shrinking its request would hide the one honest signal the namespace has. Leave it exactly as it is.

How this project is graded

The grader reads the live cluster: what every pod declares, where the scheduler placed it, what the cluster recorded about the container that overruns its ceiling, whether its neighbours were touched, and the class assigned to each pod in its status. It then runs node-ledger.sh twice and checks each section against the node and the namespace as they are at that moment - so the ledger must be read from the cluster on every run, not printed from memory. The namespace is re-seeded before each run, so anything you delete without replacing comes back.

What you will build

Everything lives in namespace node-governance. Every container runs busybox:1.36 and keeps the image, command and arguments it has today. A pod's resources are immutable, so the five governed pods are replacements, not edits.

1. The governance spec. Each pod, one container, exactly this:

pod              container   requests                    limits                       class
checkout-api     api         cpu 100m, memory 64Mi       none                         Burstable
payments-api     api         cpu 200m, memory 128Mi      cpu 200m, memory 128Mi       Guaranteed
cache-warmer     warmer      cpu 50m,  memory 16Mi       cpu 200m, memory 32Mi        Burstable
batch-loader     loader      none declared               memory 64Mi, no cpu limit    Burstable
metrics-scraper  scraper     none                        none                         BestEffort

batch-loader declares its ceiling and nothing else; what ends up in its requests is the platform's business, not yours. The five must all end up Running.

2. analytics-import stays exactly as it is - Pending, unplaced, with the scheduler's verdict attached to it. Do not resize it, do not delete it.

3. node-ledger.sh - one report, three sections, each introduced by its name on a line of its own:

ALLOCATABLE
cpu <value>                  the node's allocatable cpu and memory,
memory <value>               exactly as the node reports them

REQUESTED cpu <n> the namespace's total requested cpu, as a whole number of millicores, no suffix memory <n> the namespace's total requested memory, as a whole number of bytes, no suffix

CLASSES <pod> <class> <state> one line per pod in the namespace, sorted by pod name; <class> is the class the cluster assigned in the pod's status, <state> is "scheduled" for a pod the scheduler has placed on a node and "pending" for one it has not

The REQUESTED totals count only the pods the scheduler has actually placed: a pod still waiting has reserved nothing, so it contributes nothing. Sum every container of every placed pod, including the requests the platform filled in for a container that declared only limits.

The tests

  • Requests_governed_workloads_schedule_and_the_misfit_stays_pending - the five carry requests and are placed; the one that cannot fit is not
  • Requests_declared_per_governance_spec - each request matches the table
  • Requests_misfit_carries_the_insufficient_resource_event - the cluster's own reason for the pod that will never fit
  • Limits_declared_per_governance_spec - each ceiling matches the table, including the requests filled in from a limit
  • Limits_hog_overrun_kills_only_itself - the warmer is OOM-killed at its ceiling and restarted
  • Limits_neighbours_run_untouched_through_the_kills - every neighbour still Running, never restarted
  • QoS_each_workload_lands_in_its_defined_class - the assigned classes match the table
  • Composition_ledger_reports_node_allocatable - the ledger's allocatable is the node's own
  • Composition_ledger_requested_matches_node_accounting - the requested totals match the live sum, and fit inside allocatable
  • Composition_ledger_classes_match_live_status_on_every_run - the class table matches status, on a second run as well as the first

When every test passes

You have:

  • A node whose ledger tells the truth: every tenant an honest entry, none of them written as zero
  • One tenant's leak ending in that tenant's own container, with the neighbours running through it untouched
  • An eviction ranking that matches what the business would actually miss
  • A pod that does not fit, left visibly not fitting, with the cluster's reason attached
  • One command that reconciles allocatable against requested and names the class of every workload on the node
  • The next topic unlocked

Verify

Click Test.

More in Resource Requests & Limits

See the module
Requests & SchedulingGive workloads the cpu and memory requests the scheduler does its arithmetic with, prove that a request above the node's allocatable memory can never be placed, and read the node's own allocatable and allocated accounting.—No data yetmediumLimits & EnforcementCap a leaking container with cpu and memory limits, watch the kubelet enforce the memory wall with an OOM kill and a restart while its neighbour under the same ceiling runs untouched, and see requests default from limits.—No data yetmediumQoS ClassesCompose pods that land in Guaranteed, Burstable and BestEffort from the request and limit rules alone, including the case where one under-declared container costs the whole pod its class, and read the assigned classes back from status.—No data yetmedium

We use cookies that are necessary to run Staffcoder, and optional analytics cookies to see which pages get used. Analytics stays off unless you accept. Read our Cookie Policy.