cgroups v2 from scratch

Every container you’ve ever run is, underneath, a normal Linux process with two things bolted on: a set of namespaces that change what it can see, and a cgroup that limits what it can use. Namespaces get most of the attention because they’re the visible magic — a process that thinks it’s PID 1 on its own machine. cgroups are the quieter half. They’re the reason a container can’t eat all the RAM on the box and take its neighbors down with it.

This post is my notes from sitting in front of a Linux shell and building cgroups by hand — no Docker, no systemd unit, just mkdir and echo into files. The “from scratch” part matters: once you’ve created a cgroup with a directory and capped its memory by writing a number to a file, the abstraction stops being mysterious. It’s a filesystem.

Everything here is cgroup v2. v1 still exists and still ships, but v2 has been the default on every major distro for years now, and it’s what containers and systemd actually use today.

The one big change from v1: a single hierarchy

The thing to understand before touching anything is why v2 exists at all.

In cgroup v1, every controller (the kernel subsystem that enforces one kind of limit — memory, cpu, io, and so on) had its own independent hierarchy. You could mount the memory controller as one tree and the cpu controller as a completely different tree, with different groupings. A process could be in one memory cgroup and an unrelated cpu cgroup at the same time.

This sounds flexible. In practice it was a mess. The controllers couldn’t coordinate because they didn’t share a structure — the memory controller had no idea which cpu group a process belonged to, so things like “account this page-cache writeback to the right io group” were basically impossible.

cgroup v2 throws that out. There is one hierarchy. Every process lives in exactly one cgroup, and all controllers operate on that same tree. You enable or disable individual controllers per-subtree, but the structure is shared.

It’s a filesystem

cgroup v2 is exposed as a pseudo-filesystem of type cgroup2, mounted (on basically every modern system) at /sys/fs/cgroup. You don’t usually mount it yourself — the init system does it at boot — but you can confirm it:

$ mount | grep cgroup2
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime,...)

If you ever needed to mount it on a bare system, it’s a one-liner:

mount -t cgroup2 none /sys/fs/cgroup

Once mounted, the entire API is files. You create a cgroup by making a directory. The kernel automatically populates that directory with the control and stats files — you never create those yourself:

$ cd /sys/fs/cgroup
$ mkdir demo
$ ls demo
cgroup.controllers   cgroup.procs        cpu.stat        memory.current
cgroup.events        cgroup.subtree_control  io.stat     memory.events
cgroup.freeze        cgroup.threads      memory.max      memory.stat
...

A few of these are worth naming now, because they show up everywhere:

• cgroup.procs — the list of process IDs in this cgroup. Read it to see who’s here; write a PID to it to move that process in.
• cgroup.controllers — which controllers are available to this cgroup (handed down from the parent).
• cgroup.subtree_control — which controllers are enabled for the children of this cgroup.
• *.max, *.current, *.stat, *.events — the per-controller limits, live readings, statistics, and event counters.

Moving a process is as blunt as it sounds. Writing a PID into cgroup.procs migrates it:

$ echo $$ > /sys/fs/cgroup/demo/cgroup.procs   # put this shell in 'demo'
$ cat /proc/self/cgroup
0::/demo

That 0::/demo line in /proc/self/cgroup is the v2 tell: a single 0:: entry (no per-controller list like v1 had) pointing at the cgroup path relative to the mount.

Enabling controllers, and the “no internal processes” rule

Creating a directory gives you a cgroup, but it doesn’t automatically give you limits. A controller has to be enabled for a cgroup’s children by writing to the parent’s cgroup.subtree_control:

$ cd /sys/fs/cgroup
$ cat cgroup.controllers          # what's available at the root
cpuset cpu io memory hugetlb pids
$ echo "+memory +cpu" > cgroup.subtree_control
$ cat demo/cgroup.controllers     # now memory + cpu are available in demo
cpu memory

The +/- syntax enables and disables. A controller can only be enabled for a child if it’s available in the parent — controllers flow downward, and a parent can never hand out a controller it doesn’t itself have.

Here’s the rule that trips everyone up the first time. In v2, a cgroup that has controllers enabled for its children cannot also contain processes directly (the root cgroup is the one exception). This is the no internal processes rule. The reasoning: a controller distributes a resource between a cgroup’s children, and it’d be ambiguous to also have loose processes sitting at that same level competing with the child groups.

So the moment you try to do both, the kernel says no:

$ echo "+memory" > demo/cgroup.subtree_control
$ echo 1234 > demo/cgroup.procs
bash: echo: write error: Device or resource busy

The fix is the structure every real system actually uses: processes live in leaf cgroups. You push the work down into children and reserve the inner nodes for grouping and distribution.

inner cgroup: no processes, distributes resources leaf cgroup: holds the processes, carries the limits

The memory controller, and the kill this site is named after

This is the controller worth knowing in detail, partly because it’s the one that bites, and partly because it’s the literal subject of this domain.

The headline knobs:

• memory.max — the hard limit. The cgroup’s usage cannot exceed this. Default is max (unlimited).
• memory.high — a soft limit. Crossing it doesn’t kill anything; it puts the cgroup under heavy reclaim pressure and throttles allocations to slow it down. This is the knob you actually want for graceful behavior.
• memory.low / memory.min — protection in the other direction: memory the kernel tries (low) or refuses (min) to reclaim from this group when the system is under pressure.
• memory.current — how much the cgroup is using right now.
• memory.events — counters: low, high, max, oom, and oom_kill.

Let me cap a cgroup hard and watch it die. I did this in my homelab on a throwaway VM, because the whole point is to trigger an OOM kill on purpose.

$ cd /sys/fs/cgroup/demo
$ mkdir hog && echo "+memory" > cgroup.subtree_control
$ echo 64M > hog/memory.max          # 64 MiB hard cap, no swap headroom

# launch a shell into the leaf, then allocate ~200MB
$ echo $$ > hog/cgroup.procs
$ python3 -c 'x = bytearray(200 * 1024 * 1024); input()'
Killed

Killed. The process asked for 200 MiB inside a 64 MiB box. The kernel tried to reclaim pages, couldn’t free enough (there’s nothing reclaimable in a fresh bytearray), and invoked the OOM killer scoped to that cgroup — it doesn’t go looking at the rest of the machine. The counters tell the story afterward:

$ cat hog/memory.events
low 0
high 0
max 41
oom 1
oom_kill 1

That max 41 is the number of times an allocation bumped into the limit and forced reclaim before the final oom_kill 1. This is the exact mechanism behind a container showing OOMKilled in kubectl describe pod: the container’s memory cgroup hit memory.max, reclaim failed, the cgroup-scoped OOM killer fired. The node as a whole was fine. One box, one limit, one dead process.

A detail that surprises people: memory.current includes page cache, not just anonymous heap memory. A process that reads a large file can push memory.current up with cached pages. That’s usually fine — page cache is reclaimable, so it gets dropped under pressure rather than triggering a kill — but it means memory.current reads higher than the “real” working set, and that’s by design.

high = slow down, stay alive max = hard wall; the kill is scoped to this cgroup only

The cpu controller: weights and quotas

CPU limiting comes in two flavors, and they answer different questions.

cpu.weight answers “when the CPU is contended, what share does this group get?” It’s proportional, defaults to 100, and ranges from 1 to 10000. A group with weight 200 gets twice the CPU of a sibling with weight 100 — but only when they’re actually competing. If nothing else wants the CPU, a low-weight group can still use all of it. This is work-conserving: idle capacity isn’t wasted.

cpu.max answers “what’s the absolute ceiling, even if the CPU is idle?” It’s a hard quota written as two numbers, $QUOTA $PERIOD, both in microseconds:

$ echo "50000 100000" > batch/cpu.max   # 50ms of CPU per 100ms = half a core

That group gets at most 50ms of CPU time in every 100ms window. Once it’s spent its quota, it’s throttled until the next period — even if every other core on the machine is sitting idle. This is not work-conserving, and that’s exactly the point: it gives you predictable, capped behavior. cpu.max of "max 100000" (the default) means no ceiling.

The usage and throttling stats live in cpu.stat:

$ cat batch/cpu.stat
usage_usec 4192310
user_usec 3011200
system_usec 1181110
nr_periods 240
nr_throttled 173
throttled_usec 8021442

nr_throttled and throttled_usec are the ones to watch. If a workload feels slow and these are climbing, the cgroup is hitting its quota wall — the classic symptom of a too-tight CPU limit on a container. The app isn’t broken; it’s being told to wait.

io, pids, and pressure

The same pattern repeats for the rest:

• io — io.max sets per-device read/write bandwidth and IOPS ceilings (keyed by device major:minor), and io.weight does proportional sharing. Proportional io control needs a capable elevator — bfq or the blk-iocost cost model — to mean much.
• pids — pids.max caps the number of processes/threads in the cgroup. Small, boring, and the cheapest defense against a fork bomb taking down a node.

The one extra thing v2 gives you that’s genuinely useful in production is PSI — Pressure Stall Information. Each cgroup exposes cpu.pressure, memory.pressure, and io.pressure:

$ cat hog/memory.pressure
some avg10=12.40 avg60=8.10 avg300=2.30 total=98234110
full avg10=9.80 avg60=6.05 avg300=1.70 total=71200320

some is the share of time at least one task was stalled waiting on memory; full is the share where everything was stalled. Unlike a raw utilization number, pressure tells you whether contention is actually hurting — a group can be at 100% memory usage and zero pressure (it just fits), or have spare capacity and high pressure (it’s thrashing). It’s the signal that finally distinguishes “busy” from “in trouble.”

How containers actually use all this

When you docker run or schedule a pod, nothing exotic happens at the cgroup layer — it’s the same mkdir and echo you just did, done by the runtime. The container runtime (via runc or crun) creates a leaf cgroup for the container, writes the configured limits into memory.max, cpu.max, pids.max, and moves the container’s init process into cgroup.procs. That’s it. The OOMKilled status, the CPU throttling, the memory accounting — all of it is the kernel files we’ve been poking at.

On a systemd machine the tree is delegated: systemd owns /sys/fs/cgroup and hands out subtrees. Your services land under system.slice, user sessions under user.slice, and a runtime that wants to manage its own children asks for a delegated subtree (Delegate=yes) so systemd promises not to reorganize underneath it. This is why you’ll see kubepods.slice or system.slice/docker-<id>.scope paths — that’s the delegation boundary, not magic.

---

The thing that stuck with me after doing this by hand is how small the interface is. There’s no daemon, no API server, no protocol. A cgroup is a directory. A limit is a number you write to a file. The OOM killer that gives this site its name is the kernel reading memory.max, comparing it to memory.current, failing to reclaim, and picking a victim inside that one directory’s subtree. Every layer of container tooling above this — runc, containerd, the kubelet — is ultimately just automating mkdir and echo. Once you’ve watched a process get killed for crossing a number you wrote into a file, “OOMKilled” stops being a scary status and starts being a sentence you can finish.