Containers — [Part2]

Limit a Linux process with cgroups v2 using systemd-run, read the kernel files behind CPU and memory limits, and watch the kernel throttle and kill a process that goes over.
Linux
Containers
Author

spriyansh29

Published

September 20, 2026

In Part 1, I started a shell in new PID and UTS namespaces and compared what it could see from inside with what the host could see from outside. The shell got its own PID 1 and its own hostname, but it was still an ordinary process. Nothing stopped it from using every CPU and every byte of memory on Betelgeuse. Namespaces change what a process can see. They say nothing about how much it can use. That is the job of control groups, usually shortened to cgroups, and that is what I want to look at in this part. The approach is the same as before. Start with a process, add one piece, and check what changed.

Before starting

As in Part 1, these commands are for a Debian or Ubuntu Linux machine with access to sudo. I’m using Betelgeuse again, which is a Raspberry Pi 5 running Raspberry Pi OS. Open two terminals connected to the same machine. I’ll call them Terminal A and Terminal B.

The only tools we need are ps, top and watch from procps, and systemd-run and systemctl, which come with systemd. If you followed Part 1, nothing new is needed. Otherwise install procps. 

sudo apt-get update
sudo apt-get install procps

There are two versions of the cgroup interface. This post uses version 2, which has been the default on Debian since Bullseye, on Ubuntu since 21.10 and on Raspberry Pi OS since Bookworm. Let’s check which one the machine is using. 

stat -fc %T /sys/fs/cgroup
cgroup2fs

A process belongs to a cgroup

Every process on a Linux system is in exactly one cgroup. The cgroups form a tree, and the kernel exposes that tree as a directory structure under /sys/fs/cgroup. Each directory is a group. Each group contains files that describe its members, its limits and how much it has used. Making a new group is as simple as making a new directory. Moving a process into it is as simple as writing its PID into a file.

In Terminal A, let’s see which group our own shell belongs to. 

cat /proc/$$/cgroup
0::/user.slice/user-1000.slice/session-3.scope

The 0:: prefix means the version 2 hierarchy. The rest is a path relative to /sys/fs/cgroup. On this machine, systemd has placed the login session in a group of its own under user.slice, and anything we start from this shell lands in the same group. Your session number will differ.

If you remember the /proc/$$/ns listing from Part 1, it had an entry called cgroup. That is the cgroup namespace, which controls which part of this tree a process can see. We’ll leave it alone today and stay in the host’s view of the tree.

Now let’s look at the top of the tree and check which controllers are available. A controller is the part of the kernel that accounts for and limits one kind of resource. 

ls /sys/fs/cgroup
cgroup.controllers      cgroup.subtree_control  io.stat                 sys-kernel-debug.mount
cgroup.max.depth        cgroup.threads          memory.numa_stat        sys-kernel-tracing.mount
cgroup.max.descendants  cpu.pressure            memory.pressure         system.slice
cgroup.pressure         cpu.stat                memory.reclaim          user.slice
cgroup.procs            cpuset.cpus.effective   memory.stat
cgroup.stat             cpuset.mems.effective   misc.capacity
cgroup.subtree_control  init.scope              sys-fs-fuse-connections.mount
cat /sys/fs/cgroup/cgroup.controllers
cpuset cpu io memory hugetlb pids rdma misc

The directories init.scope, system.slice and user.slice are the first branches of the tree and for this post, we care about cpu, memory and pids.

NoteRaspberry Pi OS and the memory controller

Raspberry Pi OS ships with the memory controller switched off. If memory is missing from the list above, add cgroup_enable=memory cgroup_memory=1 to the end of the single line in /boot/firmware/cmdline.txt and reboot. Older images keep that file at /boot/cmdline.txt. Betelgeuse already had this because Kubernetes needs it.

flowchart TB
    root["/sys/fs/cgroup<br/>Root of the tree"]
    init["init.scope<br/>systemd itself"]
    system["system.slice<br/>Services"]
    user["user.slice<br/>Login sessions"]
    session["session-3.scope<br/>Terminal A and Terminal B"]
    lab["container-cgroup-lab.service<br/>Our process, in a moment"]

    root --> init
    root --> system
    root --> user
    user --> session
    system --> lab

    classDef tree fill:#F1F5F9,stroke:#64748B,color:#0F172A
    classDef shell fill:#DBEAFE,stroke:#2563EB,color:#1E3A8A
    classDef isolated fill:#CCFBF1,stroke:#0D9488,color:#134E4A
    class root,init,system,user tree
    class session shell
    class lab isolated

Since the tree is just directories, I could create a group with mkdir under /sys/fs/cgroup and write limits into it by hand. That works, but on this machine systemd owns the tree and expects to manage it. Rather than fight it, I’ll ask systemd to do the work for me with systemd-run. It starts a program as a temporary service, which means a new group under system.slice with the limits in place before the program runs. Underneath, systemd writes the same kernel files we are about to read.

Start a process with limits

In Terminal A, let’s start a small shell loop that does nothing but keep a CPU busy, and attach some limits to it. 

sudo systemd-run --unit=container-cgroup-lab --collect \
  --property=CPUQuota=20% \
  --property=MemoryMax=64M \
  --property=MemorySwapMax=0 \
  --property=TasksMax=16 \
  --property=RuntimeMaxSec=600 \
  /bin/sh -c 'while :; do :; done'
Running as unit: container-cgroup-lab.service

CPUQuota=20% allows the group one fifth of a single CPU. MemoryMax=64M caps its memory at 64 MiB, and MemorySwapMax=0 stops it from spilling into swap. TasksMax=16 limits how many processes and threads it can hold. These are systemd resource settings, and each one maps onto a kernel file. RuntimeMaxSec=600 is a safety net. The loop stops itself after ten minutes in case I forget about it, and --collect lets systemd remove the temporary unit once it has finished. If the loop has already stopped by the time you reach a later step, run the command again.

Find the process and its group

Still in Terminal A, let’s ask systemd for the PID of the loop and the group it was placed in, then check both against /proc and /sys/fs/cgroup. 

cgroup_pid=$(systemctl show container-cgroup-lab.service \
  --property=MainPID --value)
cgroup_path=$(systemctl show container-cgroup-lab.service \
  --property=ControlGroup --value)
cgroup_dir="/sys/fs/cgroup$cgroup_path"

echo "Loop PID: $cgroup_pid"
echo "Group directory: $cgroup_dir"
cat "/proc/$cgroup_pid/cgroup"
cat "$cgroup_dir/cgroup.procs"
Loop PID: 2411873
Group directory: /sys/fs/cgroup/system.slice/container-cgroup-lab.service
0::/system.slice/container-cgroup-lab.service
2411873

The path in /proc/PID/cgroup matches the directory systemd reported, and cgroup.procs lists the PIDs that belong to the group. Right now that is just the loop. Any process it started would appear here as well, because a child starts in its parent’s group. That is the same rule we saw with namespaces.

Compare that with our own shell, which is still in the session group from earlier. 

cat /proc/$$/cgroup
0::/user.slice/user-1000.slice/session-3.scope

Read the limits

The limits we asked for are sitting in plain files inside the group directory. Let’s read them. 

cat "$cgroup_dir/cpu.max"
cat "$cgroup_dir/memory.max"
cat "$cgroup_dir/memory.swap.max"
cat "$cgroup_dir/pids.max"
20000 100000
67108864
0
16

cpu.max holds two numbers in microseconds, a quota and a period. The group may run for 20,000 µs out of every 100,000 µs, which is the 20% we asked for. memory.max is 64 MiB written out in bytes, since systemd treats M as 1024 × 1024. The other two are what we set. Nothing here is specific to systemd. A container runtime writing its own limits ends up with the same files holding the same kinds of values.

Watch the CPU limit work

In Terminal A, watch the group’s CPU statistics for a few seconds. Press Ctrl+C when you have seen enough. 

watch -n 1 "cat $cgroup_dir/cpu.stat"
usage_usec 6012345
user_usec 6000123
system_usec 12222
nr_periods 300
nr_throttled 300
throttled_usec 24000000

nr_periods counts the 100 ms periods that have passed, and nr_throttled counts how many of those ended with the group being paused. The loop never stops asking for CPU, so the two numbers climb together. In each period, it runs for about 20 ms and then waits for the remaining 80 ms, which is why throttled_usec grows roughly four times faster than usage_usec.

Let’s check the same thing from the point of view of a process tool. In Terminal B, look up the PID again and ask top for two samples two seconds apart. The first sample is an average since the process started, so we keep the second. 

cgroup_pid=$(systemctl show container-cgroup-lab.service \
  --property=MainPID --value)
top -b -n 2 -d 2 -p "$cgroup_pid" | tail -n 2
    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
2411873 root      20   0    2420   1024    896 R  20.0   0.0   1:12.34 sh

A shell loop like this would normally sit at 100% of a core. Here it is held to around 20. Betelgeuse has four cores, and the other three are untouched.

Change the limit while it runs

The limit is just a value in a file, so it can be changed without restarting anything. In Terminal A, ask systemd to raise the quota and read the file again. --runtime means the change is not saved to disk and disappears with the unit. 

sudo systemctl set-property --runtime container-cgroup-lab.service CPUQuota=50%
cat "$cgroup_dir/cpu.max"
50000 100000

Run the top command in Terminal B once more and the loop should now be sitting near 50%. The kernel picked up the new quota at the start of the next period. This is what happens when a runtime changes a running container’s CPU limit. Someone writes a new pair of numbers into cpu.max.

Connect it back to namespaces

This loop has limits, but it has not been given new namespaces. In Terminal A, compare its namespace identifiers with our shell’s, and then compare the two cgroup paths. 

readlink /proc/$$/ns/pid
sudo readlink "/proc/$cgroup_pid/ns/pid"
readlink /proc/$$/ns/uts
sudo readlink "/proc/$cgroup_pid/ns/uts"
cat /proc/$$/cgroup "/proc/$cgroup_pid/cgroup"
pid:[4026531836]
pid:[4026531836]
uts:[4026531838]
uts:[4026531838]
0::/user.slice/user-1000.slice/session-3.scope
0::/system.slice/container-cgroup-lab.service

The namespace identifiers match. Both processes see the same process list and the same hostname. The cgroup paths differ. In Part 1 we changed one of these properties, and in this part we changed the other. They are independent, and a container runtime sets both. 

flowchart LR
    process["One process"]
    ns["Namespaces<br/>What it can see<br/>Part 1"]
    cg["cgroup<br/>How much it can use<br/>Part 2"]

    process --- ns
    process --- cg

    classDef proc fill:#EDE9FE,stroke:#7C3AED,color:#4C1D95
    classDef isolated fill:#CCFBF1,stroke:#0D9488,color:#134E4A
    classDef limit fill:#FEF3C7,stroke:#D97706,color:#78350F
    class process proc
    class ns isolated
    class cg limit

Stop the loop

In Terminal A, stop the service and then try to list the group directory. 

sudo systemctl stop container-cgroup-lab.service
ls "$cgroup_dir"
ls: cannot access '/sys/fs/cgroup/system.slice/container-cgroup-lab.service': No such file or directory

The directory is gone. A cgroup with no processes left in it is removed along with its files. The group only existed for as long as something was running inside it.

Run into the memory limit

Reading memory.max tells us the limit is set. I wanted to see what the kernel actually does when a process goes over it. Let’s start a second service, this time with a loop that keeps growing. Every second it doubles the length of a string, so its memory climbs from a few kilobytes to past 64 MiB in well under a minute. I’ve left out --collect so that the unit stays around for us to inspect after it fails. 

sudo systemd-run --unit=container-memory-lab \
  --property=MemoryMax=64M \
  --property=MemorySwapMax=0 \
  /bin/sh -c 'data=x; while :; do data="$data$data"; sleep 1; done'

Straight away, in Terminal A, find the new group and watch its memory use and its event counters together. 

memory_path=$(systemctl show container-memory-lab.service \
  --property=ControlGroup --value)
memory_dir="/sys/fs/cgroup$memory_path"
watch -n 1 "cat $memory_dir/memory.current $memory_dir/memory.events"
41943040
low 0
high 0
max 0
oom 0
oom_kill 0
oom_group_kill 0

The first number is memory.current, the group’s memory use in bytes. It roughly doubles with each refresh. The lines below are from memory.events. When memory.current reaches the limit, the kernel first tries to reclaim memory from the group. Swap is off, so there is nothing to push out. It then invokes the out-of-memory killer within the group, and oom and oom_kill tick over to 1. Since the shell was the only process in the group, it is the one that gets killed. After that, watch starts reporting that the files no longer exist, because the group has been removed along with its last process. Press Ctrl+C.

systemd noticed as well. Let’s ask it what happened. 

systemctl status container-memory-lab.service --no-pager
× container-memory-lab.service - /bin/sh -c data=x; while :; do data="$data$data"; sleep 1; done
     Loaded: loaded (/run/systemd/transient/container-memory-lab.service; transient)
  Transient: yes
     Active: failed (Result: oom-kill) since Sat 2026-10-31 11:42:19 CET; 20s ago
   Duration: 23.410s
    Process: 2412990 ExecStart=/bin/sh -c data=x; while :; do data="$data$data"; sleep 1; done (code=killed, signal=KILL)
   Main PID: 2412990 (code=killed, signal=KILL)
        CPU: 1.202s

Oct 31 11:41:55 betelgeuse systemd[1]: Started container-memory-lab.service - /bin/sh -c data=x; while :; do data="$data$data"; sleep 1; done.
Oct 31 11:42:19 betelgeuse systemd[1]: container-memory-lab.service: A process of this unit has been killed by the OOM killer.
Oct 31 11:42:19 betelgeuse systemd[1]: container-memory-lab.service: Main process exited, code=killed, status=9/KILL
Oct 31 11:42:19 betelgeuse systemd[1]: container-memory-lab.service: Failed with result 'oom-kill'.

The result is oom-kill, and the process ended with signal 9. The kernel also wrote a line to its log. Let’s find it. 

sudo dmesg | grep -i 'memory cgroup out of memory' | tail -n 1
[1234567.890123] Memory cgroup out of memory: Killed process 2412990 (sh) total-vm:68420kB, anon-rss:65024kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:160kB oom_score_adj:0

The wording matters. A plain “Out of memory” message means the whole machine ran out. “Memory cgroup out of memory” means one group hit its own limit while the rest of Betelgeuse carried on as normal. If you have ever seen a Kubernetes pod reported as OOMKilled with exit code 137, this is the mechanism behind it. 137 is 128 plus 9, the signal we just saw.

Clear the failed unit so that systemd forgets about it. 

sudo systemctl reset-failed container-memory-lab.service

Back to containers

We have put a process in a group of its own, read its limits out of the kernel’s files, watched the kernel throttle its CPU and watched it kill the process for using too much memory. None of this needed a container runtime. It needed a directory under /sys/fs/cgroup and a few values written into files.

Between the two parts, the process now has a separate view of PIDs and the hostname, and a cap on CPU and memory. It still sees the host’s files. In Part 3, I’ll look at changing where / points for a process, which is where a container starts to look like one.

Have thoughts, questions, or suggestions? I’d love to hear them.