flowchart TB
root["/<br/>Host root"]
etc["/etc"]
home["/home"]
tmp["/tmp"]
lab["/tmp/container-root.k3Qd7X"]
bin["bin/busybox"]
msg["message.txt"]
proc["proc/"]
shell["Our shell<br/>root points here"]
chrooted["chroot shell<br/>root points here"]
root --> etc
root --> home
root --> tmp
tmp --> lab
lab --> bin
lab --> msg
lab --> proc
shell -.-> root
chrooted -.-> lab
classDef tree fill:#F1F5F9,stroke:#64748B,color:#0F172A
classDef newroot fill:#CCFBF1,stroke:#0D9488,color:#134E4A
classDef host fill:#DBEAFE,stroke:#2563EB,color:#1E3A8A
classDef proc fill:#EDE9FE,stroke:#7C3AED,color:#4C1D95
class root,etc,home,tmp tree
class lab,bin,msg,proc newroot
class shell host
class chrooted proc
In Part 1, namespaces gave a process its own view of process IDs and the hostname. In Part 2, cgroups put a cap on how much CPU and memory it could use. Through all of that, the process kept using the host’s files. It ran the host’s Bash, read the host’s /etc and could have written anywhere on the disk that root is allowed to. That does not feel like a container yet. When I start a container from an image, I expect to find that image’s files at /, not Betelgeuse’s. In this part, I’ll look at the oldest tool for doing that, chroot, and then put the three parts together.
Before starting
The commands are for a Debian or Ubuntu Linux machine with access to sudo, and I’m running them on Betelgeuse as before. Open two terminals connected to the same machine. I’ll call them Terminal A and Terminal B.
chroot itself comes with coreutils, so it is already installed. We need two more packages.
busybox-staticprovides a single executable containing a shell and a set of basic commands such aslsandcat.filetells us what kind of file something is. We’ll use it to check how BusyBox was built.
sudo apt-get update
sudo apt-get install busybox-static fileA process has a root directory
When a program opens /etc/hostname, the kernel has to decide where that leading slash points. Every process carries its own answer. It is a property of the process called the root directory, and children inherit it from their parent, the same way they inherit namespaces and cgroups. Normally it points at the real root of the filesystem, and we never think about it. Let’s look at ours in Terminal A.
readlink /proc/$$/root
ls /proc/$$/root
NoteExample output
/
bin dev home lib64 media opt root sbin sys usr
boot etc lib lost+found mnt proc run srv tmp var
The root link under /proc/PID shows where that process’s / is, and listing through it shows what the process sees there. For our shell, that is the ordinary root of Betelgeuse’s disk. chroot changes this link for a process to some other directory. From then on, every absolute path that process and its children resolve starts there instead.
Build a small root
If I point a process’s root at an empty directory, it will not find a shell to run. So the first job is to put something in that directory. This is also the point where BusyBox earns its place. Most programs on a Linux system are dynamically linked. They need the dynamic loader and a set of shared libraries from /lib to start, and none of those will exist in a new root unless I copy them in too. BusyBox from busybox-static has everything compiled into one file. Let’s check that first.
busybox_path=$(command -v busybox)
file "$busybox_path"
NoteExample output
/usr/bin/busybox: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, BuildID[sha1]=5f3c9d0e1b2a4c6f8e7d9a0b1c2d3e4f5a6b7c8d, for GNU/Linux 3.7.0, stripped
The important words are “statically linked”. On an x86 machine, the architecture will read differently. If it says “dynamically linked”, make sure busybox-static rather than plain busybox is installed.
Now let’s create the directory. It gets a bin with BusyBox in it, an empty proc that we’ll need at the end of the post, and a text file so we can recognise the place once we are inside.
lab_root=$(mktemp -d /tmp/container-root.XXXXXX)
mkdir "$lab_root/bin" "$lab_root/proc"
cp "$busybox_path" "$lab_root/bin/busybox"
printf 'Hello from the new root\n' > "$lab_root/message.txt"
echo "$lab_root"
ls -R "$lab_root"
NoteExample output
/tmp/container-root.k3Qd7X
/tmp/container-root.k3Qd7X:
bin message.txt proc
/tmp/container-root.k3Qd7X/bin:
busybox
/tmp/container-root.k3Qd7X/proc:
That is the whole filesystem for our process. Keep Terminal A’s shell open for the rest of the post, because lab_root is set there and the cleanup at the end relies on it. Your random suffix will differ from mine.
Enter the new root
Before using BusyBox, let’s see what happens if I ask for the host’s ls inside the new root.
sudo chroot "$lab_root" /bin/ls
NoteExample output
chroot: failed to run command '/bin/ls': No such file or directory
chroot changed the root and then tried to run /bin/ls, which now means /tmp/container-root.k3Qd7X/bin/ls. There is no such file. The host’s /bin/ls is still on the disk, but from inside the new root it is unreachable. This is the point of the exercise, so the error is expected.
Now let’s start BusyBox’s shell instead.
sudo chroot "$lab_root" /bin/busybox sh
The prompt changes to / #. The commands below run inside this shell in Terminal A. BusyBox is one executable that behaves as whichever command it is called as, so I ask it to create the usual names for its commands as links in /bin. After that, ls and cat work as normal.
/bin/busybox --install -s /bin
pwd
ls /
cat /message.txt
ls /bin | head -n 5
NoteExample output
/
bin message.txt proc
Hello from the new root
[
[[
acpid
add-shell
addgroup
/ is our directory. The message is there, and /bin is now full of BusyBox’s commands. Let’s look for a few things the host has and this root does not.
ls /etc
cat /etc/hostname
cat /proc/self/status
NoteExample output
ls: /etc: No such file or directory
cat: can't open '/etc/hostname': No such file or directory
cat: can't open '/proc/self/status': No such file or directory
All three errors are expected. There is no /etc because I never created one. /proc exists as a directory but nothing is mounted on it, so the kernel’s process information is not available in here. Tools such as ps have nothing to read. We’ll fix that at the end.
Files are hidden, but what about everything from the earlier parts? Let’s check the hostname and the shell’s PID.
hostname
echo $$
NoteExample output
betelgeuse
2419301
The hostname is still betelgeuse, and the PID is an ordinary host PID rather than 1. chroot changed one thing, the root directory. Namespaces are untouched, there is no cgroup of its own, and the process is as visible on the host as any other. Leave this shell open.
Look at it from outside
In Terminal B, find the BusyBox shell and read its root link. Run only one copy of this experiment at a time, because this command selects the most recently started busybox process.
chroot_pid=$(pgrep -n -x busybox)
echo "Shell PID on the host: $chroot_pid"
sudo readlink "/proc/$chroot_pid/root"
readlink /proc/$$/root
NoteExample output
Shell PID on the host: 2419301
/tmp/container-root.k3Qd7X
/
The PID matches the one printed inside, and the root link points at our temporary directory. Terminal B’s shell still has /. Two processes on the same machine, resolving the same paths against different starting points.
The host can reach straight through that link. Let’s read the message file by way of the process, then compare namespaces the way we did in the earlier parts.
sudo cat "/proc/$chroot_pid/root/message.txt"
sudo awk '/^NSpid:/' "/proc/$chroot_pid/status"
readlink /proc/$$/ns/mnt
sudo readlink "/proc/$chroot_pid/ns/mnt"
readlink /proc/$$/ns/pid
sudo readlink "/proc/$chroot_pid/ns/pid"
NoteExample output
Hello from the new root
NSpid: 2419301
mnt:[4026531832]
mnt:[4026531832]
pid:[4026531836]
pid:[4026531836]
NSpid has a single number, so the shell has the same PID everywhere. The mount namespace identifiers match too, which surprised me a little the first time. chroot does not create a mount namespace. Both processes share the same set of mounts, and the chroot shell simply starts its paths from a directory further down the same tree. That is also why chroot on its own is not a security boundary. A root user inside has several well-documented ways back out, and the chroot manual says as much. Container runtimes use a mount namespace together with pivot_root for the real thing. For understanding what a root directory is, chroot is enough.
Back in Terminal A, leave the chroot shell.
exitPut the three parts together
We now have all the pieces that the series set out to find. Namespaces from Part 1, a cgroup from Part 2 and a root directory from this part. Each one is a property of a process, and each one was set with a different tool. The tools can be chained, because every one of them starts a program, and that program can be the next tool. In Terminal A, with lab_root still set, let’s run them all in one command.
sudo systemd-run --pty --collect --unit=container-lab \
--property=CPUQuota=20% \
--property=MemoryMax=64M \
unshare --uts --pid --fork --mount \
chroot "$lab_root" /bin/busybox sh
NoteExample output
Running as unit: container-lab.service
Press ^] three times within 1s to disconnect TTY.
/ #
Reading from the outside in, systemd-run creates a cgroup with the limits and starts unshare inside it. --pty connects the result to this terminal so we get an interactive shell. unshare creates new UTS, PID and mount namespaces and starts chroot. chroot changes the root directory and finally starts the BusyBox shell. The --mount option is new. It gives the process its own mount namespace, so a mount made inside stays inside.
We are now at the / # prompt in Terminal A. Let’s mount /proc, which is what the empty directory was for, and then check each property in turn.
mount -t proc proc /proc
hostname container-lab
hostname
echo $$
ps
cat /proc/self/cgroup
ls /
NoteExample output
container-lab
1
PID USER TIME COMMAND
1 0 0:00 /bin/busybox sh
6 0 0:00 ps
0::/system.slice/container-lab.service
bin message.txt proc
The hostname is ours. The shell is PID 1, and ps lists only the processes in this PID namespace, because the /proc we mounted belongs to it. The cgroup is the one systemd created with the CPU and memory limits, and / holds the three entries we put there. Every result in this list comes from one of the three parts.
In Terminal B, the host’s view has not changed. The hostname is still betelgeuse, and the shell appears in the process list with a large host PID.
hostname
lab_pid=$(pgrep -n -x busybox)
sudo awk '/^NSpid:/' "/proc/$lab_pid/status"
sudo readlink "/proc/$lab_pid/root"
cat "/proc/$lab_pid/cgroup"
NoteExample output
betelgeuse
NSpid: 2420877 1
/tmp/container-root.k3Qd7X
0::/system.slice/container-lab.serviceflowchart LR
process["One BusyBox shell"]
ns["Namespaces<br/>Own PIDs, hostname and mounts<br/>Part 1"]
cg["cgroup<br/>CPU and memory limits<br/>Part 2"]
rootdir["Root directory<br/>Own files at /<br/>Part 3"]
process --- ns
process --- cg
process --- rootdir
classDef proc fill:#EDE9FE,stroke:#7C3AED,color:#4C1D95
classDef isolated fill:#CCFBF1,stroke:#0D9488,color:#134E4A
classDef limit fill:#FEF3C7,stroke:#D97706,color:#78350F
classDef files fill:#DBEAFE,stroke:#2563EB,color:#1E3A8A
class process proc
class ns isolated
class cg limit
class rootdir files
Back in Terminal A, leave the shell. Exiting PID 1 ends the PID namespace, the mount namespace goes with it, and systemd removes the unit and its cgroup because of --collect.
exit
Then remove the directory from Terminal A’s normal shell, where lab_root is still set.
sudo rm -r "$lab_root"Back to containers
What we built at the end is a process with its own process IDs, hostname and mounts, a cap on CPU and memory, and a directory of its own at /. That is most of what a container is, however Docker or Podman does a lot more than what I have done.
Have thoughts, questions, or suggestions? I’d love to hear them.