FreeBSD Jails Management

Introduction

From 2018 to 2021 I had been a happy user of Simple-jails. I was feeling like I understood and was in control¹. But then it broke.

When this happened, I just went on and switched to the Salt jails formula. Not understanding the breakage nevertheless annoyed me, and this morning I woke up with an exciting hypothesis, had time before me and was determined to find out.

The intuition was that, between a Bastille and Simple-jails jail, the difference must show in the root directory of the jail. Because in one case the update (freebsd-update) succeeds but fails in the other.

Thin jails

Skip if you already know about thin jails.

As a quick reminder, a jail is a virtual machine sharing the same kernel with the host². As such it embeds a FreeBSD base system, i.e. base.txz is downloaded and extracted to the filesystem that will constitute the jail. Now we usually distinguish between thick and thin jails. Thick jails contain a whole system. Thin jails split read-only and writeable parts of the system, so the read-only parts (like /usr/bin or /usr/lib) can be shared between multiple jails via links. Jail management tools like Bastille or Simple-jails can do both but usually default to thin jails as they’re more space-efficient.

Diving in

Initial hypothesis

The blog post FreeBSD Jails the hard way, from which Simple-jails is derived, explains very well how jails and thin jails are built.

Trying to validate my initial hypothesis, I started comparing Simple-jails and Bastille and noticed a key difference in their approach to mounting filesystems.

In Simple-jails, the jail root is an empty directory. Simple-jails mounts the base jail (read-only directories) onto it then the thin jail as _rw. I.e. the first layer is the base jail.

# cat /etc/jail.conf
…
path = "/sjails/$name";
…

# cat /sjails/zzz.fstab
/sjails/templates/base-13.2-RELEASE     /sjails/zzz/ nullfs ro 0 0
/sjails/thinjails/zzz   /sjails/zzz/_rw nullfs rw 0 0

# ll /sjails/zzz
total 88
drwxr-xr-x   9 root  wheel     9 Jul 29 14:07 _rw
drwxr-xr-x  15 root  wheel    24 Jul 29 14:07 .
drwxr-xr-x   5 root  wheel     6 Jul 29 15:16 ..
-rw-r--r--   1 root  wheel  1023 Apr  7 06:19 .cshrc
-rw-r--r--   1 root  wheel   507 Apr  7 06:19 .profile
drwxr-xr-x   2 root  wheel    48 Apr  7 06:19 bin
drwxr-xr-x  14 root  wheel    68 Apr  7 06:29 boot
-r--r--r--   1 root  wheel  6109 Apr  7 06:29 COPYRIGHT
dr-xr-xr-x  12 root  wheel   512 Jul 29 14:23 dev
lrwxr-xr-x   1 root  wheel     7 Jul 29 14:07 etc -> _rw/etc
…

In Bastille, on the other hand, the jail root contains the thin jail (writable directories). And the base jail (read-only directories) is mounted onto it³. I.e. the first layer is the writable thin jail.

# cat /usr/local/bastille/jails/zzz/jail.conf
…
  path = /usr/local/bastille/jails/zzz/root;
…

# cat /usr/local/bastille/jails/zzz/fstab
/usr/local/bastille/releases/13.2-RELEASE /usr/local/bastille/jails/zzz/root/.bastille nullfs ro 0 0

# ll /usr/local/bastille/jails/zzz/root/
total 88
drwxr-xr-x  14 root  wheel    24 Jul 29 12:54 .
drwx------   3 root  wheel     5 Jul 29 12:54 ..
drwxr-xr-x   2 root  wheel     2 Jul 29 12:54 .bastille
-rw-r--r--   1 root  wheel  1023 Apr  7 06:19 .cshrc
-rw-r--r--   1 root  wheel   507 Apr  7 06:19 .profile
drwxr-xr-x   2 root  wheel     2 Jul 29 12:54 .template
lrwxr-xr-x   1 root  wheel    14 Jul 29 12:54 bin -> /.bastille/bin
lrwxr-xr-x   1 root  wheel    15 Jul 29 12:54 boot -> /.bastille/boot
-r--r--r--   1 root  wheel  6109 Apr  7 06:29 COPYRIGHT
dr-xr-xr-x   2 root  wheel     2 Apr  7 06:06 dev
drwxr-xr-x  29 root  wheel   109 Jul 29 12:54 etc
…

OK but the result is essentially the same: both thin jails have writeable and read-only shared directories and there are no functional difference. Which invalidates my initial hypothesis. 😞

Persevering

This is actually at this point of my quest that, while trying to remember what precisely was breaking (openssl or ca_root_nss maybe?), I figured I may have left some hints in my infrastructure-as-code repository and actually found this commit message from 2021-01-02:

We're switching to full jails. The reason is: from 12.2 on, updating the base
template fails because it creates links in /etc (ex: /etc/ssl/blacklisted). We
might get away with re-creating the base template and corresponding skeleton,
but don't know if we want need to do this on each upgrade.

So the problem was actually in the base jail. Indeed another difference between both tools is how they construct the base jail.

With Bastille the base jail (/usr/local/bastille/releases/13.2-RELEASE) contains the whole system.

With Simple-jails, after fetching the base system, you need to extract the writable directories to a “skeleton”, which will serve as the template to create writable thin jails⁴:

# simple-jails fetch 13.2-RELEASE
…

# simple-jails skel 13.2-RELEASE
…

# ll /sjails/templates/base-13.2-RELEASE
drwxr-xr-x   2 root  wheel     2 Jul 29 14:07 _rw
…
lrwxr-xr-x   1 root  wheel     7 Jul 29 14:07 etc -> _rw/etc
…

# ll /sjails/templates/skeleton-13.2-RELEASE
…
drwxr-xr-x  29 root  wheel  108 Jul 29 14:07 etc
…

As it turns out the _rw directory of the base jail is empty! So that the base jail writable directories point to non-existing directories.

This worked well as long the base system (the base jail) wasn’t modified by freebsd-update. But that changed around 12.2-RELEASE where modifications to /etc started happening:

# ./simple-jails update 13.2-RELEASE
Updating 13.2-RELEASE
Looking up update.FreeBSD.org mirrors... 2 mirrors found.
…
mkdir: /sjails/templates/base-13.2-RELEASE/etc: No such file or directory
mkdir: /sjails/templates/base-13.2-RELEASE/etc: No such file or directory
Scanning /sjails/templates/base-13.2-RELEASE/usr/share/certs/blacklisted for certificates...
install: /sjails/templates/base-13.2-RELEASE/etc/ssl/blacklisted: realpath: No such file or directory
install: /sjails/templates/base-13.2-RELEASE/etc/ssl/blacklisted: realpath: No such file or directory
…

And so updating the system really became challenging because there’s no easy way to change both the base and the skeleton as they are split into two distinct universes. When applying the changes in the context of a jail, the base part is read-only. From the host, where it’s meant to be done, the writeable part is empty.

Which raises the interesting question: how could we have fixed that? Well, in retrospect, there would have been two obvious solutions:

1. manually mount the base jail as writable in one thin jail and update the system in the context of this jail.

2. mount the skeleton as _rw in the base jail and update the system from the host.

   mount -t nullfs /sjails/templates/skeleton-13.2-RELEASE /sjails/templates/base-13.2-RELEASE/_rw
   

The second solution actually works well 🏆

Conclusion

Interestingly enough, Bastille’s implementation is actually more straightforward than Simple-jails'.

Today I’m a happy user of Bastille which simply has more features⁵ and is well maintained.

Simple-jails nevertheless did a great job in its time and remains a great learning experience. The code is small and straightforward and I forgot I actually made a modest contribution early on. 😄

Lastly, one aspect worth exploring in the future is how these tools leverage ZFS, as I was trying to do with iocage some times ago.