DIY FreeBSD Jail Management

I’m now part of the elite! 🤣 I went through the rite of passage and wrote my FreeBSD jail manager. 🏆

While I started fiddling with FreeBSD jails early in my self-hosting journey (2015), it took me a while to fully adopt them (2018). Ever since I’ve been wondering what management tool people actually use. People always said “oh I just use scripts”, which sounded crazy because 1. there were always many available and mature tools, 2. writing your own is supposedly tedious and error-prone.

Earlier this week, as I was hacking on BastilleBSD, I was re-reading my post about simple-jails, trying remember implementation details.

I also stumbled on a forum post, where someone was suggesting that a jail manager could be implemented with scripts and Makefiles. 💡 Somehow I realized that, yes, actually Bastille is admittedly a script to manage jails (albeit pretty elaborate) and its templates (recipe files similar to Dockerfiles) are in essence shell scripts.

That’s when the idea emerged: I could revive and fix simple-jails and add a library to write recipe scripts. I would thus have full control over my tool and brag with everyone else that “oh I just use my scripts”.

I couldn’t resist and spent the weekend on it. As usual I learned a bunch on the way. For example how to protect a zpool from deletion:

zfs snapshot pool/data@20241201
zfs hold keepme pool/data@20241201

Or, as I wanted to write tests and have them run in an isolate environment, how to do zfs inside jails. TL;DR:

zfs create -o compress=lz4 -o atime=off -p zroot/sjail/tests
zfs set jailed=on zroot/sjail/tests
# jail.conf
allow.mount     = 1;
allow.mount.zfs = 1;
enforce_statfs  = 1;
exec.created  += "zfs jail stest zroot/sjail/tests";
exec.prestop  += "zfs unjail stest zroot/sjail/tests";
# test
jexec -l stest zfs create -o mountpoint=/tt zroot/sjail/tests/tt

Which proved insufficient because I also needed pf inside jails… So I turned to bhyve and the popular vm manager vm-bhyve.

To be fair, while my initial implementation (jail creation and recipe mechanism draft) was completed in a weekend, a couple of difficulties arose which took more like a week to resolve. Things like nested INCLUDEs, or supporting CMD var1=1 var2=2 install.sh (or pipes and redirection in CMD).

One thing I really have to say is: jail.conf syntax, while looking nice and straightforward to humans, actually sucks big time for makes parsing really cumbersome. Same goes for related tooling. Ser-/de-serialization turned out unnecessarily difficult.

     Parameters in the jail.conf(5) file, or on the command line, are
     generally of the form “name=value”.  Some parameters are boolean, and do
     not have a value but are set by the name alone with or without a “no”
     prefix, e.g.  persist or nopersist.  They can also be given the values
     “true” and “false”.  Other parameters may have more than one value,
     specified as a comma-separated list or with “+=” in the configuration
     file (see jail.conf(5) for details).

Just wow 😮

One thing I’m pretty happy is about my implementation are tests. As always they bring confidence but also help uncover unexpected issues or corner cases. By the way thank you David Farrell for the TAP shell implementation!

Interestingly I also discovered some other approaches and different needs. One example is jcreate which uses vnet jails but not zfs. jcreate also provides a script mechanism to configure jails. The script is run during creation inside the jail.

Maybe actually rolling out your own jail manager does make sense after all. I observed this pattern over the years in the open source world, where some solo developer kindly shares their solution. But excitement and interest slowly erode as scope expands (bloat?) or the maintenance load becomes unbearable. Maybe iocage, Bastille, ezjail fall into this category¹? The new sensation is now AppJail apparently, and rightly so if you ask me.

Someone confirmed this intuition in the FreeBSD Discord server about 2 years ago. When someone asked what manager they should use, they recommended to build their own from the start:

As soon as you want to do anything beyond the defaults you have to learn it anyway, but now you’ve wasted extra time learning an irrelevant extra layer

[…] And people fix their bugs.

[…] as soon as you have your own problem space independent from the maintainers… you’re stuck learning someone else’s codebase well enough to be comfortable modifying it on the fly.

Well I agree that eventually this may be best thing to do. But I will also say that I’m certainly grateful to the Bastille community for providing me with a pretty awesome tool that 1. did a great job for me for years, 2. helped me transition a good chunk of configuration management code (Salt) to Bastille templates (paradigm shift), 3. provided a good tried-and-tested model from which I could draw inspiration to implement my own.