FreeBSD Jails for Web Hosting

So I’ve built yet another web hosting platform cough prototype cough. In an era of well-established Kubernetes (k8s) that’s not very impressive. What’s interesting though is that it’s built on FreeBSD jails: think docker + k8s from scratch—albeit still rudimentary.

So I’ve come to reflect and compare both worlds.

Motivation

It started with my good friend at System-D thinking about remaking their website. Since I was already self-hosting, we had some discussions about using WooCommerce and me providing the hosting. While the project didn’t materialize, it got me to this idea 💡: now I have my dear sjail jail manager, how hard can it be to automate the provisioning of WordPress instances?

Well I will say it’s not so trivial. It actually took me months and I even almost abandoned the project. But more and more I couldn’t shake the feeling that I was basically reproducing, to humble extent, docker and k8s.

Docker and k8s

I find it fascinating to observe history repeat itself, and I am slightly ashamed to repeat it yet again.

Jails were introduced for workload isolation in the context of web hosting back in 2000.

dotCloud, another hosting company co-founded in 2008 by Solomon Hykes, Sebastien Pahl and Kamel Founadi in Paris, (re-)invented containers on top of LXC and AUFS. While isolation/virtualization was a solved problem, what docker initially solved was image creation and distribution.

Docker was first presented publicly at PyCon in March 2013 as an internal tool. Hykes explained that developers kept asking for access to the underlying container technology powering the dotCloud platform.

Wait Docker was in Go from the start, so why present it at PyCon? Not sure, but the choice of Go was partly a social/political decision as Solomon explains:

1. A compiled binary that didn’t require installing a language runtime and therefore didn’t trigger tribalism. Devops teams back then were fragmented across Python, Ruby and Java. Every tool written in one language would instantly get cloned in the others. So we wanted to avoid that.

2. We were all Python/C programmers and Go gave us the best of both.

3. The syntax was “mainstream” without too many niche or radical concepts. We thought this would help build a large contributor community.

4. We had a nerdy gut feeling that it was cool and new, and we wanted to play with it. We felt that other engineers would also feel that way, and Docker would benefit from that aura.

For me, Docker and orchestration platforms like K8s solve the long standing problem of deployment, which is exactly my endeavour with the web hosting platform.

Images

And as it turned out building images is a real pain point. Leveraging sjail and its Bastille-Dockerfile-inspired recipes, I initially automated instance creation as:

create-jail → start-jail → apply-recipe (no notion of "image" here)

But of course that was slow and wasteful. So I evolved to manually creating a template jail that can easily be cloned:

# 1. Create the image
create-jail → start-jail → apply-recipe → zfs snapshot (equivalent of `docker build`)

# 2. Create and start jail
zfs clone → start-jail (equivalent of `docker run`)

The process is quite traumatizing for now:

1. As there is no robust unified build command, iterating on building an image looks like:
   • Create → start → apply (apply currently requires a running jail because of jexec calls)
   • Failed recipe
   • Stop → destroy
   • Optional cleanup if stop or destroy failed (zfs dataset busy, vnet epair destroy failed)
   • Fix recipe
   • Repeat 🥵
2. As there is no robust unified run command either, and as we want to test a deployment, iterating looks like:
   • Snapshot and clone template jail
   • Update test jail configuration (name, path, networking mainly)
   • Start
   • Run init script (equivalent of Docker entrypoint)
   • Failed init
   • Stop → destroy
   • Fix init
   • Repeat 🥵

But OK this we can quite easily automate further.

Single process service

So I found myself somewhat replicating docker. Docker’s paradigm is now “one container, one service”—but used to be “one process per container”, as opposed to OS-level virtualized VMs (like jails—or Linux-VServers back then) running a “whole” OS.

I find the difference rather thin actually.

Chroot and docker actually do run kind of an OS: the main binary requires its runtime environment: dependencies, libraries, /proc, /dev, etc.

On the other hand jails can actually start a single process instead of /etc/rc:

# jail.conf
exec.start = "/usr/local/bin/my-daemon --foreground";
exec.stop  = "/bin/kill -TERM $!";

And that’s exactly what pot¹ or AppJails or some people do.

By the way in practice the only other processes that are started by default in jails are syslog and cron. While on the other hand, there seems to be legitimate use case for starting multiple processes in docker images, hence supervisord.

Agreed, running multiple services in a single jail is common practice — just look at Bastille’s templates. But at that point the technical distinction between jails and Docker containers with supervisord becomes blurry. The real differences lie elsewhere.

OCI Images

With Docker and K8s’ undisputed adoption and large ecosystem, the FreeBSD community has explored ways to leverage OCI (Open Container Initiative). See for example pot, AppJails.

The OCI spec defines a container image as a set of filesystem layers, a stack of tarballs, each representing filesystem changes relative to the previous layer, merged at runtime using a union filesystem like AUFS/OverlayFS.

For historical context, by 2015 the docker ecosystem had fragmented. CoreOS was building rkt with its own format, Red Hat and Google had competing visions. So in June 2015 Docker and CoreOS announced they would collaborate under the Open Container Initiative.

The filesystem layers are a direct heritage of Docker and OverlayFS. Actually docker initially only ran on Ubuntu/Debian and RedHat worked hard with Docker on porting it to their distribution which didn’t include AUFS. RedHat developed a new storage backend based on existing Device Mapper technology.

The cool part is that layers solve two problems:

• Deduplication at rest: avoid storing debian base 50 times on disk
• Incremental transfer: only pull changed layers from the registry

But the ZFS world doesn’t have these problems: deduplication is done at block-level or via clones, incremental transfer via zfs send -i. In fact, ZFS cloning and optional block-level deduplication is more efficient than file-level layer deduplication. For example two slightly different WordPress Docker images might share a debian base layer but not the WordPress layer (say one has an extra plugin), even if most files are identical. To be fair ZFS deduplication has a RAM cost though².

FreeBSD thin jails (~500MB nullfs-shared base + ~100-200MB per jail) are heavier than a debian:latest base (~50MB). Yes, fundamentally the FreeBSD base system is a full OS userland, not just libc + a few binaries. The FreeBSD community is nevertheless exploring ways to create bare bone images comparable in size to alpine-based Linux containers.

And so trying to shoehorn jails/ZFS into OCI, while a realistic way to participate in the broader container ecosystem, feels somewhat technically counter-productive.

Idea 💡: shouldn’t the FreeBSD community, and even the OCI community, reconsider their approach towards either full-ZFS or filesystem-agnostic/single-layer?

The case against the OCI wire format

Actually on Linux, the layer model is in some ways increasingly questioned within the ecosystem:

• Nydus and eStargz, lazy-loading image formats, address the problem of slow container startup, itself due to container layer pulling. As CRFS (Container Registry Filesystem) puts it:

  […] container images are, somewhat regrettably, represented by tar.gz files, and tar files are unindexed, and gzip streams are not seekable.

• bootc / OSTree (“git for filesystems”, based ComposeFS/EROFS), Red Hat’s approach to immutable OS images, essentially considers the image as a whole filesystem snapshot, rather than a stack of diffs.

• Flatpak, which uses OSTree under the hood, addresses “application sandboxing and distribution”, i.e. a universally- linux-portable software packaging and running solution.

In fact the OCI spec mandates the layers as the wire format, not as the storage format. While runtimes can use whatever filesystem they see fit locally, the moment we push/pull from a registry we’re back to tarballs.

A ZFS-based OCI equivalent?

And so contemplating a ZFS-native approach, comparing OCI’s registry protocol vs ZFS send/recv reads:

• ZFS send/recv covers:

  • Incremental transfers (zfs send -i) — better than layer diffs
  • Deduplication
  • Compression
  • Integrity (checksums are built into ZFS at the block level)
  • Snapshots as immutable image points

• ZFS lacks:

  • Content addressability: ZFS snapshots do have a GUID but no external identifier that allows for verifying the content of the image. For snapshots, zfs send $snapshot | sha256sum should be good enough, provided consistent stream version and format (-c compressed, -w raw/encrypted, -L large blocks)³. As a ZFS user property.
  • Signatures and trust.
  • Manifest and metadata (OS, architecture, labels, exposed ports, entrypoint, environment variables). Well sjail jails do store some metadata as files embedded in the snapshot. But for building a public registry, we’d want to be able to inspect metadata without pulling the full content: a JSON file along side the snapshot or better ZFS user properties.
  • Multi-platform images: the OCI image index allows a single tag to point to multiple architectures. This is a naming/manifest convention that could easily transfer to ZFS: image naming convention (zroot/sjail/templates/wordpress-amd64@1.2) or a ZFS user property.
  • Access control on the registry (pull secrets, push permissions, token-based auth). ZFS over SSH with a Gitolite-like authorized_keys-based dispatcher would give us an interesting minimal architecture with fine-grained per-user/per-image/repo-level ACLs.

But the good news is: on a trusted internal network (private jail repository), we don’t need any of this.