VNET the Hard Way

While packaging booklore into a FreeBSD jail last weekend, I ran into an issue that forced me to learn about VNET.

The Booklore API is a Java application built with Gradle. During the build, Gradle spawns a daemon process: To honour the JVM settings for this build a single-use Daemon process will be forked. The issue is that, no matter how hard I tried to configure it (by disabling the daemon or setting the listening host and port), Gradle insisted on connecting to 127.0.0.1. In a standard jail, this means sharing the host’s loopback interface, which I wanted to avoid.

After wrestling with it for an afternoon, I gave in and explored VNET. To be honest, I’d been putting it off for a while—I knew this wouldn’t be straightforward.

VNET introduction

As it turns out, there is arguably not much clear information about VNET out there, and man pages aren’t particularly helpful either. Here’s what I gathered.

VNET (originally VIMAGE) was developed around 2003 by Marko Zec at the University of Zagreb under sponsorship of the FreeBSD and NLnet Foundations ❤️. VIMAGE code first appeared in FreeBSD 8.0, but it wasn’t until FreeBSD 12.0 that VIMAGE was built into FreeBSD GENERIC kernels.

The idea is to give each virtualized environment (jail or VM) its own isolated copy of the network stack from the IP layer up. Why would we need that? Running services like VPN or DHCP servers, avoiding port conflicts, testing complex networking or routing setups, and creating networks distinct from the host, to name a few.

For jails, this involves additional networking setup. Here’s an overview:

1. Create a bridge and attach the physical interface to it: ifconfig bridge create && ifconfig bridge0 addm em0 up
2. Configure the jail with vnet;
3. Create an epair, which is a pair of connected virtual Ethernet interfaces. ifconfig epair create gives you, for example, epair1a and epair1b.
4. Attach the host-side of the epair to the bridge: ifconfig bridge0 addm epair1.
5. Configure the jail to use the other side: vnet.interface = "epair1b";. This interface disappears from the host once the jail starts.
6. When the jail stops, the host regains the jail-side interface. Destroy the pair by destroying either side: ifconfig epair1a destroy. Actually it’s a good practice to also detach the host-side from the bridge beforehand: ifconfig bridge0 deletem epair1a.

As always, there are many variations of this setup, like creating multiple bridges, or creating a distinct jail private network. But let’s focus on this approach for now.

Epair naming

So off I went, copied random blog posts, and started fiddling with a new jail’s jail.conf:

vnet;
vnet.interface = "epair_$name";                   # cool name, eh?
exec.prestart = "ifconfig epair_$name create up"; # or variations like epair_$name…

WRONG!! This fails with

ifconfig: SIOCIFCREATE2 (epair_books): Invalid argument

The name cannot be set at creation time. You must use ifconfig epair create… or ifconfig epair<n> create…. However, you can set a name or description afterwards:

ifconfig ${epair}a up descr jail:${name}
# or
ifconfig ${epair}a up name "e0a_${name}"

By the way, some posts use the jid in the epair name: vnet.interface = "epair${jid}b";. I explored this approach as well. First, ${jid} is not an implicitly-set variable like $name, so we’d have to hardcode jids in jail.conf, which didn’t feel right. Fixed jids would enable naming epairs in one go with ifconfig epair${jid} create up, but we’d probably still want to add the jail name to the epair for clarity anyway. Also, jids are meant to be dynamic like PIDs. So I abandoned the fixed jids idea.

Eventually I remembered to check the FreeBSD Handbook, which turned out to be the clearest guide—except they also use an id to identify epairs, which again raises the question of managing these ids.

Jib

jib is a script provided as an example in the base system. It seems widely used though and people usually copy it from /usr/share/examples/jails/jib to /usr/local/sbin.

Eventually I got the jail starting with jib:

exec.prestart += "jib addm $name em0";
exec.poststop += "jib destroy $name";

exec.start += "ifconfig e0b_$name inet ${ip4}";
exec.start += "/sbin/route add default ${gw4}";

jib is a script that:

1. Creates a bridge and adds the specified interface as a member (addm) → em0bridge
2. Creates an epair and renames the interfaces to e${i}a_$name and e${i}b_$name. Hence the e0b_$name we see in blogs and forums! This renaming helps track the association between epairs and jails.
3. Assigns deterministic MAC addresses to the epair interfaces. This mainly avoids ARP cache instability—with random MAC addresses, restarted jails can be unreachable for seconds to minutes until ARP caches stabilize:

kernel: arp: 192.168.1.86 moved from 02:7a:7d:27:bf:0a to 18:03:73:b6:6b:37 on e0b_books
kernel: arp: 192.168.1.86 moved from 02:9d:24:23:d6:0a to 18:03:73:b6:6b:37 on e0b_books

That worked well, so I started adding VNET support to my jail management tool sjail. But when I spun up a Bhyve VM to run tests, the VM wasn’t reachable.

The issue? A physical interface can only belong to one bridge at a time. By default, vm-bhyve creates a bridge and attaches the physical interface to it (vm switch create public). This conflicts with jib, which does the same thing.

Conversely, if you try to create VNET jails while vm-bhyve already has the physical interface in a bridge, you’ll see:

# jail -c books
ifconfig: BRDGADD em0: Device busy
jail: books: jib addm books em0: failed

The solution: manually create a bridge (bridge0), attach the physical interface (em0) to it, then attach (1) jail epairs and (2) vm-bhyve’s public switch to bridge0: vm switch create -t manual -b bridge0 public.

Since jib doesn’t support manual bridges, I had to implement a different approach in sjail.

Sjail

While scripting the VNET setup in sjail, I faced the challenge of dynamic epair identifiers: we don’t want to manage fixed ids, but setting up VNET with epairs requires multiple commands that potentially reuse previous outputs.

To illustrate, here’s how Bastille once solved this:

exec.prestart += "epair0=\$(ifconfig epair create) && ifconfig \${epair0} up name ${host_epair} && ifconfig \${epair0%a}b up name ${jail_epair}";

Fair enough, though somewhat convoluted. Another approach is to put all the setup into a script, like jib does. This is what I ended up doing, using sjail itself:

exec.prestart += "/usr/local/sbin/sjail _vnet_start $name";
exec.poststop += "/usr/local/sbin/sjail _vnet_stop $name";

Epair destroy

Finally when I got everything working and tests passing, stopping a fresh VNET jail on a real host failed:

ifconfig: interface e0b_books does not exist

Turns out there’s a small delay for the jail-side epair to be fully released. That’s probably why jib or bastille take great care to destroy the host-side (e0a_books)!

By the way, while jib doesn’t bother, it also seems to be a good practice to detach the host-side epair from the bridge beforehand: ifconfig bridge0 deletem e0a_books.

Conclusion

VNET jails take 4-5 seconds to start instead of 1-2 for regular jails. I haven’t explored many related capabilities yet, like private jail networks, IPv6, or pf tuning, but I now understand that mixing jails with different networking types based on use case can be desirable—it doesn’t have to be all one or the other.

This learning experience reminded me that, throughout my career, networking has been the most complex topic I’ve encountered, especially in the context of virtualization.