FreeBSD Jails: Filesystems and FIBs

In my Putting Jails to Work entry, I detailed how I set up the jails on joker.  Part of that entry dealt with the null filesystem mounts that I had to do on joker so that the jails would have access to an NFS-mounted filesystem.  It turns out that I did a lot more work than was necessary, because the jail architecture on FreeBSD has all that’s necessary for on-demand filesystem mounts.  Further, I learned of a different way to secure the networking of the jails, due to joker being dual-homed.

Filesystems in Jails

If you reference the previous article, you’ll see that I thought I was pretty clever writing that rc  init script to mount the null filesystems after the NFS mount.  As a reminder: the null filesystems for the jails all referenced various directories available on joker only through an NFS mounted filesystem from bane.  When I allowed the filesystems to be mounted during boot, init would attempt to mount them first before the NFS mounts.  That would fail.  The rc script was dependent on the NFS mount firing first, and then it would go through and mount the nullfs filesystems.

It worked, but it didn’t have to.  As it turns out, the jail definitions in /etc/jail.conf can also have filesystems defined, which will be mounted at start up, and unmounted at shut down.  The syntax looks just like an /etc/fstab entry for any given mount.  For instance, for riddler, I added this line in its definition in the jail.conf file:

riddler {
	mount = "/opt/var/spool/mail /local/jails/riddler/var/mail nullfs rw 0 0";
}

Now whenever the riddler jail starts up, that null filesystem will automatically get mounted. When riddler is stopped, the filesystem will get unmounted.  I added similar lines for the other jails that needed null filesystems, shut all of the jails down, unmounted the null filesystems by hand, and then edited /etc/fstab to remove the entries.  Once that was done, I restarted the jails, and noted that each of the filesystems were properly remounted.  With that success, I removed the entry from /etc/rc.conf calling on my own rc script.

Networking Jails Using FreeBSD FIBs

One challenge with the public facing jails on joker is that they all, by default, have access to the same networks that joker does.  The host server is dual-homed on both the public and private LANs.  Even though the jails were all only defined with public IPs, they all had access to joker’s private interface.  Which means they could source a packet from their public IP, it would hit joker’s routing table, the destination would be seen as attached to the private LAN, and the packet would egress that way.  The host at the other end of the connection (eg: bane) would see the source IP as the jail’s public IP; when bane responded, it would send the packet to its default route, which would send the packet to the jail.

In other words: asymmetric routing.

How Does This Work?

If a jail such as riddler only has a public IP, how the hell is it able to send packets directly to bane without first hitting the external router?  The answer has to do with FreeBSD’s FIBs, or Forwarding Information Base. Think of it as the local routing table for the server.  The host, joker in this case, has a default FIB, or FIB 0.  In that FIB, its interfaces and routes are all defined.

By default, all jails defined on the host will have access to the same FIB.  That means that even with a public IP address, a jail will have access to the rest of the interfaces on the host.  A packet is sent from the jail to the FIB first, where the host (joker) decides which way to send it.  If the destination is bane on the private LAN (see: Figure 1), the packet will egress joker‘s private interface, not the public one.

It may seem a bit confusing that a packet with a public source IP could egress the private interface on joker, but it’s a perfectly legitimate action.

Protecting Private LAN Using PF

When I first encountered this little challenge, I didn’t know about multiple FIBs on FreeBSD.  So I set about writing pf rules on joker to specifically allow certain outbound from the jails to resources on my private LAN, and then I denied all outbound from them to the rest of the private LAN.  For instance, each needed access to the GIT configuration repositories on bane.  A few of them needed access to the MySQL server as well.  But they didn’t need to be able to get to anything else on bane, or any other resource on the private LAN.

I felt this was adequate, if not a bit clumsy.  I certainly would prefer that the jails have their own routing table, or at least some way to disconnect them from all of joker‘s interfaces.  And, as it turns out: there is.

Multiple FIBs

Through Google, I found a blog entry by Savagedlight that explained how to create multiple FIBs on a FreeBSD host, and then add jails to them.  I won’t copy everything he put in his entry, but rather review what I did specifically, and why.  Per his blog, the FIBs need to be defined at boot time in /boot/loader.conf with the net.fibs sysctl.  I set mine to 3: the default FIB for joker, the FIB for the public jails, and another FIB for private jails if I ever create any.  Following his lead, I also set the net.add_addr.allfibs to 0 so that each of joker’s interfaces would not be added to the other FIBs.  After a reboot, I had 3 FIBs.  But, there was a problem…

An IPv6 Bug With net.add_addr.allfibs

After the first reboot, I noticed a small problem.  At this point, I still hadn’t defined any specific routing entries for either FIBs 1 or 2 (FIBs start at 0 (the default)).  However, when performing this command:

setfib 1 netstat -nr

I noticed that FIB 1 had IPv6 knowledge of joker‘s 2 interfaces.  It didn’t have any IPv4 knowledge of them (or any other route), but it knew about the IPv6 interfaces.  Hm.  Wasn’t that sysctl I added to /boot/loader.conf supposed to prevent that from happening?

Yes, as it turns out, it was supposed to.  But apparently that was either forgotten or missed by the FreeBSD developers.  And I’m not the first to have noticed it, according to this bug entry.  This annoyed me for all of about a second; I just figured I’d work around it with the setfib command.

Setting Up Routes in the FIBs

Following Savagedlight’s suggestion, I created a /etc/rc.local and added:

#!/bin/sh

#
# Set up FIB 1 routing table for public jails
setfib 1 route add -net XX.YY.ZZ.0/24 -iface lagg0
setfib 1 route add default XX.YY.ZZ.222
setfib 1 route add -inet6 -net [Public IPv6 LAN]:: -prefixlen 64 -iface lagg0
setfib 1 route add -inet6 default [Public IPv6 default]

# These lagg1 IPv6 routes get added even though they shouldn't.  Delete them.
setfib 1 route delete -inet6 -net [Private IPv6 LAN] -prefixlen 64
setfib 1 route delete -inet6 -net fe80::\%lagg1 -prefixlen 64

# Set up FIB 2 routing table for private jails
setfib 2 route add -net 192.168.10.0/24 -iface lagg1
setfib 2 route add default 192.168.10.254
setfib 2 route add -inet6 -net [Private IPv6 LAN]:: -prefixlen 64 -iface lagg1
setfib 2 route add -inet6 default [Private IPv6 default]

# These lagg0 IPv6 routes get added even though they shouldn't.  Delete them.
setfib 2 route delete -inet6 -net [Public IPv6 LAN]:: -prefixlen 64
setfib 2 route delete -inet6 -net fe80::\%lagg0 -prefixlen 64

The basic idea was to set each FIB’s routing table entries to include the appropriate interface on joker, along with the default route. After which, delete the IPv6 entries that shouldn’t be in each FIB.

With any luck, future revisions of the kernel will have this little bug fixed up, and I won’t need to go through and delete the IPv6 entries.

Adding Jails to FIBs

This part was pretty easy.  Each of the jail definitions in /etc/jail.conf just needed another line added to them.  For example:

riddler {
	exec.fib=1;
}

Once done, a simple:

service jail restart

did the trick. All of the jails I’d previously defined were now using FIB 1, and had no direct access to any of joker‘s other interfaces.

A traceroute from riddler to bane, for instance, should first hit the router.  Previously it didn’t, it just egressed joker‘s lagg1 interface and hit bane.  But now:

riddler# traceroute -n 192.168.10.200
traceroute to 192.168.10.200 (192.168.10.200), 64 hops max, 40 byte packets
 1  10.0.0.1  0.202 ms  0.226 ms  0.086 ms
 2  10.0.0.1  0.124 ms !H  0.147 ms !H  0.110 ms !H

You can see the !H entries because traceroutes are blocked from the public jails to the private LAN. But the important bit is the first hop: it hit the router. Remember when I discussed converting my router to FreeBSD that I added proxy ARP into the mix, and I readdressed the public Ethernet interface on the router to 10.0.0.1/24. Well, that’s the first hop you see in the traceroute. Perfect.