Final Linux to FreeBSD Conversion: The Router

This post will show up in both the FreeBSD section as well as the Network one.  It’s sort of a combination of the two.  I’ve been detailing my server conversions from Linux to FreeBSD on this blog, but I held off on the final device, and that’s the router.  It turned out to be a hell of a challenge that I intend to document here.

Services the Router Provides

My router has 3 interfaces on it:

1. igb0 – Facing my ISP
2. em1 – Facing my public LAN
3. em0 – Facing my Private LAN

It’s providing both LANs a natively-routed path between each other.  In other words, if something on the private LAN opens a connection to something on the public LAN, that connection is not translated by a NAT.  Likewise, the public LAN can route back to the private LAN, assuming its allowed through the filter.  The public LAN also routes natively outbound towards my ISP.  The only time a NAT happens is when something on the private LAN talked outward towards the ISP.

It’s providing DHCP services for the private LAN, some of which are statically configured (like the laptop I’m writing this on), others are dynamic.  It’s also providing internal DNS resolution; it can only be queried from the two aforementioned LANs.  Finally it has an IPv6 tunnel between itself and Hurricane Electric, because my ISP can’t (or won’t) provide native IPv6.

I’m not going to document getting each of the services running on the router because that will make this entry insanely long.  Setting up DNS, DHCP, and SLAAC on FreeBSD is very easy and there are plenty of online resources to help.

Verizon Sucks

I have Verizon’s FIOS, which is a fiber to the premises service for Internet connectivity.  Specifically, I have one of their business class setups, which among other things gives me access to statically assigned IP addresses.  This is a good thing: you’re actually reading this via one of them.

The challenge: The IP addressing scheme.  They have static IP packages that they offer, which sound really close to CIDR blocks, but aren’t.  What do I mean by that?  Well, they have a “5 static IP address” package.  They also have a “13 static IP address” package.  Not a 6 and 14 respectively, which means they aren’t a /29 and /28 respectively.  They’re 5 (or 13) static IPs that are part of a larger /24, with the default route set to Verizon’s .1 IP address in the broadcast domain.

What’s worse is that the static address packages are nearly on CIDR bit block boundaries.  My 13 IP addresses start at .210 and end at .222.  Note that a /28 would be .208/28, but I don’t own .208 or .209.  Someone else could be assigned them.

To call this “retarded” would be an insult to handicapped folks.

I can hear Verizon bleating now, “We’re trying to reserve IPv4 space!”  You have plenty of unused, untapped IPv4 space.  And the circuit point to points could be easily addressed out of RFC1918 space.  It would increase the routing tables slightly within Verizon, but who cares?  Your competitors at the MSOs all do it that way, why can’t you?

I know why: it’s because you suck.

Enough ranting.

Why is this architecture and addressing scheme a problem?  Simply put: it puts my public servers on the same open LAN as the rest of that entire /24 without any router-based access control lists.  While I’m happy to run host-based ACLs on most things facing the Internet, I’d also feel a lot more comfortable with a network-based security blanket covering them all.  Specially since some of the devices either don’t support host-based ACLs, or their respective solutions are beyond bad (think: Windows).  However, since my servers are all on the same L2/L3 broadcast domain as the rest of the /24, I can’t protect them with my own router.

Bridge Interface

Or can I?  The common solution is to do bridging on the local router.  In other words: combine the Ethernet interface that’s connected to the Verizon ONT and the Ethernet interface connected to your public VLAN together into an L2 bridge.  Both Linux and FreeBSD (and most other OSs I suspect) support packet filtering on L2 bridges.  So with my Linux router, I made a bridge out of the 2 interfaces, added a public IP to the new bridge interface, and began packet filtering on it.  Each public server still set its default router to Verizon’s .1.  But any packet through the bridge was inspected by Linux’s iptables (and ip6tables).

The Conversion

I studied up on how to create L2 bridges on FreeBSD and found that it’s fairly easy.  In my /etc/rc.conf:

ifconfig_igb0="up"
ifconfig_em1="up"
cloned_interfaces="bridge0 gif0"
ifconfig_bridge0="addm igb0 addm em1 inet XX.YY.ZZ.222/24"
ifconfig_bridge0_ipv6="inet6 IPv6_ADDR prefixlen 64 auto_linklocal"
defaultrouter="XX.YY.ZZ.1"

The private LAN interface was a lot easier as it was straight L3:

ifconfig_em0="up"
ifconfig_em0="inet 192.168.10.254/24"
ifconfig_em0_ipv6="inet6 IPv6_ADDR prefixlen 64"

IPv6 Tunnel With Hurricane Electric

The configuration for FreeBSD’s gif tunnel is pretty simple.  I added that to my /etc/rc.conf as well:

# Set up tunnel with HE for IPv6
ifconfig_gif0="tunnel XX.YY.ZZ.222 216.66.22.2"
ifconfig_gif0_ipv6="inet6 2001:470:7:9af::2 2001:470:7:9af::1 prefixlen 128"
ipv6_defaultrouter="2001:470:7:9af::1"

PF: The Bane of my Existence

Before doing the conversion to FreeBSD, I spent a lot of time combing through my Linux iptables (and ip6tables) rules on the router, and carefully converting them to over to FreeBSD’s pf format. I actually had a whole /etc/pf/pf.conf written, and kicked it into gear via /etc/rc.conf:

# Firewall ... pf to protect the network
pf_enable="YES"
pf_flags=""
pf_rules="/etc/pf/pf.conf"
pflog_enable="YES"
pflog_file="/var/log/pflog"

However, as I soon learned during this project: I hate me some pf. I really, really, exceptionally and truly hate pf.  I’ll summarize my rant on pf later in this document.  But for now: the first problem I ran into immediately was: nothing was passing properly through the bridge0 interface.  The public servers couldn’t get out to the Internet reliably, and the Internet couldn’t get to their public services.  I went through my pf.conf file repeatedly trying to see what I missed, and could find no issue with my logic.

If there’s a benefit to pf, it’s that it actively logs to a dump file that can be read with tcpdump:

tcpdump -n -e -ttt -i pflog0

As soon as I did that, I realized the problem: pf defaults to filtering on the L2 interfaces that make up a bridge as well as the bridge interface.  This is beyond stupid.  The command:

sysctl net.link.bridge.pfil_member=0

stops that idiotic behavior.  Once I applied that sysctl, packets began flowing properly for the public LAN.  To validate that my filtering logic on the bridge interface was correct, I tried connecting to one of my public servers from off-net on a service (port) that was not opened in pf.  Sure enough, I watched the incoming packets pass the igb0 interface, hit the bridge0 interface, and get dropped by pf.

There’s more to complain about pf as well as the rest of the filtering solutions within FreeBSD.  I’ll get to that later.

IPv6 GIF0 Tunnel: Not Staying Up

The next problem I noticed as that the tunnel between my router and HE wouldn’t stay consistently up.  I could reset the interface and then telnet -6 www.google.com 80 and get a connection.  At some random time later, it would stop working.  I never did figure out why this was the case, but I have some suspicions.  The IPv4 egress interface on my router is bridge0, and that’s also the tunnel termination on my end.  That same interface is also the IPv6 default route for my public servers, so it has an IPv6 address on it.  This was never a problem for Linux, but I think it’s what was causing FreeBSD to trip up with the tunnel.  It’s a theory that I haven’t explored any further, so don’t take that as gospel.

Poison ARP From Verizon

Refer back to the earlier part of this document describing the idiotic network setup that Verizon forces upon its business class customers.  Another problem that I quickly noticed was that the router wouldn’t reliably route from my private LAN to my public LAN.  I wanted to blame pf (again), but it turns out it wasn’t pf.  It had to do with bad ARP replies coming from Verizon’s router for my IPs!  The router would cache the appropriate ARP entry for joker (for instance) and then after the first connection to joker, the router would learn a new ARP entry for joker, pointed at the MAC address of Verizon’s router.

What the… ?  In the following example, deadshot is my laptop and is on my private LAN:

deadshot$ telnet xx.yy.zz.210 80
Trying xx.yy.zz.210...
Connected to joker.
Escape character is '^]'.

The first connection worked fine.  But when I hopped on the router and looked at its ARP cache entry for joker:

lateapex-gw# arp -n xx.yy.zz.210
? (xx.yy.zz.210) at 54:e0:32:be:cf:c1 on bridge0 expires in 1197 seconds [bridge]

That MAC address isn’t joker‘s, it’s the Verizon Juniper router at XX.YY.ZZ.1.  With that poisoned cache entry, deadshot could no longer get to joker:

deadshot$ telnet xx.yy.zz.210 80
Trying xx.yy.zz.210...

Static ARP Entries

My first attempt to address this problem was a hack, but it worked.  For each public IP, I added a static ARP entry on the router pointing to the appropriate MAC address.  For instance:

lateapex-gw# arp -S xx.yy.zz.210 0c:c4:7a:31:e3:d8
xx.yy.zz.210 (xx.yy.zz.210) deleted
lateapex-gw# arp -n xx.yy.zz.210
? (xx.yy.zz.210) at 0c:c4:7a:31:e3:d8 on bridge0 permanent [bridge]

That MAC address is on joker‘s lagg0 interface.  Sure enough, once that was in place, deadshot could repeatedly get to joker:

deadshot$ telnet xx.yy.zz.210 80
Trying xx.yy.zz.210...
Connected to joker.
Escape character is '^]'.
^]
telnet> Connection closed.
deadshot$ telnet xx.yy.zz.210 80
Trying xx.yy.zz.210...
Connected to joker.
Escape character is '^]'.
^]
telnet> Connection closed.

I wrote a simple rc script and put it in /usr/local/etc/rc.d.  It read in a config file in /usr/local/etc and set static ARPs according to what was in the file.  The only down side if this solution is that for any new machine on the public LAN, I’d have to edit that static file.  But it seemed like a small price to pay.

BINAT as a Solution

The static ARP entries worked, but the hack (static ARPs) on top of another hack (L2 bridging) kept bugging me.  Someone suggested I look at binat as a solution.  The NATing in pf can be set up to statically bidirectionally-NAT for any given entry.  Meaning that I would break the L2 bridge and reassign that XX.YY.ZZ.222 IP address to the Ethernet interface facing Verizon.  From there, I would add a new RFC1918 IP block to the Ethernet interface facing the public LAN, and readdress all of my public servers accordingly.

In order to use binat, I would have to add a new rule for each of my public IPs.  Something like this in /etc/pf/pf.conf:

vz_int = "igb0"
ext_int = "em1"
int_int = "em0"

host1_priv = "192.168.0.210"
host1_pub = "XX.YY.ZZ.210"
host2_priv = "192.168.0.211"
host2_pub= "XX.YY.ZZ.211"
# .
# .
# .
# until host_11
#
# NAT rules
pass on $vz_int from $host1_priv to any binat-to $host1_pub
pass on $vz_int from $host2_priv to any binat-to $host2_pub
# .
# .
# .
# until host_11

The problem with this solution is that it’ll never allow my private LAN hosts to talk to my public IPs.  The binat won’t work, because it’s only applied to the external interface: igb0.  If my laptop on the private LAN tried to telnet to joker’s public IP, port 80, it would never work.  That NAT wouldn’t be applied.

The work around for that is: split-horizon DNS.  I would have to change my internal DNS resolution for my public servers to answer with their 192.168.0 IPs instead of their public ones.  That way, when I tried to telnet to joker‘s port 80, it would go to the 192.168.0.210 address, not the XX.YY.ZZ.210 address.

This is another: hack on top of a hack.  I don’t like the confusion and obfuscation that NAT causes when it comes time to trouble shoot.  And split-horizon DNS is something I despise.  It’s a tool used by amateurs who don’t know how to set up networks and security properly.  No thanks.

Proxy ARP to the.. um.. Rescue

I’m actually ashamed to admit this.  As a “big network” guy, I despise proxy ARP and all of the challenges it brings to a network.  Specially troubleshooting said.  My boss back at AOL had a sign printed on his door with the words “Proxy ARP” inside a red circle and a slash through it.  No good.

However, in a small network such as what I have here, it might actually be applicable.  Someone on one of the FreeBSD mailing lists suggested I use it.  The thought had never even dawned on me to try it.  In the /etc/rc.conf file:

ifconfig_igb0="inet XX.YY.ZZ.222/24"
ifconfig_igb0_alias0="inet XX.YY.ZZ.221/24"
ifconfig_em1="inet 10.0.0.1/24"

# for Proxy ARPing because VZ sucks
static_routes="block1 block2 block3 block4"
route_block1="XX.YY.ZZ.210/31 -iface em1"
route_block2="XX.YY.ZZ.212/30 -iface em1"
route_block3="XX.YY.ZZ.216/30 -iface em1"
route_block4="XX.YY.ZZ.220/32 -iface em1"

# Set Proxy ARP
arpproxy_all="YES"

The basic idea is that the interface facing the public LAN gets a fake IP address that won’t be used for anything.  Every server on that LAN keeps its public IP and sets its default route to the router’s XX.YY.ZZ.222 interface, which is the one facing Verizon.  So that the router knows how to get to the public IPs on my LAN, a series of static routes are then put in pointing to the interface em1, not any specific gateway IP.  The last line causes the routing service in FreeBSD to automatically set the sysctl variable

net.link.ether.inet.proxyall: 1

This actually works, and works well.  It returns the router to a routing function, rather than an L2 bridge.  Go figure: using a broken network technology to address a broken network design.  Yay Verizon!

IPv6 Tunnel Stays Up

Once I enabled the proxy ARP and a form of real routing, the IPv6 tunnel with HE stayed up solid.  I don’t know if one had to do with the other, of if something else changed in the mean time.  But from that point forward, I was able to send and receive IPv6 traffic without hiccups.

Routing With FreeBSD: Summary

This whole process took me a lot longer than it should have.  I ultimately think that the issue is: FreeBSD is a great server OS, but it’s not great for routing.  I don’t mean routing via real protocols such as OSPF and/or BGP.  Rather what a user at home might need to do with it.  It clearly isn’t as mature as Linux is in this realm.

The bridge interface should have just ignored those ARP responses from Verizon.  I shouldn’t have had to search for a work-around for that problem.  I was using the exact, same configuration with my Linux router (L2 bridge), but it never once acknowledged any of those Verizon “poisonous” ARP responses.  It just ignored them.

And pf.  Pf is garbage as a routing filter.  It works very well as an end-point (ie: server) filtering mechanism, but it’s not well optimized for passing packets through a box.  Neither are ipf, nor ipfw.  I really think that one of the issues here is that the FreeBSD community needs to decide, as a group, which filtering mechanism they want and throw the rest of them away permanently.  That’s going to make some folks very angry, but they need to get over it.  Linux had a small collection of packet filtering solutions before IPTables emerged as the de facto.  FreeBSD needs something like that event.

Further, whichever solution is chosen needs to be modified heavily.  IPTables has 3 default tables it watches (ignoring the NAT table).  Inbound, Outbound, and FORWARDING!  That last one is very important for a router because it’s forwarding packets.  The admin of a router should not have to worry about inbound AND outbound packets for other devices.  The inbound and outbound should be targeted at the router itself.  Anything passing through the router should fall into a forwarding table.  It would make writing the filtering rules immensely easier, and still be as secure.

But that’s my opinion.  What do you think?