This entry will be another fairly quick one. In the previous entry regarding the server, routing, etc, I described how I had to work around the fact that FRR didn’t honor FreeBSD’s assigning and deletion of IP addresses from interfaces. Specifically: I had the line
redistribute connected
in the server’s BGP configuration. That means any of the server’s interfaces with IP addresses on them would have those IP addresses added to BGP. And, it should mean that if an IP address is removed from one of those interfaces, the corresponding entry would be withdrawn from BGP. But as I found: that wasn’t happening. I’ll try to explain with a small amount of detail.
When a FreeBSD jail is started, you can configure it to add an IP address to the server’s physical interface. And when that jail is stopped, the IP address is removed from the interface. FRR was seeing the new IP address, but it was NOT seeing the IP address being deleted. So when the jail was stopped, FRR thought its corresponding IP address was still on the interface.
Here’s an example, using an RFC1918 IP address on my public interface.
arkham# ifconfig lagg0.50 alias 10.200.0.1/32 arkham# ifconfig lagg0.50 lagg0.50: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=600703<RXCSUM,TXCSUM,TSO4,TSO6,LRO,RXCSUM_IPV6,TXCSUM_IPV6> ether a0:36:9f:0d:ea:c0 inet 108.28.193.217 netmask 0xffffffff broadcast 108.28.193.217 inet 108.28.193.219 netmask 0xffffffff broadcast 108.28.193.219 inet 108.28.193.212 netmask 0xffffffff broadcast 108.28.193.212 inet 10.0.0.2 netmask 0xffffff00 broadcast 10.0.0.255 inet 108.28.193.210 netmask 0xffffffff broadcast 108.28.193.210 inet 108.28.193.213 netmask 0xffffffff broadcast 108.28.193.213 inet 10.200.0.1 netmask 0xffffffff broadcast 10.200.0.1 groups: vlan vlan: 50 vlanpcp: 0 parent interface: lagg0 fib: 1 media: Ethernet autoselect status: active nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
The key point here is the line in the interface’s output showing 10.200.0.1 is configured as an IP alias. If I check FRR’s vtysh shell:
arkham.private.lateapex.net# show int lagg0.50 Interface lagg0.50 is up, line protocol is up Link ups: 1 last: 2020/09/15 09:24:23.39 Link downs: 0 last: (never) vrf: default index 7 metric 1 mtu 1500 speed 10000 flags: <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> Type: Unknown HWaddr: a0:36:9f:0d:ea:c0 inet 10.0.0.2/24 inet 10.200.0.1/32 unnumbered inet 108.28.193.210/32 unnumbered inet 108.28.193.212/32 unnumbered inet 108.28.193.213/32 unnumbered inet 108.28.193.217/32 unnumbered inet 108.28.193.219/32 unnumbered Interface Type Other input packets 8192879, bytes 32838236838, dropped 0, multicast packets 99463 input errors 0 output packets 7065411, bytes 1194089187, multicast packets 139 output errors 0 collisions 0
The line that reads:
inet 10.200.0.1/32 unnumbered
means that FRR is seeing the new /32. Cool. If I had redistribute connected in BGP, my server would start announcing 10.200.0.1/32 to it. Now let me remove it from the OS:
arkham# ifconfig lagg0.50 inet 10.200.0.1/32 delete arkham# ifconfig lagg0.50 | grep 10.200.0.1 arkham#
Gone. It doesn’t show up in the ifconfig output. But how about in vtysh?
arkham.private.lateapex.net# show int lagg0.50 Interface lagg0.50 is up, line protocol is up Link ups: 1 last: 2020/09/15 09:24:23.39 Link downs: 0 last: (never) vrf: default index 7 metric 1 mtu 1500 speed 10000 flags: <UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> Type: Unknown HWaddr: a0:36:9f:0d:ea:c0 inet 10.0.0.2/24 inet 10.200.0.1/32 unnumbered inet 108.28.193.210/32 unnumbered inet 108.28.193.212/32 unnumbered inet 108.28.193.213/32 unnumbered inet 108.28.193.217/32 unnumbered inet 108.28.193.219/32 unnumbered Interface Type Other input packets 8193318, bytes 32838657020, dropped 0, multicast packets 99463 input errors 0 output packets 7065693, bytes 1194127091, multicast packets 139 output errors 0 collisions 0
It’s still there. Which means it’s still being announced to my router even though the IP is no longer on the interface.
I joined the FRR slack channel this weekend and started chatting with the developers there. Fortunately I know a few of them because they’re Cumulus guys. Donald Sharp volunteered to run with the fix, and he figured it out pretty quickly. He had a patch submitted to their git repository, awaiting approval. It turns out it was a few lines in one of the C files that needed changing, so I hand-jammed them in myself. Once I had FRR rebuilt on my server, it was doing exactly what I needed.
Now, with redistribute connected in the server’s BGP config, I can start and stop jails and have their IPs appear and disappear from the router. For instance, my jail madhatter has IP 108.28.193.213, and is currently seen by the upstream router:
lateapex-gw.lateapex.net# show ip route 108.28.193.213/32 Routing entry for 108.28.193.213/32 Known via "bgp", distance 20, metric 1, best Last update 02:36:53 ago * 10.0.0.2, via ix1, weight 1
If I tell arkham to stop that jail:
arkham# service jail stop madhatter Stopping jails: madhatter. arkham#
I can check the router again to see if the /32 is still there:
lateapex-gw.lateapex.net# show ip route 108.28.193.213/32 % Network not in table
Gone. Fire the jail back up and check the router:
arkham# service jail start madhatter Starting jails: madhatter.
lateapex-gw.lateapex.net# show ip route 108.28.193.213/32 Routing entry for 108.28.193.213/32 Known via "bgp", distance 20, metric 1, best Last update 00:00:16 ago * 10.0.0.2, via ix1, weight 1
It’s back.
Cool. Yes kids, routing on your servers is fun and useful. Do it!
