If you’ve been following my two sub-threads on this blog regarding FreeBSD jails and fighting spam, you know the status: I have a FreeBSD server called joker that has multiple jails on it. One jail is riddler, and it handles all outbound SMTP connections. But not inbound. Further, you know that I have a reasonably complex anti-spam system set up on joker using a combination of SpamAssassin along with Neal Horman’s spamilter. I won’t summarize them any further; if you need to catch up, read the previous entries.
I have all incoming SMTP connections going to joker, because of spamilter and its excellent companion: ipfwmtad. As previously explained, ipfwmtad takes input from spamilter to block incoming connections from known bad MTAs via FreeBSD’s ipfw or Linux’s iptables. FreeBSD jails can’t have ipfw running (without jumping through a lot of hoops). They must rely on their host for it. So, it means that spamilter and ipfwmtad have to run on joker, not riddler.
Separate Spamilter and Ipfwmtad
My goal with jails on FreeBSD was to spin up a jail per externally facing service such as SMTP, DNS, web servers, etc. However, with the aforementioned restriction, the host joker was still taking inbound SMTP. I figured the key to moving incoming into a jail was to separate the spamilter daemon from the ipfwmtad daemon. In other words: have spamilter running on one server (where sendmail is) and have ipfwmtad running on another.
More to the point: riddler would be running sendmail to allow incoming SMTP, and it would have spamilter running to provide anti-spam coverage. But joker would have to be running ipfwmtad because it’s the host, and the host has ipfw running. I’d also have to get SpamAssassin running on riddler along with procmail. That way all of the message filtering could be done within riddler, but only after the port 25 connection made it through joker‘s ipfw.
Fortunately, Neal’s code made it look like this could be an reasonably easy change. In the hndlrs.c file, spamilter makes a network socket connection to localhost, port 4739 to talk to ipfwmtad:
int sd = NetSockOpenTcpPeer(INADDR_LOOPBACK,4739);
It seemed to me that perhaps he could change that a bit to allow the target address to be configured by the end user, but default to 127.0.0.1. And if so, I could have spamilter on riddler talk to an ipfwmtad process running on joker. I sent a note off to Neal asking him to consider it, and within a short period of time…
New Spamilter Code and New Config Options
After Neal’s code change, the ipfwmtad process could be started and told to listen on a specific IP address, using the -I <address> option. Further, there’s a new config file: /usr/local/etc/spamilter/ipfwmtad.acl which defines which IP addresses (or networks) are allowed to connect to ipfwmtad. Finally, the /usr/local/etc/spamilter/spamilter.conf file got a few more entries in it to help it talk to a remote ipfwmtad.
Spamilter on Riddler
When I created the riddler jail, I had the forethought to do so with the FreeBSD source code included. Before snagging a copy of Neal’s latest code for riddler, I had to install the git package. From within the jail:
pkg install git
cd /usr/src
git clone https://github.com/nkhorman/spamilter.git
The rest of the install and configuration was identical to the previous one on joker, as outlined in the first spam fighting blog entry. Only this time, I had to make some slight changes to the /etc/rc.conf:
# Spamilter
spamilter_enable="YES"
ipfwmtad_enable="NO"
I specifically didn’t want or need ipfwmtad starting in the riddler jail, so I set its variable to “NO”. Then in /usr/local/etc/spamilter/spamilter.conf:
IpfwHost = joker's_IP
IpfwPort = 4739
IpfwUser = ipfw_user
IpfwPass = ipfw_pass
The key points here are that I’m using joker’s IP address, not 127.0.0.1. And in order to do that, I have to include a user and password in the config file. That user/pass combination should correspond to an existing entry in the target server’s /etc/passwd file. But that user doesn’t need to have any rights or access to anything. It’s purely for authentication. With that, I created a user on joker and set its password accordingly, but its home directory was set to /nonexistent, and its login shell to /usr/sbin/nologin.
At this point, having followed all of the installation and configuration steps outline in the first blog entry, I had sendmail talking to spamilter on riddler. I was almost ready to cut the MX record in my DNS entry over to riddler, but I had a few more things to do.
Ipfwmtad on Joker
The /etc/rc.conf file on joker had to change a little bit. Namely, I didn’t need spamilter, or even sendmail or SpamAssassin starting any longer. However, I did still need ipfwmtad to start:
# Sendmail
sendmail_enable="NO"
# Spam Assassin and Milter
spamd_enable="NO"
# Spamilter needed just for ipfwmtad
spamilter_enable="NO"
ipfwmtad_enable="YES"
ipfwmtad_flags="-I joker's_IP"
Neal’s code distribution didn’t include an rc file just for ipfwmtad. The spamilter rc file wouldn’t launch anything if it saw the “NO” spamilter_enable variable. So I quickly hacked together my own /usr/local/etc/rc.d/ipfwmtad:
#!/bin/sh
# PROVIDE: ipfwmtad
# REQUIRE: psql
# BEFORE: LOGIN
# KEYWORD: nojail shutdown
. /etc/rc.subr
name="ipfwmtad"
ipfwmtad_enable=${ipfwmtad_enable:-"NO"}
ipfwmtad_flags=${ipfwmtad_flags:-""}
extra_commands="status"
start_cmd="ipfwmtad_start"
stop_cmd="ipfwmtad_stop"
status_cmd="ipfwmtad_status"
proc_ipfwmtad="/usr/local/bin/ipfwmtad"
ipfwmtad_status()
{
rc=0
if [ "${ipfwmtad_enable}" = "YES" ]; then
pid_ipfwmtad="`check_process ${proc_ipfwmtad}`"
if [ -n "$pid_ipfwmtad" ]; then
echo "ipfwmtad is running as pid ${pid_ipfwmtad}"
else
echo "ipfwmtad is not running."
fi
fi
return $rc
}
ipfwmtad_start()
{
rc=1
if [ "${ipfwmtad_enable}" = "YES" ]; then
if [ -z "`check_process ${proc_ipfwmtad}`" ]; then
${proc_ipfwmtad} ${ipfwmtad_flags}
rc=0
fi
fi
return $rc
}
ipfwmtad_stop()
{
local pids
if [ "${ipfwmtad_enable}" = "YES" ]; then
pid_ipfwmtad="`check_process ${proc_ipfwmtad}`"
if [ ! -z "${pid_ipfwmtad}" ]; then
kill $sig_stop ${pid_ipfwmtad}
pids="${pids} ${pid_ipfwmtad}"
fi
fi
if [ -n "${pids}" ]; then
wait_for_pids ${pids}
return 0
fi
return 1
}
load_rc_config $name
run_rc_command "$1"
With that, ipfwmtad would now start on joker, latching on to joker‘s public IP. One last config file on joker needed to be edited so that riddler could connect to joker’s ipfwmtad. In the file /usr/local/etc/spamilter/ipfwmtad.acl:
allow | riddler_IPv4/32 |
allow | riddler_IPv6/128 |
SpamAssassin in Jail
The previous steps allowed me to split the spamilter and ipfwmtad up onto different machines, one of which being a jail. However, I still needed to get SpamAssassin running on riddler, which meant procmail as well. Further, I needed to make sure that riddler was writing the mail to a spool file that joker could read from because I intended to leave my Dovecot IMAP daemon on the latter.
Fortunately this was fairly easy. SpamAssassin initially complained about not being able to latch on to the loopback interface of the jail, but that was easily solved by telling it to use riddler’s public IP instead. In riddler’s /etc/rc.conf:
# Spam Assassin
spamd_enable="YES"
spamd_flags="-i -A riddler_IPv4 -A [riddler_IPv6]"
The last bit on riddler was to add my user’s home directory. Previously, I didn’t need it to have a home directory within the jail; the only reason I had an entry in riddler‘s /etc/passwd was for saslauthd to authenticate outbound SMTP. However, procmail needed to be able to read my ~/.procmailrc file, which meant I needed a home directory. I created one, copied the .procmailrc file over from my home directory on joker, and then started SpamAssassin.
One More Nullfs on Joker
Recall from one of the previous entries that I had a series of nullfs mounts on joker for the jails to access an NFS volume. My NFS server bane has, among its exported filesystem, joker‘s /var/mail spool directory:
joker# ls -la /var/mail
lrwxr-xr-x 1 root wheel 19 Nov 13 16:59 /var/mail@ -> /opt/var/spool/mail
joker# df /opt/var/spool/mail
Filesystem Size Used Avail Capacity Mounted on
bane.private.lateapex.net:/local/export 11T 629G 10T 6% /opt
I wanted the sendmail process on riddler to write incoming mail to the same spool directory. I needed another nullfs mount. From joker:
joker# mount -t nullfs /opt/var/spool/mail /local/jails/riddler/var/mail
And from within riddler:
riddler# df /var/mail
Filesystem 1K-blocks Used Avail Capacity Mounted on
/opt/var/spool/mail 11757442224 659306944 11098135280 6% [restricted]
Move MX Record to Riddler
After completing all of these steps, I was ready to move the MX record for my domains over to riddler. I made the appropriate changes on my authoritative DNS server, and within a short amount of time, riddler was taking all inbound. Spamilter and SpamAssassin were working fine together from within the jail, and the former was communicating with ipfwmtad on joker to block incoming SMTP connections from bad MTAs.
Of course, this gave me another idea: if I could move the ipfwmtad process to the jail’s host, what’s stopping me from moving it to a different machine altogether? Like, perhaps, a router running FreeBSD? …
Stay tuned.
