HOWTO Spam Filtering with DSPAM and Postfix

From Gentoo Linux Wiki

[edit] Intro

This is a HOWTO for implementing DSPAM as a content filter with Postfix. This is not the only way, but it works for me. If you want to try more advanced things like per-user training and preferences, merged groups and all that clever stuff then this can still provide the starting point. I'll assume a working Postfix and MySQL. There are plenty of docs to install those.

[edit] Who would want to?

Target audience would be a person already running Postfix, interested in anti-spam and maybe already running SpamAssassin (SA). I had used SA for years and bolted on Pyzor, DCC and Razor2 to bolster its performance. I've also refined Postfix to keep much of the junk out.

[edit] Why use DSPAM

Well the DSPAM website will probably give some clues.

I've run DSPAM on half a dozen servers for a couple of years. It's fast, accurate and not many False Positives.

[edit] Rough guide

Have working Postfix. Emerge DSPAM and config. Config Postfix to use DSPAM as a content_filter on incoming email only. Emerge dspam-web and config. Enjoy.

[edit] Let's begin

DSPAM is no longer masked but you'll currently need ~x86 to get version 3.8.0. We also add some use flags, so

Code:

 echo "mail-filter/dspam ~x86" >> /etc/portage/package.keywords
 echo "www-apps/dspam-web ~x86" >> /etc/portage/package.keywords
 echo "mail-filter/dspam mysql logrotate clamav daemon" >> /etc/portage/package.use
 emerge dspam
 

Code:

emerge --config =dspam-3.8.0-r1

Make sure this matches your version. You will need to type in the MySQL root password twice and then select 2 for speed when setting up the database. Option 1 is space and pick this if you know better :-) And then MySQL root password again.

Here's how that looks

Code:

hostname ~ # emerge --config =dspam-3.8.0-r1

Configuring pkg...

Which backend do you want to configure? (available backends are mysql) mysql Please enter your administrative MySQL account (default root):

* When prompted for a password, please enter your MySQL root password
*
* Creating DSPAM MySQL database "dspam"

Enter password: Enter password:

* Creating DSPAM MySQL database for virtual-users users
*   Please select what kind of virtual_uids table you like to use.
*     [1] Virtual users added automatically (use this if this server is the primary MX)
*     [2] Virtual users added manually (use it if this server is a secondary MX)
*
 Press 1 or 2 on the keyboard to select table type

Enter password: Enter password: * Creating DSPAM MySQL user "dspam"

Note that if you want (or need) to change the MySQL password for the dspam user, you can login to mysql as root and enter the command

SET PASSWORD FOR 'dspam'@'localhost' = PASSWORD('passwordhere');

Add a new user called filter for DSPAM to process under. I tried this with user dspam but it fails when you try the web interface due to Apache not liking the low UID when suexec the CGI scripts.

useradd -g users -m -s /bin/bash -G dspam filter

will do the job.

Add user filter as a trusted user in

# Trusted Users: Only the users specified below will be allowed to perform
# administrative functions in DSPAM such as setting the active user and
# accessing tools. All other users attempting to run DSPAM will be restricted;
# their uids will be forced to match the active username and they will not be
# able to specify delivery agent privileges or use tools.
#
Trust root
Trust mail
Trust mailnull
Trust smmsp
Trust daemon
#Trust nobody
#Trust majordomo
Trust filter

I used to retrain by forwarding spam to a special account, but I gave that up as most people couldn't be bothered. I use IMAP and create a sub-folder called Spam for them to drop the junk into. I run a script on a cronjob to retrain them and delete.

There are other ways of achieving the same thing.

Have a look at the config file /etc/mail/dspam/dspam.conf. As we are using as a content_filter we don't need any of the Trusted/UntrustedDeliveryAgent stuff.

We do need.

DeliveryHost        127.0.0.1
DeliveryPort        10025
DeliveryIdent       localhost
DeliveryProto       SMTP

This makes DSPAM stick the scanned email into Postfix using SMTP on port 10025 of local machine. Only change these if you know what you are doing.

Further down the file.

 Preference "spamAction=quarantine"
 Preference "signatureLocation=message"  # can be 'message' or 'headers'
 Preference "showFactors=off" # changed from on
 ServerPID /var/run/dspam/dspam.pid
 ServerMode auto
 ServerParameters        "--user filter --deliver=innocent"
 ServerDomainSocketPath  "/var/run/dspam/dspam.sock" # socket to receive email from Postfix
 

We add DSPAM to one of the Postfix config files /etc/postfix/master.cf thus

dspam     unix  -       -       n       -       10      lmtp

We create a new SMTP daemon listening on port 10025 (remember that from earlier?). In /etc/postfix/master.cf

 127.0.0.1:10025 inet    n       -       n       -       -       smtpd
  -o smtpd_authorized_xforward_hosts=127.0.0.0/8
  -o smtpd_client_restrictions=
  -o smtpd_helo_restrictions=
  -o smtpd_sender_restrictions=
  -o smtpd_recipient_restrictions=permit_mynetworks,reject
  -o mynetworks=127.0.0.0/8
  -o receive_override_options=no_unknown_recipient_checks
 

[edit] Are we there yet?

At this point we have a configured, but not running, DSPAM. /etc/init.d/dspam start should fire it up and have it listening on the socket mentioned above. If we issue

Code:

/etc/init.d/postfix reload

we will have Postfix listening on port 25 for usual email and 10025 for filtered email. Run netstat -tunlp if you want to verify.

Code:

netstat -tunlp | grep master
tcp        0      0 127.0.0.1:10025         0.0.0.0:*               LISTEN      30772/master
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      30772/master

Postfix has this daft idea that content_filter should be run on ALL email. I don't want to scan my outgoing email for spam. So I call the content_filter thusly.

Edit /etc/postfix/main.cf

 smtpd_recipient_restrictions =
        permit_mynetworks
        reject_unauth_destination
        check_recipient_access pcre:/etc/postfix/dspam_incoming
        permit
 

You may have other entries and you'll want the check_recipient_access bit after permit_mynetworks and after any other reject lines.

/etc/postfix/dspam_incoming contains

/./     FILTER dspam:unix:/var/run/dspam/dspam.sock

This will match on _anything_ and cause the action FILTER. It will use LMTP (Local Message Transport Protocol) to stick the message into DSPAM's socket. Ouch.

Reload Postfix again and you are ready.

Start ClamAV and be sure that ClamAV MySQL and DSPAM are run at boot with

Code:

rc-update add clamd default
rc-update add dspam default
rc-update add mysql default

[edit] What have we done so far?

We've installed MySQL to support DSPAM, installed DSPAM and ClamAV. We have configured MySQL, DSPAM and Postfix to play nicely.

[edit] How does it work?

Postfix accepts email on port 25 as usual. If it's from your LAN, or any mynetworks location, it just carries on as usual. If it's not it gets passed to DSPAM via a socket. DSPAM eats the spam and passes the rest back to Postfix on Port 10025. Then it follows the usual route through to the mailbox.

[edit] Does it really eat the spam?

No, I lied. It quarantines it in a mailbox file /var/spool/dspam/data/local/dspam/dspam.mbox . If you don't like this, you can change DSPAM to deliver. All the email gets tagged in the headers.

Code:

X-DSPAM-Result: Innocent
X-DSPAM-Result: Spam

So you can filter it.

I open the quarantine using Mutt

Code:

mutt -f /var/spool/dspam/data/local/dspam/dspam.mbox

I also use the web interface for releasing email and some training. For the moment send any spam that gets through to you to spam@your_domain and it will re-train this error as spam for the future.

[edit] Web Interface?

The web interface is great. It presents the content of the quarantine, so you can release any false positives (quite rare in my experience). You can retrain messages via the history page and check accuracy. Under this installation method one user called filter will handle the email for all users. Any user can retrain a message by emailing to spam@example.com, so they don't all need to get to the web interface.

[edit] Installing the Web Interface

This drove me slightly mad a few times but I do have a workable method now. I'll assume you have a working Apache2 install for now, but that is one of the easier servers to install. None of my installs are Internet facing (not port 80 anyway) and the usual caveats apply if you chose to do differently.

emerge dspam-web

You'll see this

* The CGIs need to be executed as group dspam in order to write
* to the dspam data directory. You will need to configure apache
* manually to do this. Another option is to add the user apache
* to the dspam group. You can do this automatically by running:

* emerge --config dspam-web-3.6.4

* This app requires basic auth in order to operate properly.
* You will need to add dspam users to the .htpasswd file or
* configure a different authentication mechanism for the user
* accounts.

This fails if you don't use the = sign, so instead run

emerge --config =dspam-web-3.6.4

This doesn't really install, so much as place all the files on your system somewhere. More on this later.

[edit] Configuring Apache2

I used the user directory function for this, but there are sure to be a few other methods.

Uncomment the following and add the extra lines as follows:-

<Directory /home/*/public_html/cgi-bin>
   Options ExecCGI
   AuthType Basic
   AuthName "dspam"
   Require valid-user
   AuthFileName /var/www/localhost/password
   #Use AuthUserFile instead of AuthFileName for apache 2.0.55 (that was my version) and above
   SetHandler cgi-script
</Directory>

enable by adding -D USERDIR to

Then create the web directory for the user filter.

mkdir ~filter/public_html

Get the files 'installed' by dspam-web into the right place

cp -R /usr/share/webapps/dspam-web/3.6.4/hostroot/* ~filter/public_html/

And a couple more

cp /usr/share/webapps/dspam-web/3.6.4/htdocs/* /var/www/localhost/htdocs/

Then make sure ownership is ok.

chown -R filter:users ~filter/public_html/

At this point, after an Apache restart, you should be able to navigate to http://hostname/~filter/cgi-bin/dspam.cgi . You won't get a warm welcome but you should get something. The image and the stylesheet fail to load until you change the web root from "/" to " ".

while you are in there edit the default domain

[edit] Identity Crisis

At this point DSPAM web UI needs to know who you are in order to present information. You can use basic auth and .htaccess .htpasswd.

htpasswd2 -c /var/www/localhost/password filter
New password:
Re-type new password:
Adding password for user filter

This creates the file containing the password.

When you navigate to http://hostname/~filter/cgi-bin/dspam.cgi you will be prompted for a username and password. This is all exchanges in the clear so consider SSL or other encryption if your network is not safe.

[edit] One More Thing

There are a few loose ends to tie up. Without all this it wouldn't need masking after all.

mkdir /var/spool/dspam/log/

chown -R dspam:dspam /var/spool/dspam/

chmod 4511 /usr/bin/dspam

Note:- you will have to do this command again if you re-emarge or update dspam

That should do it.

[edit] What next?

I'll cover some more advanced things and add them on as time permits.

Resist the urge to start doing anything other than training it as you go along. You can get together some old email or even download a corpus of spam and ham (the opposite of spam) but don't. Accuracy will improve with training over time. No shortcuts.

Post on the Gentoo forum if you get stuck.