HOWTO Spam Filtering with Gentoo, Postfix, Amavis
From Gentoo Linux Wiki
| Installation • Kernel & Hardware • Networks • Portage • Software • System • X Server • Gaming • Non-x86 • Emulators • Misc |
Contents |
[edit] Introduction
The tutorial has been taken from the forums.
[edit] Install Gentoo
To begin, I'm going to assume you have Gentoo up & running on a box. I have been using gentoo-sources with no problems on my spam filter, but I'm sure if you used a different kernel you could squeeze more performance out of it.
For reference, here's the machine I set this all up on:
Code:
Single Processor Intel Pentium III 450MHz(Katmai) 128MB RAM RAID 1 on SCSI 8GB Drives USE_FLAGS = "-X"
For safety's sake, I think you should have some type of RAID on this system. If you need help, this is an excellent 'HOWTO' on software RAID 1 on Gentoo (its what I used) http://forums.gentoo.org/viewtopic.php?t=8813&highlight=software+raid
I'm currently doing spam filtering for 3,700 Exchange mailboxes, and catching between 500 - 1000 spam messages per day. And to be honest, the machine listed above doesn't even break a sweat.
[edit] Emerge postfix, amavisd-new
Ok, once your Gentoo box is booting up, emerge the following:
Code:
emerge postfix emerge amavisd-new emerge spamassassin
(Spamassassin is no longer a dependency for amavisd so you have to emerge it manually)
[edit] Configure Postfix
Now you need to configure Postfix to work as a relay to the Exchange server only. You don't want mail being sent from this box. I'm taking the following information directly from Scott Henderson's excellent HOWTO ( http://www.geocities.com/scottlhenderson/spamfilter.html - moved to http://www.freespamfilter.org/) Please note that I am not running postfix chroot, although you could & Scott gives instructions on doing this. If you experience local delivery problems, this can be due to chroot, so try setting chroot to 'n'. Scott also covers a lot of other things, like sending root mail to another mailbox, improving logging performance, etc. so I recommend you read through his site. I'm just going to cover the basics here.
Edit the file /etc/postfix/master.cf and add this at the bottom
Code:
smtp-amavis unix - - y - 4 smtp -o smtp_data_done_timeout=1200 -o disable_dns_lookups=yes -o smtp_send_xforward_command=yes 127.0.0.1:10025 inet n - y - - smtpd -o content_filter= -o local_recipient_maps= -o relay_recipient_maps= -o smtpd_restriction_classes= -o smtpd_helo_restrictions= -o smtpd_sender_restrictions= -o smtpd_recipient_restrictions=permit_mynetworks,reject -o mynetworks=127.0.0.0/8 -o strict_rfc821_envelopes=yes
Make sure to get tabs between the - y - parts...
This sets up postfix to forward mail to this machine on port 10025, the port Amavisd listens on. [Edit: amavisd is listening on 10024. The second "instance" of postfix is listening on port 10025. The second instance of postfix is not needed if you are just relaying to another box (MS Exchange), but is used for a all-in-one filtering mail server.] [Edit: You should add -o smtp_send_xforward_command=yes to amavis, otherwise it won't be able to blacklist/whitelist MYNETS in amavis! Also you should run 4 threads in parallel, two were too few for me.] jmelika: Thanks for the tip added, but could you please modify the config accordingly? I don't want to mess it up trying to add the additional threads you recommended.
Next, edit the file /etc/postfix/main.cf and add the following at the bottom. Note that I've replaced my specific info with a filler.
Code:
myorigin = my.network.com myhostname = spamfilter.my.network.com mydestination = my.network.com mynetworks = x.x.x.x/y biff = no smtpd_banner = SMTPD_BANNER_HERE message_size_limit = 25000000 local_transport = error:no local mail delivery local_recipient_maps = transport_maps = hash:/etc/postfix/transport smtpd_helo_required = yes #smtpd_helo_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain #smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient content_filter = smtp-amavis:[127.0.0.1]:10024
The line mynetworks defines what networks you will trust and relay mail for. Its important to put exactly the machines that you will be sending mail to!! I usually put my local subnet & the IP of the server. BTW, it takes the list in network/subnet style, so for a single machine, it would be 10.0.0.1/32, for a class B it would be 10.1.0.0/16
Of interest are the following lines
Code:
smtpd_helo_required = yes #smtpd_helo_restrictions = reject_non_fqdn_sender, reject_unknown_sender_domain #smtpd_recipient_restrictions = permit_mynetworks, reject_unauth_destination, reject_non_fqdn_recipient content_filter = smtp-amavis:[127.0.0.1]:10024
the lines starting with 'smtpd' are some anti-spam settings for postfix. I have had problems with them, so I usually comment them out, but they are there should you want to use them. I've also had problems with postfix doing blackhole lookups (Internet databases of spam senders), so I usually don't do with postfix. All of this is covered in Scott's site listed above.
Secondly, you may want to add these lines:
Code:
header_checks = pcre:/etc/postfix/pcre-header.cf body_checks = pcre:/etc/postfix/pcre-body.cf
This will allow you to put regular expressions in those files to test against the incomming mail headers and bodys. IMPORTANT If you add these lines, you need to at least have files (even blank) in the /etc/postfix dir. If anyone wants, send me a PM & I'll send you my copies of the files. I have built up quite a bit of stuff in them.
Next, you may want to edit the file /etc/postfix/access. Here you can put ip ranges, senders and domains to be rejected or accepted by postfix. Personally, I do not use this file, I just use the pcre-header & body files. If you do edit the access file, you need to run:
Code:
postmap /etc/postfix/access
after editing it. Otherwise postfix will not recognize your changes.
Ok, that should be the end of configuring postfix. You can start the postfix service now & check it by telneting into localhost on port 25. Got a response? So far, so good.
If you do change any postfix configuration files, esp. if you decide to use the pcre-header & body approach like I do, when you are done editing, just run
Code:
postfix reload
to have the changes recognized.
[edit] Configure Amavisd-new
Ok, now amavisd-new is kinda like this glue that holds together all these different mail processing programs. Its a pretty neat idea, and once you understand that amavisd-new just coordinates these programs, its easy to work with. A stock install of amavisd-new gives you two great programs, clamd & SpamAssassin. Clamd will handle anti-virus scanning, and SpamAssassin will handle the spam filtering.
First, make sure you have the correct permissions set, otherwise amavisd-new will not start.
Code:
chown amavis:amavis /var/amavis/db chown amavis:amavis /var/amavis/quarantine chown amavis:amavis /var/amavis/tmp
Next, edit /etc/amavisd.conf & make the following changes
Code:
Locate $mydomain = 'example.com' and change to your domain Check that $daemon_user and $daemon_group are set to 'amavis' (done by the Gentoo ebuild I believe) DO NOT remove the # from @bypass_virus_checks_acl!! You want to do virus scanning! NOTE: in recent amavis-new.conf files, the @bypass_spam_checks_maps is uncommented (approx line 167), you need to COMMENT this line for spam checking to work!!! FURTHERMORE, on approx line 1074, you MUST COMMENT out @bypass_spam_checks_maps!!! Otherwise spam checking will again be disabled.
Remove the # from in front of $warnspamsender = 1; Change $mailfrom_notify_ lines to point to "your.admin.mailbox\@your.domain.com"; Locate $spam_quarantine_to = 'spam-quarantine'; and add a # to the beginning. Go down one line & REMOVE the # from $spam_quarantine_to = "your.admin.mailbox\@your.domain.com";
Ok, that should get amavisd-new passing mail off to the anti-virus & spam filters correctly. Now, here are some important tips for amavisd-new:
Many of the settings for SpamAssassin are taken from amavisd.conf! So, you can edit the SpamAssassin conf file all you want, but the changes won't work... so, here's what you have to set in amavisd-new
Code:
Find the section # SpamAssassin settings Notice the lines $sa_tag_level_deflt, $sa_tag2_level_deflt, $sa_kill_level_deflt. These are where you set the scores that SpamAssassin gives to a mail to mark it as spam. I have mine set to: $sa_tag_level_deflt = 0.0; $sa_tag2_level_deflt = 5.0; $sa_kill_level_deflt = $sa_tag2_level_deflt; so, a score of 5 marks it as spam & kills the mail.
NOTE, $sa_kill_level_deflt does not need to equal $sa_tag2_level_deflt, in fact tagging at 5.0 and killing at 8.0 is a good place to start, and monitor spam for a bit.
Its important to put the scoring information in the amavisd.conf file, as it will be ignored if put in the SpamAssassin conf file.
Virus Scanning
To be honest, this works out of the box. Clamd sets up a cron job to check for updates daily, and I've never had a problem. Clamd logs to /var/log. Great app & it catches a TON of viruses for us.
First we have to install it...
emerge clamav
Recall, you may want to update the virus definitions regularly... freshclam will update the virus definitions, you may want to add it to root (or the amavis users) crontab. NOTE, the clamd team recommends not updating at any interval of 10mins after the hour, as the servers are loaded at that time. Here is an example crontab.
# run freshclam 33mins after the hour, every other hour
33 */2 * * * /usr/bin/freshclam --quiet
Remember to put a MAILTO other than root if this is roots crontab. root should never receive mail.
[edit] Configure SpamAssassin
As I mentioned above, the most important setting for SpamAssassin, the scoring of mails, is controlled through /etc/amavisd.conf. All other configuration for SpamAssassin is done in /etc/mail/spamassassin/local.cf Here's a few settings you can play with
Code:
bayes_auto_learn 1 bayes_auto_learn_threshold_nonspam 1 bayes_auto_learn_threshold_spam 14.00
These settings manipulate the bayes learning feature of SpamAssassin. I would recommend setting the threshold to 'learn' high, as otherwise you will get a lot of false positives. A spam score of 14 seems to do a good job for me. Lower & you will see things like many aol.com emails getting marked with a high Bayes score.
Also, local.cf is used to put SpamAssassin rules into. I have many rules that I use. PM me & I'll send them to you.
Final Setup
You should now be ready to start filtering mail! You need to add some programs to your default runlevel
Code:
rc-update add postfix default rc-update add amavisd default rc-update add clamd default
Start these services /etc/init.d/<service> start & start testing it out! You can set your SMTP server to the spam filter machine in your local email client & start sending emails. See if they get through...
[edit] Final Thoughts
There is lots and lots of additional stuff you can do. You can play around with the headers that are added to the emails, you can quarantene differently, set up different notifications, and on & on. The amavisd.conf file is very well commented, so read through it.
Notice that I didn't touch the Exchange server at all. If you wanted to get advanced, you could do LDAP lookups to verify that the recipient exists. This would be overkill in my case, and probably too much overhead.
Fixer sent me the following link & said he had validation of accounts against Exchange up & working within 2 hours. If you are like his net admin (and like most of us using Exchange) and are quite paranoid about the stability of Exchange, this would help reduce the load on Exchange.
http://www.unixwiz.net/techtips/postfix-exchange-users.html
Be careful in setting the score limits. Its better for users to get some spam (& tell you about it) than it is to get a bunch of false positives. Users will understand some spam getting through, but will not like valid messages getting bounced.
Personally, I've leaned towards putting spam senders into the pcre files I mentioned above. If I know there is some unique identifier in an email (usually a sender address or web link), I'll add it to the postfix. This has two advantages; it reduces load on the server (amavisd never sees the mail), and doesn't fill up the Exchange mailbox you set up with stuff you know is spam. On the other hand, there is no notification.. just a SMTP reject.
You may want to install logrotate to help manage logs. They get big fast. Also, I use mtail to color my logs when I tail them. A simple grep statement & mtail can give you a good visual of how many mails are getting bounced, passed, etc.
If you want configuration files, PM me.
[edit] Resources
Scott Henderson's site - where I learned most of this stuff
Postfix - difficult documentation tho...
Amavisd-new - just read the config file
SpamAssassin Config Settings - remember, the score settings need to be in amavisd.conf
