Email server migration and Using Dspam with Postfix and Procmail

Creative Commons License
Email server: migration and using Dspam with Postfix and Procmail by Isabelle Hurbain-Palatin is licensed under a Creative Commons Attribution-Share Alike 3.0 Unported License.
Based on a work at pasithee.fr.

Table of contents

Who is this for?

This document may be useful to people considering migrating a small, personal email server from one server to another. It may also be useful to those wanting to simply integrate Dspam in a Postfix+Procmail configuration.

This document does NOT provide a production-ready configuration for any site. This is MY personal configuration, probably flawed and insecure. Use it at your own risks; I am NOT liable for any damage and do provide this information with strictly NO warranty. You have been warned :) This documentation is also not extensive and probably not exactly adapted to your case anyway; your mileage may vary.

This document is NOT a Postfix/Procmail documentation either. I suppose you have a decent Postfix proficiency along this document.

Moreover, this article has been written on July 6th, 2008. I know I will probably let this unattended once I've written it; if you just stumbled upon it and 2008 is, well, a few years ago, please consider finding a more up-to-date documentation.

I can't promise I will answer to you if you contact me, but you can still try sending an email to ihurbain at this domain if you really wish to contact me.

One last short element: lines begining by $ are commands executed by regular users ; lines beginning by # are commands executed by the root user or an user with sudo priviledges.

Introduction

I've been using my own dedicated server for my email, websites, and other stuff for a bit more that two years now. While perfectly happy with the previous one's capacities, I recently felt that it was time to tidy the whole thing, and that the easy way to do this was to get another server, install it from scratch and move my data and services.

Plus, I've been quite unhappy with SpamAssassin those last months, finding I had waaay to many false negatives (probably my fault though, training my antispam is NOT one of my favorite activities) and heard that DSpam had a nice reputation - why not switch and test it?

So, this is the story of a migration between two servers (whose configuration is detailed in the next section), and the story of my dspam configuration. Configuring dspam is not that easy; I thought one more piece of documentation on the subject couldn't harm.

Server configuration

My new server is a nice machine in a nice datacenter somewhere in France. I installed Debian Etch 4.0 on it - I'm more an APT- than a RPM-girl. My previous server was on Ubuntu Server 8.04 LTS (previously 6.10 LTS) - I would have liked to reinstall an Ubuntu Server on the new one, but due to vserver problems (that's another story anyway, probably for another article), I ended up installing Debian. Anyway, on a server the difference between both distributions is not that sensible.

Nothing special on the distribution; I try to stick to the packages as much as I can to avoid upgrades hells. I just configured sudo to feel at ease with my Ubuntu reflexes.

The old setup was quite simple, but adapted to my needs:

The new configuration stays the same, except we use dspam instead of Spamassassin, and a new server.

The name resolution is made through the MX records of my domains. I actually have two MX records for each domain, with different priorities. The MX with the highest priority (which I call MX1) is pointing to my mail server. The MX with the lowest priority is pointing to a backup server that does NOT deliver emails. In case the main server (pointed by MX1) is unreacheable, the emails are sent to the second server, which holds them while MX1 is unreacheable, and transfers them back when the main server is up again. Note that I only change the highest priority records for MX for each of my domains; the secondary MX comes handy when I'm stopping Postfix on my main server. This way, I can stop Postfix without risking bouncing or loosing mails.

Installing and testing Postfix

Installation

The first step is to install and configure Postfix. Installation is easy:

# apt-get install postfix
  

As I was migrating a configuration, I didn't have much work to do on the configuration either. In fact, I copied my old configuration and only modified the myhostname field of my main.cf file and let the rest of my previous configuration "as is". My previous configuration is based on [1], a French translation and adaptation of [2].

Test

To test Postfix, the easiest way is to connect to the server from an outside computer using telnet on the 25 port, try to drop emails on it and see if it delivers/relays the emails it's supposed to deliver/relay and if it drops the emails it's not supposed to deliver/relay. Simply use this kind of syntax:

$ telnet xxx.xxx.xxx.xxx 25
Trying xxx.xxx.xxx.xxx...
Connected to xxx.xxx.xxx.xxx. 
Escape character is '^]'.
220 serveur ESMTP Postfix
HELO adresse_client
250 serveur
mail from:<foo@example.com>
250 2.1.0 Ok
rcpt to:<bar@example.com>
554 5.7.1 <bar@example.com>: Relay access denied
rcpt to:<user@mydomain.com>
250 2.1.5 Ok
data
foo bar test
.
250 2.0.0 Ok: queued as E9CFF11B3095
quit
221 2.0.0 Bye
Connection closed by foreign host.

Once the connection is established (after the line "Escape character is '^]'), the lines beginning by a number (220, 250, 554, 221) are status lines from the server; the others are commands you send to the server. The session in itself is quite classical: saying hello, saying "heya, I have a mail from this person", trying a first recipient (@example.com, which is NOT relayed or distributed and which is thus denied) and a second recipient (at a domain you actually configured on Postfix).

If you can, test all the domains you are delivering and/or relaying.

Once this is done, it's time to add some Procmail to the mix.

Adding Procmail

In my configuration, Procmail actually delivers the mail to the Maildir box of the users. At first, I use a very simple configuration. /etc/procmailrc only contains DEFAULT=$HOME/Maildir/ and ~/.procmailrc is only a few lines too:

MAILDIR=$HOME/Maildir
LOGFILE=$MAILDIR/procmail.log

#-------------- Catch-all -------------------
:0:
.Box.inbox/

This just defines the maildir, the logfile, catches all mail and put them into my inbox folder, that happens to be /Box/inbox.

Test again that everything is fine and that the emails are delivered.

Installing dspam

I chose to install dspam with a MySQL backend because iMil had some size problems with his sqlite backend. So, I installed a MySQL server :

# apt-get install mysql-server

and added a password to the root account of MySQL. Then, I installed dspam and its MySQL backend. Installing the MySQL before installing the dspam backend is a good idea because the dspam backend configuration tool on Debian can create the database and user that it needs.

Installing dspam and its backend are as easy as

apt-get install dspam libdspam7-drv-mysql

Some tweaking is needed on the Debian /etc/dspam/dspam.conf file, mainly those two lines:

StorageDriver /usr/lib/dspam/libmysql_drv.so
...
Opt out
instead of
StorageDriver /usr/lib/dspam/libhash_drv.so
...
Opt in

The first modification tells dspam to use the MySQL driver. The second line modifies how dspam is used. Opt in says that any user wanting to use dspam must be declared explicitely. Opt out says that any user wanting to NOT use dspam must be declared explicitely. As I'm a single user on this computer, I prefer Opt out to avoid adding more configuration files.

You must also modify the RUN value of /etc/default/dspam and give it the value yes so that the dspam server can be run.

After all this you should be ready to run /etc/init.d/dspam start to start dspam.

Training dspam

Dspam being a bayesian filter, it won't do anything good before it's trained. To train it, I took a subset of my previous (regular) mail, a subset of my old spam box (filtered by SpamAssassin) and fed it to dspam. First, I cleaned those mails of the previoux SpamAssassin headers. I used spamassassin -d, which is quite a long process, and may be suboptimal (in terms of speed), but I thought is was safer than trying to do it myself with sed/regexp/perl/whatever. So, in a directory with some (between 1000 and 3000) mail, I launched this for spam (and an equivalent for "ham", a.k.a. "non spam"):

for i in * ; do spamassassin -d < $i > ../spam-cleaned/$i; echo -n . ; done
  

Once you have some nice cleaned corpus of spam and ham (you must feed both spam and ham to dspam), give them to dspam:

dspam_train username spam-cleaned ham-cleaned

dspam is trained by user (because users can have different profiles of ham/spam); as I have only one user, myself, this is enough.

Connecting dspam with Procmail

Okay, we now have a setup with Postfix+Procmail on one side and Dspam on the other side. Time to connect all this! We will put everything in ~/.procmailrc. First, let's make a backup of everymail arriving, before filtering them - it's a safety net.

:0 c   
backup/

:0 c
! user@anotherprovider
  

This actually is a "double" safety net, as I copy my emails into backup AND forward them to another mailbox to another provider. Just in case.

Then, the real configuration of dspam comes:

# dspam spam filtering:
:0fw
| /usr/bin/dspam --stdout --deliver=spam,innocent --client --username user

:0: 
* ^X-DSPAM-Result: spam
.Spam-dspam/

--deliver=spam,innocent is used so that ALL the emails are delivered after the filtering. user is to be replaced with your user name.

With this configuration, every email is filtered through dspam and then "forwarded" to the rest of the procmailrc file, that distributes the emails in different folders depending on the results of Dspam. These lines are supposed to be between the backup rules and the sorting rules, including the catch-all at the end of the file.

Installing courier

And there, you see that you have forgotten to install an IMAP server :) I use courier-imap-ssl, and the default configuration is okay for me. Reconfiguring my email client, Claws Mail, showed me that everything was OK.

Migrating existing emails

First synchronization

This first synchronization minimizes the downtime during the final synchronization. We want to transfer the huge majority of the old emails to the new server so that only the emails that arrived during the whole migration process have to be transfered during the final, "all-postfixes-shutdown" synchronisation.

To gain some time, you can launch a first synchronization while you proceed to the previous operations. The first synchronization can be quite raw, just using rsync to make a rough copy of the current mail, and not bothering of what happens on the server (emails arriving, emails being read, etc.) The slashes are important and significant ; read man rsync for more information.

  rsync -a -z /home/user/Maildir user@newserver:/home/user/
  

Subsequent synchronization

Depending on the time needed for your whole setup and migration, and on the number of emails you receive, you can use rsync between the first and the last synchronization to make "inbetween" synchronizations (so that the final synchronization is relatively fast).

Last synchronization

The last synchronization is the most difficult and requires a bit of testing. DON'T try this unless you have a backup MX (MX2). I do NOT guarantee that you won't loose any email that way.

First, stop both Postfixes, on the old server and on the new server. We don't want emails to arrive during the synchronization - that would be risking email losses.

With bost Postfixes stopped (you may as well stop IMAP servers), make the last synchronization with rsync. This time, you can use the option --delete-after to ensure a clean copy.

Change your MX configuration so that MX1 now redirects to the new server. You still have MX2 running, so you shouldn't be too anxious about the fact that Postfix is still stopped.

But, the thing is, DNS propagation can take some time. How to ensure that the incoming emails are sent to the new server and not the old one? The solution holds in putting the old server as a relay to the new server.

Setting up the relay

Modify the configuration of the old server to relay emails to the new server. My husband found me that documentation, which I adapted and used to my needs. Basically, all domains (relayed and delivered locally) become relayed domains (with the option relay_domains). A new line is added, defining a file for transport_maps. Moreover, I temporarily setup inet_interfaces to localhost. This way, the incoming mail stays temporarily sent to the MX2, allowing local tests while not jeopardizing the delivery of regular emails. My main.cf looked like this (note that /etc/mailname is Debian-specific):

append_dot_mydomain = NO
myhostname = myservername
myorigin = /etc/mailname
relay_domains = domain1, domain2, domain3, etc.
mynetworks = 127.0.0.0/8
home_mailbox = Maildir/
alias_maps = hash:/etc/aliases
notify_classes = resource, software, protocol
inet_interfaces = localhost
mailbox_command = /usr/bin/procmail
recipient_delimiter = +
transport_maps = hash:/etc/postfix/transport

The /etc/postfix/transport file contains as many lines as domains to relay to the new server, in this syntax:

domain_to_relay smtp:[new_server_name]

The brackets ARE important, they tell Postfix to not bother with MX values and to just relay the emails to the server between the brackets. Launch postmap to hash the transport file:

# postmap /etc/postfix/transport
  

We cannnot just use a regular relaying as MX is not up to date on the first server - I do not manage my DNS servers myself!

Another solution would be to use, instead of the transport file and the transport_maps options, a simple relayhost option, probably like this (I did not test this):

relayhost = [new_server_name]

I got this information after having transfered all my emails and set up the other solution!

Testing the relay

Now you can test that the relay works correctly. Connect to the old server with telnet localhost 25. This will work as Postfix listens only to localhost. Send a few test mails, both ham and spam, to check that everything is relayed and distributed to the right mailbox.

Going production!

Once the tests are conclusive, do not forget to edit again the main.cf on the old server to change the inet_interfaces back to its previous value.

You now have the old server, still receiving emails from the world while the MX modification is propagating, but relaying them to the new server. The new server filters them with dspam and delivers them. Yay!

Adding Postfix training aliases

Depending on the accuracy of your training, you'll probably see some false negatives and false positives arriving respectively in your inbox and in your spambox. The easy way to tell dspam that it did a mistake is to add two aliases to /etc/aliases:

alias_for_spam: "|/usr/bin/dspam --process --class=spam --source=error --user user"
alias_for_ham: "|/usr/bin/dspam --process --class=innocent --source=error --user user"

Do not forget to launch newaliases. You may see some bounces the first times you send emails to the aliases. Note that the command is executed, on Debian at least, by nobody; for it to be able to modify the settings of a given user (the one specified with the --user option), you have to add it in the Trusted users section of /etc/dspam/dspam.conf (Trusted directive). Note that I have no idea about the security implications of this. As we say, "works for me" ;).

If you experience more bounces, look at /var/log/syslog. You may find the source of the errors there.

I've also been told about a solution with .forward files - a ~/.forward+spam, matching emails to user+spam@host, containing the dspam command. This actually is simpler and lighter than to define Postfix aliases.

Conclusion

I am aware that this article "mixes" two different problems, the migration and the installation of dspam. However, that's how I proceeded on my installation - hope this small blurb will be useful to someone! (Maybe me, on the next server migration :) ).

References