Een mailserver op Ubuntu 12.04: Postfix, Dovecot, MySQL

Deze post beschrijft de installatie van een mailserver op Ubuntu 12.04.
De volgende pakketten ga ik installeren:

• Postfix 2.9.1
• Dovecot 2.0.19
• MySQL 5.5.22, with anti-spam packages in the form of
•   amavisd-new 2.6.5,
•   Clam AntiVirus 0.97.3,
•   SpamAssassin 3.3.2, and
•   Postgrey 1.3.4.
• Munin
• Postfix Admin 2.3.6

1) Introductie

Building a Linux mailserver from scratch to your own liking is a painful process unless you happen to be one of the few folk who do that day in and day out – there’s no way around that fact. A mailserver generally consists of a range of different packages that separately handle SMTP, POP/IMAP, local storage of mail, and spam-related tasks: they must all talk to one another correctly, all have small novels in place of configuration documentation, and there is no one obvious best practice for how users are managed, how to store user data, or how to glue the various different components together. There are any number of different viable setups for moving mail between Postfix and Dovecot, for example. Further, the whole assembly tends to be unforgiving on matters such as permissions, choice of users for specific processes, and tiny errors in esoteric configuration files. Unless you know what you are doing the end result will likely be either insecure, non-functional, or otherwise misconfigured.

There are a number of fairly up to date recipes for creating mailservers out there; one of the better ones is an Ubuntu recipe by Ivar Abrahamsen, which gives you Postfix for SMTP, Courier for IMAP/POP, MySQL to store account information, virtual user mail directories, and an array of anti-spam tools that are highly effective when working in concert. It’s a good set of documents, as the author places an emphasis on producing a secure mailserver as the end result.

There are also a great many partial recipes and out of date guides that are frankly more of a hindrance than a help – especially when it comes to Dovecot, which has changed greatly between its 1.* and 2.* versions. The configuration is completely different, and so are many of the administrative and tool binaries.

I’ve used Abrahamsen’s guide as a basis for my mail servers running on AWS for a few years now. Upgrade time always rolls around eventually, however, and this time I decided to move to a new setup to mark the release of Ubuntu 12.04: swap out Courier for Dovecot and try out a web front-end for managing mail users. Finding a good all-in-one-place guide was a challenge, however – hence this document.

2) Outlining the Goal

The end result of following this guide will be a secure mail server for your domain equipped with the following software packages:

• Postfix: sends and receives mail via the SMTP protocol. It will only relay mail on to other mailservers if the mail is sent by an authenticated user, but anyone can send mail to this server for local delivery.
• Dovecot: a POP and IMAP server that manages local mail directories and allows users to log in and download their mail. It also handles user authentication.
• Postgrey: greylists incoming mail, requiring unfamiliar deliverers to wait for a while and then resend. This is one of the better tools for cutting down on spam.
• amavisd-new: a manager for organizing various antivirus and spam checking content filters.
• Clam AntiVirus: a virus detection suite.
• SpamAssassin: for sniffing out spam in emails.
• Postfix Admin: a web front end for administering mail users and domains.
• Horde Groupware Webmail Edition: a webmail interface for users.

The server will accept plain text or encrypted SMTP and POP/IMAP connections at the standard ports, but will not allow user authentication without encryption. It will pass through a minimal set of mail headers for mail sent by local users, removing identifying information from the original sender’s mail software.

3) Fire up an Ubuntu 12.04 AWS Instance with a Suitable Security Group

Start up an Elastic Block Store (EBS) server instance – at the time of writing, Ubuntu 12.04 is one of the options right there in the quick start menu for launching a new instance. Mail servers don’t generally have to be all that big if you aren’t in the business of email; a micro instance has served me just fine for a fairly well trafficked web site with a mailing list of thousands, for example. That said, the server produced by following this guide runs at close to 80% memory utilization for a micro instance when operating unloaded – a sudden blizzard of unexpected web traffic would probably cause issues. So adjust your expectations accordingly.

Firewall settings in AWS are managed through assignment of Security Groups. You’ll probably want to create one before starting the server. The Security Group should allow inbound TCP traffic from any IP address to these ports: 25 (SMTP), 80 (HTTP), 110 (POP3), 143 (IMAP), 443 (HTTPS), 465 (SMTPS), 993 (IMAPS), and 995 (POP3S). That is in addition to whatever rules you might have for SSH access over port 22 – it is not a good idea to leave that open to the world, so lock it down to the IP address ranges you use.

In fact it is a good idea to restrict all inbound traffic to the server to your IP address ranges while you are building it. You can adjust the rules to allow traffic from the rest of the world after you’re certain that everything is secure and shipshape.

4) Some Basic Configuration

The baseline Ubuntu instance is lacking in near every package you might need, so you are building from fairly close to scratch. You’ll log in as the “ubuntu” user and then switch to root; most of what you need to do requires root access:

You must set up an Elastic IP to give your server a permanent IP address. By default, an AWS instance will have its own strange-looking hostname, so changing to the domain the server will have is the first item on the list:

Now set the contents of /etc/hostname to be the hostname:

And add your hostname to the first line of /etc/hosts:

Now you’ll want to regenerate the server’s default self-signed SSL certificate so that it matches the domain name. You may have purchased an SSL certificate for your mail server, but it is perfectly possible and completely secure to run a mail server using a self-signed certificate. The only consequences will be warning screens when using webmail hosted on the server and warnings from Microsoft Outlook when connecting via POP, IMAP, or SMTP.

5) Now Build a LAMP Web Server

You will need the mailserver to also be a LAMP (Linux, Apache, MySQL, PHP) web server, since you will want webmail and a web-based administrative interface for managing users. So turning your Ubuntu instance into a web server is a good place to start. There is a shortcut to install the basic LAMP packages, so start by updating the repository data and installing the packages. Notice the “^” at the end of the command there – it is necessary:

During this install you will be asked to choose a root password for MySQL. Choose something sensible, and then move on to adding an array of basic additional packages for PHP – such as APC bytecode caching, memcache support, cURL, an XML parser, and GD image processing. Add more to suite your own taste and the applications you want to support on this server.

6) Configure PHP

The default configuration for PHP and the additional packages mentioned above is sufficient for most casual usage. So unless you have something complicated or high-powered in mind, you should probably only change the expose_php setting in /etc/php5/apache2/php.ini. Set it to “Off”:

7) Configure Apache

The expected end result for Apache is that it will serve a single site with a couple of running web applications: webmail and Postfix Admin hidden away in a subdirectory. All traffic will be directed to HTTPS – there is no good reason to allow non-secure access to any of what will be on the web server.

Firstly configure the following lines in /etc/apache2/conf.d/security to minimize the information that Apache gives out in its response headers:

Make sure that mod_rewrite, mod_ssl, and the default SSL virtual host is enabled – you’ll need these line items to be able to force visitors to use HTTPS.

The default site configuration in /etc/apache2/sites-available/default can be edited to look something like this for the sake of simplicity:

But of course your taste and needs may vary. Keeping the same simple approach, the upper portion of the SSL configuration in /etc/apache2/sites-available/default-ssl can be set up as follows:

If you are using a purchased rather than self-signed SSL certificate, and you probably have a CA certificate bundle from the issuer, then you’ll want to further change these lines in /etc/apache2/sites-enabled/default-ssl:

To push visitors to HTTPS, put something similar to the following snippet into /var/www/.htaccess:

8) Install and Configure Memcached

You will need to install Memcached to support the webmail applications intended to run on this server:

The default configuration file at /etc/memcached.conf is good enough for a small server: it locks down access to localhost and provides generally sensible configuration parameter values. If you are building a larger machine for heavy usage, you will probably want to bump the memory allocation to be higher than the default of 64M:

9) Install the Mailserver Packages

Now we’re ready to start in on the harder stuff. As for the LAMP server, there is a shortcut for installing the basic packages for a mail server. Again, note the “^” at the end of the command:

When Postfix installs, you will be asked to choose a general type of mail configuration – select “Internet site”. You will be asked for the system mail name, which is the hostname of your mailserver – e.g. mail.example.com. What this gives you is pretty much just bare bones, aimed at a mailserver that manages its users as straightforward Unix users, and which doesn’t use a SQL database to store data. So we need the rest of the cast – such as MySQL support for Postfix and Dovecot, and the coterie of spam-mashing packages. You might also have to install IMAP support for Dovecot, as it may or may not be included in the mail-server packages:

The php5-imap package actually supports POP3 as well as the IMAP protocol, and will be needed by Postfix Admin and many of the possible options for PHP webmail applications. Restart Apache to have that running and ready:

Next you’ll want some optional packages that extend the abilities of the spam and virus detection packages, such as by allowing greater inspection of attached files:

You probably also want a package for dealing with RAR-format archives – but I’ve found unrar-free to be somewhat buggy and unstable, while unrar is not free. So you may just choose to skip that and shrug.

10) Create a Mail Database and User in MySQL

Log in to MySQL as the root user, entering the password you set earlier:

Now set up a database and user for the mail software. This database will store information on user accounts and mail domains, using schema set up by the Postfix Admin package:

11) Install Postfix Admin and the MySQL Schema

Postfix Admin is installed as follows. To start things off, download the package from Sourceforge, unpack it, and move it into a subdirectory of your webroot. You will probably also need to change ownership to the www-data user:

Next is an interesting sort of a two-phase setup process. Firstly alter the following lines in/var/www/postfixadmin/config.inc.php:

Note that the last items above are only for the purposes of defining how Postfix Admin stores its data – they don’t set system paths for mailboxes. The actual system paths to virtual mailbox directories are defined in the Dovecot configuration outlined in a later section of this post.

Next open up a web browser and visit your mail server at:

Follow the instructions on that page to choose a setup password, and generate a hash of that password. Add that hash to the configuration file and save it:

Then return to the setup page. You can now use the password you selected in order to create an initial administrator account. Postfix Admin will also automatically create its database schema at this point.

It is probably wise to restrict access to /var/www/postfixadmin/setup.php after having used it. Create a file/var/www/postfixadmin/.htaccess and put the following instructions into it:

12) Create the Domain and Accounts in Postfix Admin

Now navigate to the main Postfix Admin login page:

Log in as the newly created administrator account, and then choose the “New domain” option under “Domain List” in order to create the example.com domain. You can then add mail users (“Add mailbox”) and aliases (“Add alias”) while viewing your domain. This will populate the schema, but it won’t do anything else yet as none of the other mailserver components are configured to look at the database at this point.

Postfix Admin does have another useful function during this long setup process – it allows you to send mail to local users through the web interface, which is helpful when testing your configuration and chasing down errors.

13) Create a User to Handle Virtual Mail Directories

Virtual mail users are those that do not exist as Unix system users. They thus don’t use the standard Unix methods of authentication or mail delivery and don’t have home directories. That is how we are managing things here: mail users are defined in the database created by Postfix Admin rather than existing as system users. Mail will be kept in subfolders per domain and account under /var/vmail – e.g. me@example.com will have a mail directory of/var/vmail/example.com/me. All of these mail directories will be owned by a single user called vmail, and Dovecot will use the vmail user in order to create and update mail files.

Note that the user and virtual mail directory folder are using the “mail” group, and allowing other users in that group to modify the contents.

14) Configure Dovecot

Dovecot will manage IMAP and POP3 connections, local mail directories, and receive incoming mail handed off from Postfix. It will also manage authentication for SMTP connections – no point in having two separate authentication systems when Dovecot can handle both cases. Configuration is spread across a number of files in /etc/dovecotand subfolders thereof, and might seem a little intimidating, but it’s all laid out fairly logically. The first thing to do is to ensure that Dovecot is looking for user data in the database created by Postfix Admin, so edit or create the file/etc/dovecot/conf.d/auth-sql.conf.ext to have the following contents:

Now edit these lines in /etc/dovecot/dovecot-sql.conf.ext such that it uses the MySQL database created by Postfix Admin:

Then change the controlling definitions in /etc/dovecot/conf.d/10-auth.conf such that Dovecot will read the SQL configuration files. While you are there, you should also make sure that plaintext authentication is disabled unless the connection is encrypted or local:

Next up, tell Dovecot where to put the virtual user mail directories. That requires the following changes in/etc/dovecot/conf.d/10-mail.conf:

If you are bringing your own SSL certificate to the party, you have to let Dovecot know about by editing these lines in /etc/dovecot/conf.d/10-ssl.conf. Remember to include your CA certificate bundle if provided with one by the certificate issuer:

Next, edit these lines in /etc/dovecot/conf.d/10-master.conf to add the Postfix option:

You may have to explicitly set a postmaster address in /etc/dovecot/conf.d/15-lda.conf; if you see “Invalid settings: postmaster_address setting not given” showing up in the mail log, then this is the fix for that. Make sure that a suitable alias or mailbox exists for your chosen postmaster address:

You’ll want to change the Dovecot configuration to be accessible to both dovecot and vmail users:

A final note on Dovecot: it only creates a user’s mail directory when mail is first delivered to that virtual user. So creating a user in Postfix Admin will not result in the immediate creation of a mail directory under /var/vmail, and that’s just fine.

15) Configure Amavis, ClamAV, and SpamAssassin

Before configuring Postfix, we may as well take a short detour into configuring the spam and virus tools. Their default configuration is close to what most people will need, and tools like SpamAssassin auto-detect many of the optional additional packages you may have installed. If you have specialist needs or greater knowledge, you can of course spend a fair amount of time here crafting intricate rules. For the casual user, this is a quick and straightforward process, however. Note that here we are putting off the portions relating to integration with Postfix – e.g. additions to the master.cf file – into the Postfix section of this post.

First add Amavis and ClamAV users to one another’s groups to enable them to collaborate:

Then turn on Amavis by editing /etc/amavis/conf.d/15-content_filter_mode – the software is disabled by default, so uncomment the @bypass… lines:

Now enable SpamAssassin by editing these lines in /etc/default/spamassassin:

SpamAssassin under Amavis will only check mail that’s determined to be arriving for local delivery. There are a couple of ways to tell Amavis which mails are for local delivery, but here we’ll set it up to check the database set up by Postfix Admin. Edit /etc/amavis/conf.d/50-user to look like this:

You will have to restart these processes to pick up the new configuration:

16) Configure Postfix

Postfix handles incoming mail via the SMTP protocol, and its configuration files have be set up to allow it to integrate with the various other packages we have installed so far. At a high level, we want Postfix to hand off incoming mail to the spam and virus checkers before passing it on to Dovecot for delivery, and to authenticate virtual users who are connecting over SMTP in order to to send mail.

Firstly create files describing for Postfix where to find information on users and domains. Note that the “hosts” directive in these files must be exactly the same as the “bind-address” in /etc/mysql/my.cnf. If one side says “localhost” and the other side says “127.0.0.1” then you may find that Postfix cannot connect to MySQL – strange but true. Here are the needed Postfix files:

/etc/postfix/mysql_virtual_alias_domainaliases_maps.cf

/etc/postfix/mysql_virtual_alias_maps.cf

/etc/postfix/mysql_virtual_domains_maps.cf

/etc/postfix/mysql_virtual_mailbox_domainaliases_maps.cf

/etc/postfix/mysql_virtual_mailbox_maps.cf

Now create the file /etc/postfix/header_checks, which will contain some directives to remove certain headers when relaying mail. This improves privacy for the sending users by such things as stripping the original IP address and mail software identifiers, for example. This file will be referenced in the main Postfix configuration:

The following is the complete main Postfix configuration file at /etc/postfix/main.cf, which contains a fair number of complex choices and options on how mail is relayed and how SMTP behaves. It is far beyond the scope of this post to explain each and every choice of best practice or configuration parameter in detail. I strongly suggest that you spend some time reading up on Postfix configuration, as this is where it is easy to fall down and produce a suboptimal or faulty mailserver.

To be clear, if you are using a purchased SSL certificate – and have a CA certificate bundle from the issuer – then you will have to alter these lines in /etc/postfix/main.cf:

Further, if you are running Postfix version 2.10 or later, which might be the case if you are reading this recipe for pointers on an installation on a later version of Ubuntu, then you will need to add the following lines:

You must also add some material to /etc/postfix/master.cf, and here is the entire file for clarity, including much of the default material from the package install – such as commented options:

Note that Amavis is restricted to three processes, which should be fine for most casual to moderate use. The processes are memory-heavy, so start low and add more only if you need to due to volume of mail – see the notes in this guide for pointers on how to do that.

17) Restart Everything, and Test the Server

Restart all the necessary processes to pick up configuration changes:

Now start testing! Keep an eye on /var/log/mail.err and /var/log/mail.log for error messages and try logging in to POP and IMAP, sending mail to an account created on the server, and sending mail from the server. If you find issues, then Google is your friend when it comes to searching on specific error messages in order to identify where the configuration is wrong, or when something unexpected crops up.

18) AWS Mail Restrictions and Reverse DNS Lookup

Once configured, with IP address set and DNS records set up, you’ll need to have a reverse DNS lookup put in place for your server, and the AWS outgoing mail restrictions lifted. You do that through the standard customer service form. This doesn’t take long, and it can actually happen earlier in the process if necessary, prior to the server completion.

 

22) Install and Set up Monit for Monitoring

Monit is a very useful monitoring tool that helps rescue your server from failed processes. Install it through apt-get:

The following are a set of fairly trivial instructions that set monit to watch over the important server processes – but without issuing notifications or doing much more than restarting on failure. Note that the Amavis configuration specifies a fairly infrequent check; it’s possible to get into a situation with Amavis where it refuses connections because you’re sending mail too rapidly and it hits the server’s maximum number of concurrent connections per process (which is set at a low 128 for Ubuntu). Having Monit then restart it at that point just makes things worse, boosting load and slowing things down. Mail will be queued and reattempted for any period while Amavis is truly down and waiting on Monit to restart it.

Create the following files in the Monit configuration directory.

In /etc/monit/conf.d/amavis:

In /etc/monit/conf.d/apache2:

In /etc/monit/conf.d/dovecot:

In /etc/monit/conf.d/mysql:

In /etc/monit/conf.d/memcached:

In /etc/monit/conf.d/postfix:

In /etc/monit/conf.d/spamassassin:

In /etc/monit/conf.d/sshd:

Then restart Monit to pick up the new orders:

Monit offers options for notifications, a web console, restarting on high load, logging activity, and many other amenities, so you may want to add more to this very basic configuration.

Notes on Serving Multiple Domains

You can create multiple domains in Postfix Admin if so desired, under Domain List -> New Domain. If you want to use this mail server for more than one domain, you must (a) add the domains in Postfix Admin, and (b) consider whether or not to create domain-specific configuration files for Horde.

Additional domains added in Postfix Admin can be aliased to existing domains (under Virtual List -> Add Alias Domain), such that address@example1.com is always forwarded to address@example2.com, or they can stand as distinct domains with their own accounts, forwards, and so forth.

Depending on your use case, you might also want to adjust some of the .htaccess rules to support users accessing the site at mail.example1.com, mail.example2.com, and so forth – such as expanding the redirect to SSL to recognize all of the domains used.

Horde should work for multiple domains with just the one configuration file, but this may not be optimal for your usage. Try it and see. If not, then you will have to create parallel Horde configuration files for each domain you are using. See the Horde documentation for more on this, as well as the notes for $conf[vhosts] in the Horde configuration web interface.

Notes on Managing Quotas

If you’ve been following carefully, you’ll note that nothing has been said so far on the matter of user disk space quotas – it was not an important goal for the work that prompted the creation of these instructions. As things stand the necessary fields for quota managment exist in the MySQL database but are not used, as (a) the quota module isn’t enabled by default in Dovecot, and (b) Postfix Admin is set not to use quotas by default.

So if you want to enable disk quotas, first alter the Postfix Admin quota configuration in/var/www/postfixadmin/config.inc.php:

Next, you will want to enable and configure the quota and imap_quota modules in Dovecot. The former manages quotas while the latter enables reporting on quotas via IMAP. You will want to look through the following documentation for instructions on how to do this:

• Quota (Dovecot 2.*)
• Quota Configuration (Dovecot 2.*)

These configuration changes will be made in 10-mail.conf and 90-quota.conf in the /etc/dovecot/conf.d folder.

Bypassing Spam and Virus Checks for Local Mail

If you’re in the business of sending out newsletters or frequent updates from local software where you completely control the content in those emails, then you probably don’t want to run spam and virus checks for those items. It’s a pointless use of server processing cycles, and a newsletter run can hammer the server if you are making it process the full range of checks on each and every one of those mails.

To have amavisd-new skip the checks for mail originating from a known set of IP addresses (e.g. locally, from a web application on another server, etc), edit /etc/amavis/conf.d/50-user to add these lines:

Replace 111.111.111.111/32 with whatever set of IP address ranges you want to bypass amavisd-new checks. All mail arriving from those sources will fall into MYNETS for amavisd-new and therefore bypass checking. If bypassing by IP address doesn’t fit your needs, you can find ways to skip checks for some users, destinations, or sources in a helpful, if dated guide to amavisd-new and Postfix integration.

Some Final Notes on Security

You’ll note that there are a fair number of configuration files that contain database passwords for the mail and Horde data in this server, and that includes PHP files sitting in the webroot. This is not really the dominant security concern: the mail users are virtual and only the server administrator should be logging in as a system user. On AWS the default setup is for SSH login to use keys rather than passwords, and only the ubuntu user has a key setup to allow login. You can also easily lock down the SSH port to selected IP addresses via the security group applied to the server. Further, you can set .htaccess directives to ensure that no web visitor can directly view configuration files – and thus they are only used as includes, which covers the rare case where some error causes PHP files to be served by Apache as plain text. MySQL access is from localhost only, in any case.

All in all the lowest bar from a security perspective is probably that the mail server built here runs a couple of complicated PHP web applications with database access. A serious breach there would involve a way to upload and execute an arbitrary PHP script or shell command with the www-data user’s permissions, or various other XSS attacks allowing for session hijacking of administrators – either way, or just by getting into the mail and Horde databases, compromise of the webroot is compromise of all of the important functions of the server. Horde has had multiple vulnerabilities in past years, but at some point you have to pick your software. On the whole which given the choice I’d rather go with the output of established development communities whose members have a demonstrated track record of vulnerabilities found and fixed, and where there are a large number of eyes directed at the codebase.

These are all good reasons for setting up your webmail on a different server from the one running Postfix and Dovecot – something to bear in mind.

Of course being on AWS – or indeed pretty much any sort of easily available hosting in the US wherein the server is not in your front room – means that the US government has free access to your data any time they particularly feel up to the task, and you may never know a copy was taken. One of the welcome forthcoming evolutions in virtual hosting services will be some form of turn-key encrypted server operations such that you can have the convenience of an AWS-style service but without the transparency it affords the present day panopticon-in-the-making.

Further, it is apparently the case that all email traffic between mail servers is being recorded by various governmental agencies. Unfortunately the present state of SMTP in the wild is that many or most mail servers do not implement the ability to pass emails over an encrypted connection: so while it’s easy to connect to your mail server via POP, IMAP, or a webmail interface with encryption, the email traffic between servers is often plain text. Forcing your server to only use encrypted connections with other servers will mean that a large fraction of your email traffic in both directions will be rejected or unsent. Thus the configuration provided for Postfix in this post is for optional encryption – emails sent and received will be encrypted if the mail server on the other end of the connection can support it.