# Setting up OpenSMTPD & Dovecot Mail Server

By

Tevdore Zhvania

●

In 

linux, 

mail server

●

Published 2021-09-19

So, you want to set up your own email server? In that case, welcome.

There are many reasons to run a custom email server, ranging from privacy concerns about providers like Google, to just wanting to do it for fun and/or learning. Since you’re here, I assume you’ve already found a reason.

Beware: this is a messy topic, and the available documentation is even messier, so it could take a while before you get it to work properly. I’ve compiled this guide according to my experiences in an attempt to make this dark art more accessible, but your mileage may vary considerably. I hope you find it useful.

This guide is aimed at people who are comfortable with the Linux/*BSD command line.

Setting up email is relatively complex compared to e.g. a static website, because you need to configure not one, but two server programs, and you need to shoehorn modern security features into email’s Stone-Age design. I’ll start by explaining the general structure of a mail server setup.

The programs involved in the exchange of emails are called agents. Officially there are 5 different types of agent: MUA, MSA, MDA and MRA. But unfortunately, it's reasonable to treat the MRA and MSA as being part of the MUA and MTA, respectively.

The Mail user Agent (MUA) is simply the client on your device at home that you use to send and receive emails, and this guide assumes you already have a favorite program for this, e.g. Thunderbird. Nowadays it's fashionable to use web interface for all kinds of tasks (emails included), but that's also beyond the scope of this guide.

The Mail Delivery Agent (MDA) is a program that watches over the server's copy of your mailbox: it manages your inbox, remembers which messages you have or haven't read, keeps a copy of your drafts, etc. When you open your mailbox, your MUA will connect to your server's MDA usind the IMAP protocol (or POP3, but that one's obsolete).

The Mail Transfer Agent (MTA) is responsible for making messages arrive at the right destination. When you send an email, your MUA will pass it on to your server's MTA, Which will in turn pass it on to the recipient's mail server. Likewise, when someone sends you an email from another server, the MTA will receive it and hand it over to the MDA so you can read it later. In both cases the MTA speaks the SMTP protocol.

In short:

1. MUA is your email program on PC;
2. MDA is a place where MUA goes to see all your mail stuff;
3. MTA is where MUA goes to send or receive your email.

In this guide our MDA will be Dovecot, which is a very popular choice for that role. As for the MTA, there exist several options, the most popular being Postfix and Exim. However, this guide uses the newer, lesser-known OpenSMTPD, which is in my experience is much easier to set up: Postfix and Exim have complex configurations and are geared towards large-scale email providers, whereas OpenSMTPD is more beginner-friendly.

The base email system is horribly insecure on its own, so we still need to duct-tame on some security features. in this context, "security" has two meanings: spam protection and privacy protection (encryption).

Spam protection also means two things here: defending yourself against spammers, and preventing your emails getting flagged as a spam. The former is optional, but the latter is not: big providers such as Google and Microsoft use infamously strict spam filters, and if they decide that your server is a spammer, there's almost nothing you can do about it. Spam protection techniques will be discussed in more detail over the course of this guide.

Privacy protection is important in the 21st century: you don't want a random router in the internet to read all your emails, which may contain sensitive information such as private conversations and account password reset links. You should therefore try to make sure that emails are transported over an encrypted channel. To do this, you have two options for encryption: mandatory and opportunistic encryption

Mandatory encryption is only practical for client-server communication (not server-server), and is provided by IMAPS and SMTPS, which wrap the IMAP and SMTP protocols in TLS, in the same wat that HTTPS does for HTTP.

For server to server communication, the only option is opportunistic encryption in the form of STARTTLS, where communication is only encrypted if both parties agree after a short unencrypted discussion. The last part is vulnerable to MitM attacks, where anyone along the path of the email server's discussion can alter the exchange to block the use of encryption, which sometimes actually happens in practice.

The only way to make sure that STARTTLS is used in that case is to refuse any exchange unless the servers agree to use encryption. Unfortunately, that's a risky approach that i can't recommend, because not all servers support encryption (unbelievable, right?). For Example, I've received airline booking confirmations, full of personal details, and made with billion-dollar companies, sent across the Internet without any protection.

This guide includes instructions to enable encryption, but assumes that you already have a TLS certificate for that. If not, find a guide to get one from Let's Encrypt (it's free), and remember that you'll need to renew it every few months. Using a self-signed certificate may work, but I don't recommend it.

In the rest of this guide I'll assume that you have a public full-chain TLS certificate at /etc/ssl/certs/example.com.pem, and a private encryption key at /etc/ssl/private/example.com.pem

Obviously, you'll need a server to run the MTA and MDA on. You can host your own at home, but the more reliable option is to rent one in a data center (VPS). This guide was written with a Linux server in mind, but in theory it should also work on the BSDs (OpenBSD, FreeBSD, NetBSD, etc.) with minimal adaptation.

The server must be online 24/7, you must have root SSH access, it must have a static IP address, and TCP network port 25 must be open. Especially check last one: you may need to explicitly ask your home ISP or the server provider to enable port 25, because they often close it to prevent spam. You can usually do this from their web interface.

You also must have a domain name, which I'll call example.com. This will be necessary for basically everything: DNS records, TLS certificate, MTA network configuration, etc. If you don't have one yet, you can choose between many registrars to rent one from. Personally I use and recommend Porkbun.

Note that It's a bad idea to use a domain like foo.bar.com, where you control the foo part but not the bar part: in that case, a spammer in control of qux.bar.com could negatively affect your reputation in the eyes of other email providers.

Lastly, when setting up an email server, you also have the choice between using the system users or virtual users. With system users, if an email arrives for john@example.com, then the MTA and MDA will expect that there exists a john Unix user on the server to deliver it to. With virtual users, you have much more flexibility, so that's what we'll use. All email will be managed under asingle unix user/group called vmail. Create it as follows:

groupadd -g vmail vmail
useradd -g vmail vmail

Now we must set up all the necessary DNS records, which is usually possible from the domain registrar's web interface. It may take a while for your changes to propagate over the Internet, so i recommend udoing this section now and the rest tomorrow.

First, you should already have an A and/or AAAA record to associate your domain example.com with the server's IP address. For email it is essential that you alse have reverse DNS set up correctly. If you're renting your server remotely, you can often do this from the provider's configuration tool, otherwise, you should create a PTR-type DNS record, although that's beyond the scope of this guide.

Once you're done, I recommend testing tour DNS records using the MX Lookup online tool.

To inform the rest of the internet that your server is an email server, create an MX (Mail eXchanger) DNS record for your domain. Note the dot at the end of the domain name:

When a message is sent to an email address ending in @example.com, the sender will query DNS for any MX records for example.com. There it will find a domain name (in this case example.com again), for which it will look up the IP address using an A/AAAA record. The domain name in the record must not have an associated CNAME record; it must be directly translatable to an IP address.

You may have multiple MX records, containing different domain names, each with a preference number (42 in the example above). The sender will try MX records with lower numbers first, and if that server is unavailable, it will try a higher number. If you have multiple mail servers (which is a good idea), you can thus declare those as follows:

Here, a server sending an email to your domain example.com will try to send it to the IP address of example.com will try to send it to the IP address of mx1.example.com first (because it has the lower priority in MX record), and if it fails, it will move on to mx2.example.com. If both mx1 and mx2 have the same number, then the sender will randomly choose one, which is useful for load balancing, although that;s probably overkill for a private server.

The Sender Policy Framework (SPF), is a feature which helps prevent spammers from impersonating your server in an attempt to get around blacklists. This security feature is required nowadays: if you don’t use it, you’ll probably get flagged as spam.

SPF works by specifying which IP addresses are authorized to send emails from your domain name. You must publish this information in a TXT-type DNS record (not SPF-type, which also exists!) with the following contents:

Everything after the version v=spf1 is a list of verification mechanisms for a spam filter to try out in the given order. The -all at the end says to reject your email if all of the previous mechanisms fail. See the SPF spec for details.

I recommend only using the mx mechanism, which tells the verifier to look at the A/AAAA addresses of the domains in your MX records. This allows you to add, remove, or change your servers without needing to update this record.

Then we have DomainKeys Identified Mail (DKIM), which is more comprehensive form of anti-impersonation, and, like SPF, is practically mandatory in the modern era.

It adds a cryptographic signature to all emails from your server, which receiver's spam filter will verify using the email's contents, and a public key that you need to publish in a DNS record. Again, you should implement both SPF and DKIM, despite their overlap.

to set up DKIM, create an RSA keypair, using the openssl utility:

openssl genrsa -out /path/to/dkim/private.key 2048
openssl rsa -in /path/to/dkim/private.key -out /path/to/dkim/public.key

The minimum size is 1024 bits, but i recommend 2048 bits. Bigger is better, but because DNS is involved, you can't stretch it to 4096 bits without causing discomfort to some servers. And I think it goes without saying that you should keep the private key private.

Importantly, the DKIM DNS record cannot be attachet directly to your domain example.com; instead, it should belong to subdomain of the form <selector>._domainkey.example.com, where <selector> is an alphanumeric string you can choose (e.g today's date), just remember your choice for later when configuring the DKIM signer. And if you change your key, keep old record around for a while so old emails can still be verified.

Your DKIM policy must be published in a TXT record as follows, where <pubkey> is the public RSA key MI...AB stored in /path/to/dkim/public.key, with the newlines removed:

<selector>._domainkey.example.com. TXT "v=DKIM1; t=s; h=sha256; p=<pubkey>"

Here, v=DKIM1 is the version and must be the first tag. The flag t=s enables strict mode, as recommended by the DKIM spec, meaning that emails sent from subdomains immediately fail verification. The optional tag h=sha256 blocks the use of old SHA1 algorithm.

Lastly, we have Domain-based Message Authentication, Reporting and Conformance (DMARC), which is technically optional, but highly recommended, because it will make you look more legitimate in the eyes of Google and Microsoft. It can modify the behaviour of SPF and DKIM, and also provides advice about what a receiver should do if one of your emails fails verification.

To enable it, create yet another TXT record, which, similarly to DKIM, must belong to the subdomain _dmarc.example.com, and give it the following contents, where <admin> is an email address of your choosing, which may or may not belong to your domain:

_dmarc.example.com. TXT "v=DMARC1; p=reject; sp=reject; pct=100; aspf=s; adkim=s; fo=1; ruf=mailto:<admin>"

The version tag v=DMARC1 must come first, followed by p= and sp=, which control what to do to unverified messages coming from the main domain and subdomains, respectively. Unsurprisingly, reject means that delivery should be refused, none asks to let it through anyway, and quarantine tells the filter to take a closer look or to put it in a spam folder. The percentage pct=100 says how many of your emails to apply the policy to. Next, aspf=s and adkim=s enable strict mode for both SPF and DKIM, which blocks subdomains from passing the test. Finally, fo=1 asks the filter to create a forensic report if any type of verification fails, and ruf= gives an address to send it to. If in doubt, see the DMARC spec.

Dovecot is a very popular IMAP server, focused on being lightweight, simple, and secure, and has extensive and up-to-date documentation. It's very flexible and scalable, and keeps up well with the latest security best-practices.

If you installed Dovecot via a package manager, you will probably have lots of configuration files in the /etc/dovecot directory. I want you to delete all of them. Yes rm -rf that crap. Dovecot is simple to configure, and doesn't cate where you put it's settings, so having all of that chaos in /etc/dovecot just makes thing unnecessarily confusing.

Create a new configuration file /etc/dovecot/dovecot.conf, and start by filling the detais of your TLS certificate, making clear that unencrypted connections are unacceptable:

ssl = required
ssl_cert = </etc/ssl/certs/example.com.pem
ssl_key  = </etc/ssl/private/example.com.key

ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes

disable_plaintext_auth = yes

The final disable_plaintext_auth option tells Dovecot to reject any passwords that were sent unencrypted. This means it must be hashed or sent over an encrypted connection, or both.

Next, tell Dovecot which protocols to use and where to expect them as follows:

protocols = lmtp imap

service lmtp {
	unix_listener lmtp {
		user  = vmail
		group = vmail
	}
}

service imap-login {
	inet_listener imap {
		port = 143
	}
	inet_listener imaps {
		port = 993
	}
}

LMTP is the Local Mail Transport Protocol, which is basically SMTP but for exchanges within a single server or over a trusted network. When an email is received, Dovecot will start a child process under the vmail user/group to deliver the message to its recipient.

Since we set ssl = required earlier, clients will only get their mail if the STARTTLS handshake was successful during the IMAP exchange, or if they connect via IMAPS to force the use of encryption. You can therefore optionally remove one of the two inet_listeners according to your preferences.

Next, we need to inform Dovecot which email addresses it should handle, and what to do with their messages. Create a file /etc/dovecot/users for this, which describes a user on each line in a similar format as /etc/passwd:

In this guide, we’re using the vmail user for all accounts, so leave the uid, gid, and homedir fields blank. We’ll be storing all emails in vmail’s home directory. The user field should be the email address excluding the @example.com (in fact, you can include it, but this guides assumes a small-scale server managing only one domain, so we exclude it). Create the password hash to put in the password field as follows:

# If your server is fast and has lots of RAM:
$ doveadm pw -s ARGON2ID-CRYPT
# If you're using a potato:
$ doveadm pw -s SHA512-CRYPT

After you’ve entered your password, simply copy-paste the entire hash string outputted by the program into the password field.