setup STARTTLS with sendmail and openSSL

STEP 1: Enable STARTTLS in sendmail:

1. Install openSSL ( ) as described in the openSSL’s INSTALL

file. After installation, be sure that the openSSL program is in your path

(cp /usr/local/ssl/bin/openssl /usr/bin), otherwise the program does

not work.

2. Create or edit /usr/src/sendmail-8.12.7/devtools/Site/site.config.m4 and

insert the following lines:

APPENDDEF(`confINCDIRS’, `-I/usr/local/ssl/include’)

APPENDDEF(`confLIBDIRS’, `-L/usr/local/ssl/lib’)


APPENDDEF(`conf_sendmail_LIBS’, `-lssl -lcrypto’)

3. Rebuild and install sendmail with the -c option (see README in


cd /usr/src/sendmail-8.12.7

./Build -c

./Build install

4. Check to see if sendmail is compiled with STARTTLS:

/usr/sbin/sendmail -d0.1 -bp

5. Edit /usr/src/sendmail-8.12.7/cf/cf/ and insert the following


define(`confCACERT_PATH’, `/etc/mail/certs/’)dnl

define(`confCACERT’, `/etc/mail/certs/cacert.pem’)dnl

define(`confSERVER_CERT’, `/etc/mail/certs/cert.pem’)dnl

define(`confSERVER_KEY’, `/etc/mail/certs/key.pem’)dnl

define(`confCLIENT_CERT’, `/etc/mail/certs/cert.pem’)dnl

define(`confCLIENT_KEY’, `/etc/mail/certs/key.pem’)dnl

6. Backup and regenerate /etc/mail/

cd /etc/mail


cd /usr/src/sendmail-8.12.7/cf/cf


./Build install-cf

7. Now you have to create three files: cacert.pem (CA certificate), cert.pem

(x.509 certificate, signed by CA) and key.pem (x.509 private key). This is

how you do this:

cd /usr/local/ssl/certs

../misc/ -newca

../misc/ -newreq-nodes

../misc/ -sign

mkdir /etc/mail/certs

cp demoCA/cacert.pem /etc/mail/certs

cp newreq.pem /etc/mail/certs/key.pem

cp newcert.pem /etc/mail/certs/cert.pem

chmod 400 /etc/mail/certs/key.pem

When the command ‘ -newca’ asks for a Common Name, fill in the name of

your organization. When the command ‘ -newreq-nodes’ asks for a Common

Name, you must enter the hostname of your smtp server and it must be the

same name as your smtp-server field on the mailclient, e.g.

8. Restart sendmail :

kill `head -1 /var/run/`

/usr/sbin/sendmail -L sm-mta -bd -q30m

telnet localhost 25

9. Check if sendmail supports STARTTLS. Issue a ‘EHLO localhost’ command.

You should see a line 250-STARTTLS:

root@server:/# telnet localhost 25


Connected to localhost.

Escape character is ‘^]’.

220 ESMTP Sendmail 8.12.7/8.12.7; Wed, 19 Feb 2003

10:59:57 +0100

EHLO localhost Hello root@localhost [], pleased to meet












250 HELP

Check your logfiles if you don’t see it (increase LogLevel to 14 in your So far for the server side setup.

STEP 2: Client side setup:

The next step is to configure your mailclients for SSL smtp connection and

install the client personal and root certificates on them. If you don’t

install these certificates, the client will complain that it cannot verify

the server certificate. This is normal, because you are using a self signed

servercertificate. You do not have this problem if you are buying a

certificate from a trusted provider.

For Outlook Express 6 e.g., you wil see this warning:

“The server you are connected to is using a security certificate that could

not be verified. A certificate chain processed, but terminated in a root

certificate which is not trusted by the trust provider.”

This is how you create your client certificate:

cd /usr/local/ssl/certs

../misc/ -pkcs12

It will create a file newcert.p12 that you can import in your client. You

will have to fill in a password, that the client have to use when he imports

the newcert.p12 file, so don’t use the same as you have used earlier.

— For Outlook Express 6 you can follow these steps:

1. Copy the file newcert.p12 to a directory on your Windows client and

rightclick on it.

2. Choose install PFX and follow instructions. It will install a personal

and a root certificate. That’s great. You can check it in Explorer >

Internet Options.

3. In Outlook, turn on the SSL option for your outgoing mailserver.

— For Netscape Messenger 4.7 you can follow these steps:

1. Start Messenger, click on the little lock on the bottom left corner.

2. Choose Certificates > Yours > Import a certificate

3. Import the file newcert.p12.

4. Click on Signers, select your CAcert, and edit it to enable all features.

5. Click on Yours, select your personal cert, and click Verify. You

should get “successfuly verified”.

6. Enable Secure SMTP in Messenger config.

Now send a mail to yourself and check the message source. It must contain a

Received header with SSL information. You can also check your sendmail log.

If you see Verify=OK, then the server verified the presented client

certificate as OK. If you see Verify=NO, then the client didn’t present a

certificate and you are probably using Outlook Express. This is what I found

on the Internet: Outlook Express as of Internet Explorer 5 will work, but it

will not present any client certificate. So you can encrypt your email

transfer but you cannot authenticate (and relay) with client certificates

(source: … /test.html ). Post

a follow-up for comments on this.

I get Verify=OK with Netscape Messenger 4.7 and Verify=NO with Outlook

Express 6.

STEP 3: Allow relaying based on client certificate:

The last step is to allow relaying based on a trusted client certificate.

This is very useful, because your client can then send mail to you

mailserver, independent of his IP. It only works if the server can verify

the client (Verify must be OK). Do the following:

1. Open your sendmail logfile and search for the “cert-issuer” field that

came from your client.

2. Copy the content of this field to your access database file (probably

/etc/mail/access_map, see also )

and insert CERTISSUER: and RELAY, like this:


SomeEmail RELAY

3. Create database map:

makemap hash access_map because you want to relay. You can use [email protected]. It will bounce your

mail and you can check all headers. Remember that this does this setup only

provide a secure transmission from sender to mailserver. Your mail will

probably go unsecure from there.

Now, that’s it. I hope this document is usefull and correct, don’t mail me

for suggestion/corrections but please follow-up to this post.

You can find more information on:]]>

Leave a Reply