Earlier this month, I needed to be able to send automated email in order to send daily roundups of stuff produced by my bots. Most of the APIs I saw for sending lots of email was for mass marketing (with the kind of pricing and data policies you’d expect to come along with that). So, I thought, email is well-established*, so it must be in reach of the common developer! I’ll just set up my own email server and mailing list.
Well, you know how this is going to go.
Configuring the service on your server
The mildest part of this undertaking was installing and configuring a mail transfer agent (MTA). The MTA is the thing that the mail clients send mail to and receive mail from. The MTA relays that mail to other MTAs, which eventually route the mail to its ultimate destination.
On Ubuntu 18 (and probably earlier versions), postfix is installed by default. Installation done!
The configuration file for it is at /etc/postfix/main.cf. I changed the following things in it:
# Set the domain to your domain. My domain happens to be smidgeo.com.
mydomain = smidgeo.com
# Set your hostname to be on your domain.
myhostname = mail.smidgeo.com
# Tell postfix to use say that mail from here some from @$mydomain.
myorigin = $mydomain
# Send notification emails about all of these kinds of problems to root@$mydomain.
notify_classes = resource, software, bounce, policy
I then restarted postfix ([sudo] service postfix restart).
I tested by sending mail from the server (I tried both sendmail and mutt) to an address on Fastmail. After a while, I guessed that it wasn’t going to make it.
This is the most brutal problem with debugging mail stuff: Sending mail isn’t instantaneous, and you can’t tell how long is a reasonable time to wait to see if something failed. Sometimes mail doesn’t get to its destination for an hour, then makes it after that.
Tailing the mail log before you send an email helps to at least verify that postfix got the mail. tail -f /var/log/mail.log. A bunch of stuff will appear in the log if postfix gets your email and attempts to send it out. Sometimes, it’ll log that a relay rejected an email it sent, with a reason, which really helps!
Setting up DNS so that your mail server appears proper
I had known about MX records, but I wasn’t sure how necessary they were. My mail wasn’t going through, so it looked like they were necessary. Referring to the DNS section of the mail server guide, I created the following records:
Arecord for the mail domainMXrecord to point to the mail domainTXTrecord that saysv=spf1 mx -all(I don’t actually know what that means.)
Over in my postfix config, I initially did not have a fully qualified…something. It was either myhostname or myorigin. As a result, I got a mail.log entry that let me know Fastmail had rejected my MTA’s email:
Jul 21 21:04:00 smallcatlabs-disruption-pod postfix/smtp[19654]: 42AE5DF3AD: to=jimkang@fastmail.com, relay=in1-smtp.messagingengine.com[66.111.4.72]:25, delay=1.1,delays=0.02/0.01/1.1/0.02, dsn=5.5.2, status=bounced (host in1-smtp.messagingengine.com[66.111.4.72] said: 504 5.5.2 root@smallcatlabs-disruption-pod: Sender address rejected: need fully-qualified address (in reply to RCPT TO command))
Once I fully qualified the domain name, emails sent from my MTA were accepted by Fastmail’s relay!
The next barrier: Gmail
Since everyone’s on Gmail (actually one of the milder symptoms of the concentration crisis), I then tested by sending to Gmail. Gmail bounced my email. They have guidelines that give hints about why they bounce stuff. The most likely reason was that I did not set up a reverse DNS record.
Reverse DNS is not a standard, and as far as I can tell, in practice, it’s only used to verify that a spammer is sending email using a hijacked domain. I had some trouble finding out how to set up a reverse DNS record. My domain registrar, Hover, told me they couldn’t do it and suggested other companies.
I wanted to avoid something as disruptive as moving registrars, so I looked around a bit more on Duck Duck Go. Turns out that my virtual private server provider, Digital Ocean, provides reverse DNS for free. You just have to name your droplet (their branded term for a server instance) to match the domain name!
And yet another several rakes to step on
With the reverse DNS record set up, Gmail stopped bouncing mail from my server. However, it also put it in the spam box 100% of the time.
To get out of Gmail’s spam box, there are a few dozen suggestions on Google’s sender guidelines page, including using DKIM, SPF, and DMARC. Any one of those suggestions could have potentially taken days to implement, and there is no guarantee that following every one of them would get my emails out of the spam box.
I’m not some kind of email superfan, so I decided to avoid getting into this world.
Since the mail issue was getting trust from Gmail, I decided to leverage someone else’s reputation.
TinyLetter
Tinyletter is a mailing list service that I use for my projects newsletter. Tinyletter is a MailChimp product that Google already trusts. So, if I had something compose my bot emails and send email through TinyLetter, I could get that email through to my mailing list subscribers that are on Gmail.
In addition to having a UI in which you can compose and send emails to your mailing list, TinyLetter provides a secret email. You can send a message to that email, then TinyLetter will post that message to your mailing list.
The complication — for a fully automated mailing list — is that it requires you to confirm that you want to send the email. TinyLetter replies to your email with another email asking you to reply to that second email without changing the subject in order to send the first email to the mailing list.
Auto-reply
The first thing I needed to do to get auto-replies to these confirmation requests working was to actually be able to recieve mail at my server. Receiving email hadn’t worked, and I didn’t care. Well, now the time for caring was at hand.
One more of of the following measures have fixed receiving email at my server:
- I opened port 25 to the world by running
ufw allow 25. (I neglected to check whether it was closed before I ran this, so it may not have mattered.) - Properly included my domain in
mydestinationin/etc/postfix/mail.cf. Before, I had mail.smidgeo.com, even though the mail is actually addressed to @smidgeo.com. It has to match the part after the @ in emails sent to the server.
Next, I needed something to auto-reply to TinyLetter confirmation emails. I was hoping to not write something, so I checked out vacation, a nicely established simple Unix program for sending auto-replies that say the user is on vacation.
After getting it to work, I found out you can’t (as far as I could tell) make it reply using the Reply-To field instead of the From field of the incoming message.
This ended my vacation journey. TinyLetter confirmation emails come from some Mandrill remailer thing; sending your reply there does nothing. You have to send your reply to the email in the Reply-To field for TinyLetter to actually send the message to the mailing list.
So, I added my own little auto-reply program to email-rss-sample, my Node package for grabbing RSS, formatting the sampled entries into HTML email, then sending them via sendmail.
I hooked it up to receive incoming mail through stdin by adding this line to my .forward file (people that went to college in the ‘90s may remember .forward files as text files you just put a single email into):
\<username that the program runs as>, "|<location of this project on your server>/respond-to-confirmation.js"
That pipes email for the user to the program, which checks various conditions, then shells out to sendmail to send a reply.
What I ended up with
In the end, to email samples of what my bots did each day to a mailing list, I needed this system:
In the diagram above, every box represents some sort of program. Every arrow is a communication between a program with another program, containing a payload. The payload could be an email message, an RSS feed, or text. Each communication has a number indicating the order in which it is executed.
The part where RSS posts from my bots are sampled are formatted and assembled into an HTML summary is fairly straightforward and unit testable. Once we move on to the volleys of emails, things become less deterministic and controllable.
All that said, it’s been working fine for about a week. Sign up if you want some art generated each day! I’ve only had to apolologize once so far.
P.S. Why I thought it would be easy
Email is a technology that has been around since nearly the beginning of the Internet.
Here in 2019, 70% of people use either Gmail or Yahoo Mail, at least according to that poll. But in the ‘90s (and before), email traffic was handled by servers run by a diverse group of institutions. If you went to a university, your email was likely received and relayed by some computers on campus. If you worked at a company, your work email probably also went through servers in your company’s buildings. ISPs (and there used to be many ISPs) each had their own email servers.
This history made it seem like a stable technology that could be administered by someone with a 1999-level of technical understanding. Unfortunately, the spam wars (and maybe the consolidation of the control of email into very few hands), have made trust in email a much more complex issue.
As a result, sending email from your own server, like so many things, is more or less out of the hands of a person that has a day job and a limited amount of time. It is the domain of large corporate aggregators that trust each other.