E2E Email Testing

Something I’ve been wanting to do for a while: end-to-end testing of my email infrastructure. A recent incident offered the opportunity.

The incident

Some days after re-installing servers to the latest FreeBSD version, some acquaintance on Slack was kind enough to report that their messages to me were not delivered:

This was confirmed by the error message 4.7.1 timeout processing message in the mail logs:

Jul 22 15:19:19 mail postfix/cleanup[19292]: A0EBF81F0: milter-reject: END-OF-MESSAGE from mail-ot1-f47.google.com[209.85.210.47]: 4.7.1 timeout processing message; from=XXX to=YYY@foudil.fr proto=ESMTP helo=<mail-ot1-f47.google.com>

The culprit turned out to be Rspamd. More specifically the DNS resolution:

Jul 22 15:43:23 mail rspamd[20065]: <97f205>; lua; rbl.lua:293: error looking up 48.210.85.209.rep.mailspike.net: query timed out
Jul 22 15:43:23 mail rspamd[20065]: <97f205>; proxy; dkim_module_key_handler: cannot get key for domain 20251104._domainkey.gmail.com: dns request to 20251104._domainkey.gmail.com failed: query timed out
Jul 22 15:43:23 mail rspamd[20065]: <97f205>; lua; rbl.lua:293: error looking up s1gf9xow5tp1f7z8run945dw7yztizpq.email.rspamd.com: query timed out
Jul 22 15:43:23 mail rspamd[20065]: <97f205>; spf; spf_record_dns_callback: spf error for domain gmail.com: cannot resolve REDIRECT DNS record for _spf.google.com: query timed out

Fast-forwarding the investigation, it turns out Rspamd has its own DNS resolution, librdns, which default to reading /etc/resolv.conf and load-balancing entries in round-robin.

Why does rspamd have its own DNS resolution? The reasons put forward include:

• Async parallel queries. Rspam is built on an event loop (libev). glibc/libc resolvers are fundamentally synchronous and blocking.
• Portability across Linux/BSD. Glibc doesn’t really have a good async story either. getaddrinfo_a is limited and thread-pool based, not event-loop friendly.
• Control resolver behavior (retry/timeout/EDNS0/DNSSEC) instead of relying on OSes.

Why is this a problem in my setup? /etc/resolv.conf has name server entries that might not be reachable from inside a jail. Still Rspamd was trying them and failing. The solution? Set name servers explicitly:

# /usr/local/etc/rspamd/local.d/options.inc
dns {
    # Rspamd uses *round-robin* and its own resolver implementation (librdns).
    # Reason is that rspamd is built around an event loop which needs async
    # resolver code. Default nameservers are read from resolv.conf. We thus
    # better be explicit, at the cost of duplication. Format is
    # <ip>:<port>:<weight>
    nameserver = {{ rspamd_nameservers }};
}

Follow-up actions

Great so now it’s fixed, how do we prevent this from happening again?

Note this issue was not caught by existing monitoring: ports were reachable and the server was responding.

Well the solution is an email roundtrip test: send an email and make sure it’s delivered.

As I couln’t find any ready-made tool, I had to come up with my own: a cron script that:

• SMTP-sends an email to a specific test adress at my server.
• IMAP-checks that the email has arrived.
• Notifies another channel than my email server.

So actually the test not only tests delivery, it also tests retrieval. All the better.

Notification

The notification part proved difficult for me: 1. I’m not plugged into many notification channels in my private life; 2. most channels require a paying subscription or some involved setup.

I settled on:

• https://healthchecks.io/ for heart-beat (aka dead man’s switch) monitoring¹.
• Self-hosted ntfy instance for push notification to phone. Requires phone app.

To be honest I didn’t think this through and took it more as an experiment. In retrospect, I could just send the notification email to another fail-over server.

The S in SMTP

Great so I just deployed my script to a VPS and immediately observed rejects:

smtplib.SMTPDataError: (554, b'5.7.1 Spam message rejected')

Ah. SPF says I can’t send from=XXX@foudil.fr to=YYY@foudil.fr from a random server. Remember I want to test universal mail delivery on port 25, not authenticated submission on 587.

Ok so I also tried an additional hop via a legit ISP: authenticated submission to ISP from=XXX@myips.com to=YYY@foudil.fr.

smtplib.SMTPDataError: (550, b'5.7.1 Spam Detected - Mail Rejected.  Please see our policy at: http://XYZ/#spam_detected')

Oh I guess I can’t send automated emails with my private address then. 🙄

The solution²: allow-list my script on my email server of course! 🤦

Allow-listing

This (also) is actually trickier than I’d imagined.

1. You need to allow at two levels: rspamd and postscreen. See rspamd multimap module and postscreen doc.

2. Don’t blindly allow-list in Rspamd (short-circuiting the pipeline)! It would defeat the purpose of detecting things like DNS resolution failures:

   IP_WHITELIST {
     type = "ip";
     map = "${LOCAL_CONFDIR}/local.d/ip_whitelist.map";
     ## We don't want to bypass completely (prefilter = true; action = "accept";)
     ## as monitoring needs to test *some* filtering logic, like dns resolution.
     ## Hence the preferred negative scoring.
     score = -10.0;
   }
   

Conclusion

I’ve left some details aside, but infra-wise that’s:

26 files changed, 702 insertions(+), 9 deletions(-)

not counting the script code and the setup on external providers. That was quite an unexpected investment!