loading . . . Did This Email Arrive In Your Inbox? So uhh...hi.
I hope you're not just reading this on my website, through ActivityPub, , or RSS, but also in your email clients. Because here's the thing. Ideally, this post was sent as a newsletter through my Ghost site. Duhh?
Usually, on Magic Pages that happens through Mailgun. Yet, this post is sent through my own server. From beginning to end.
But let's take a step back.
Ghost sends bulk emails through Mailgun, a commercial (and not quite cheap) email service. For this, Ghost has always had the ability to set a Mailgun API endpoint in its configuration. Basically, you can point it wherever you want.
That's how some people proxy the Mailgun requests and then do all kinds of stuff with it. Mostly re-route requests to Amazon SES, since that only costs a fraction.
But there's always been two big issues for me with these kind of proxies: email analytics and data sovereignty.
## Email Analytics
Natively, Ghost can display open rates for email newsletters, show bounces, etc. These analytics are generated by making regular calls to the Mailgun API (every 5 minutes, in fact).
Now, if you proxy the requests Ghost makes to Mailgun...you'll need to stub these analytics requests as well 🙃
Most proxies I have seen don't do that. There's one that does though:
GitHub - tilak999/mailgun-ses-proxy: Proxy server to use AWS SES instead of mailgun in ghost CMSProxy server to use AWS SES instead of mailgun in ghost CMS - tilak999/mailgun-ses-proxyGitHubtilak999
While that proxy does take care of analytics, the problem I see is the fact that Amazon SES does not have similar analytics endpoints. You need to set up a whole set of AWS products to pipe events and use them. So, lots of moving parts.
But even if that part is somehow solved, there's still the issue of...
## Data Sovereignty
You're still just passing data to someone else. As mentioned, most of these proxies try to send emails through Amazon SES, or other commercial bulk email senders.
Now, I know what some of you might be thinking: "Yeah, but that's for a good reason. You're sending bulk email."
And that's right. Even though email is one of the initial open standards, sending bulk emails through SMTP isn't easy.
But let's focus on one problem at a time.
* * *
These two issues – proper analytics and data sovereignty – stuck in my head for a few days. I became somewhat fascinated by the architecture of email sending infrastructure, and how everybody is seemingly relying on a few players, just to get some basic things to work.
And I asked myself: how do big organisations deal with that? I am absolutely sure that not everybody pays Mailgun, Amazon SES, Postmark, etc.
And even if: how do Mailgun & Co. send emails at scale? It's not like they can use some secret sauce that nobody else has access to, right? They are also just built on technology that everybody has available.
I found the answer in **Mail Transfer Agents** (MTA).
While SMTP is responsible for accepting a message you want to send to someone (and then dealing with it under the hood), an MTA is the part of the "chain" that actually hands the email over to the recipient's server.
Usually, an MTA is part of an SMTP server, but without the fluffy "I accept your message"-part around it. There are a ton of commercial MTAs out there, but what I found particularly fascinating (read: "getting obsessed with it") was a project called KumoMTA:
KumoMTA - Enterprise Open-Source On-Prem/Private Cloud MTAKumoMTA is the world’s first Open-Source MTA designed from the ground-up for large-volume senders. Get a fast, flexible, supported solution.Horizontal_KumoMTA-TM-299x132Mike Hillyer
KumoMTA was built by a bunch of people who have worked at huge email delivery companies. They created an open source Mail Transfer Agent that is able to send 7 million emails per hour reliably on a single server.
For comparison, Magic Pages sends about a million emails per month. So yeah, this thing could handle my entire monthly volume in about...8.5 minutes.
KumoMTA actually powers some commercial email sending providers like AWeber or AHASend. And behind the scenes, most other ESPs use something similar.
However, KumoMTA is basically just a really, really fast email delivery engine. No built-in analytics, no bounce management, no hand-holding. Just "give me an email, I'll deliver it at massive scale."
Which meant I'd have to build all the other stuff myself. So, uhhh...proxy time!
## How You (Hopefully) Got This Email
Well, realistically, it's rather "API Wrapper That Just Works As A Proxy Within The Ghost Context"-time.
In the last couple of weeks, I've been experimenting with KumoMTA. First locally, then on a real server, figuring out configurations, DKIM signing, all that stuff.
And then I wrote an API that Ghost _thinks_ is Mailgun.
Once I am done writing this post and press the "Publish" button, here's what will have (hopefully) happened:
Ghost will make an HTTP request to what it thinks is Mailgun, but is actually my server. My server said "thanks, I'll handle this," queued the email, and immediately told Ghost "yep, all good."
Then, in the background, my server took your email address, composed the message with proper headers, and made an internal HTTP request to KumoMTA running on the same machine. KumoMTA grabbed the DKIM key for my domain, signed the email cryptographically, looked up your email provider's mail servers, and delivered it directly. Your email provider will place the email in your inbox, and notify you.
No Mailgun. No Amazon SES. No SMTP. No third party at all. Just two servers talking to each other.
And when you opened this email (and your email client doesn't block it), a tiny 1x1 pixel image got loaded from my server. My API wrapper logged that event and stored it in a database. When Ghost asks "hey, did anyone open that newsletter?" every 5 minutes, my server says "yep, here's the data".
All of it running on a single server that costs me around 4 Euro per month, in neat little Docker containers.
## What Does This Actually Mean?
I'm not switching all of Magic Pages over to this setup tomorrow. This is running on my personal blog because, well, developers are curious creatures. I wanted to see if it actually works, how reliable it is, whether the analytics are accurate, if emails actually get delivered.
It's an experiment. And uhh yeah...you're my guinea pig 🙃
Given the versatility, I can see this as a solution to offer more flexible email pricing on Magic Pages for big senders. A dedicated IP address on Mailgun is $59/month. By spinning up a server with my API wrapper, I can virtually send unlimited emails for a fraction of that cost. And I could do it relatively quickly, given the proxy architecture.
Not only would this cut my cost, but also give options to these high volume senders on Magic Pages. Dedicated IP, full control over emails, a server that can scale virtually endlessly, if needed, without per-email cost.
However, let's recognise one thing. The "proper" way to solve this would be for Ghost to support bulk email adapters and write one for KumoMTA.
Right now, Ghost has adapters built in for storage, caching, and SSO (the latter, I just found in the documentation – I haven't actually tried that myself). But for bulk email? You get Mailgun, and that's it.
There's regular discussion about this in the Ghost community. The Ghost team acknowledges that moving to an adapter-based approach would solve these email infrastructure challenges, but it's "not currently on the roadmap." They're happy to support contributors who want to work on it, though:
Support sending bulk email from other providersI’ve updated the title of this thread to make it a more general topic around adding support for other mail providers. As mentioned on the PR that was opened, if we move forward with this, it needs to use our adapter-based approach to solve the underlying architecture. There are two major pieces to this: bulk mail sending email analytics I think if the first were solved, more people would be interested in contributing to 2. Although the Ghost team have some plans to improve aspects of email…Ghost ForumHannah
Since 2022, when this was published, there have been a few attempts to implement bulk email adapters in Ghost, but none of made it into a PR that fully satisfied the Ghost team's requirements (which are very valid, in my eyes – e.g. start small by not adding UI elements).
So until Ghost has proper email adapters, we're stuck with proxy workarounds. Depending on how this email goes, I might also implement it on the magicpages.co site, since that has a bigger subscriber base.
However, I rather see this as an experiment and not something that will replace all email sending on Magic Pages.
And: if it turns out to work, I want to release it as a full open source end-to-end solution. More on that in the future though. Still lots of things I need to fix in the code 😄
Oh, uhh...PS: If you got this via email, please let me know by quickly replying 😂 https://www.jannis.io/did-this-email-arrive-in-your-inbox/
