Author here. To give some context on why listmonk was built, at work (regulated financial business), we have to deliver e-mails, mostly important updates, to 1.5mn+ customers regularly. We used phpList for the longest time and then tried MailTrain and Sendy before finally deciding to reinvent the wheel after running into a number of issues, of which, a few important ones are mentioned below.
- Performance. Unreasonably long amounts of time to send out e-mails. phpList degraded to the point of taking several days to process a campaign. listmonk can spawn N goroutines (~threads) and push e-mails to multiple SMTP servers. On a commodity ec2 instance, we're able to send 1.5mn+ e-mails in a couple hours.
- Subscriber imports were extremely slow. Direct integration to keep subscribers in sync with external CRMs was cumbersome. Direct DB inserts were complicated due to the complex table structures. listmonk imports 10k records/sec into a Postgres DB on a commidty ec2 instance.
- Segmentation. Often, we have to rapidly segment users by custom attributes and conditions and relay an update to them. listmonk supports SQL expressions to segment users on their attributes that are defined as arbitrary JSON maps (thanks to Postgres JSONB type).
- Unavailability of dynamic templates. listmonk templates support Go template expressions so it's possible to write logic in messages to make them dynamic.
I would like to add that listmonk is still work in progress and requires a number of essential features before it can come out of alpha.
Return Path has a certification service that certain providers use as a signal to the “legitimacy” of the sender (don’t know if that was the best phrasing...).
> - Segmentation. Often, we have to rapidly segment users by custom attributes and conditions and relay an update to them. listmonk supports SQL expressions to segment users on their attributes that are defined as arbitrary JSON maps (thanks to Postgres JSONB type).
Out of the box, it isn't, but it's straight forward to do it with APIs (for individual entries and for bulk CSV imports) or with direct inserts to the DB (the table schema is simple). In production, we've made a small daemon that listens to a PubSub queue published by our CRM to keep the mailing list up to date.
About attributes, every subscriber entry in listmonk can take a JSON map of attributes that can be queried.
Hey, thanks for phpList! We were on it for the longest time, and personally, I first used it some 14 years ago.
We did try several things; Postfix tweaks, SES / Postal SMTPs, rate changes in the phpList config etc., but nothing seemed to work. This, along with the other factors (segmentation, dynamic templating), lead to listmonk.
This looks like the free/open-source version to Sendy based on more modern tech like Golang/Docker/etc I've been hoping for, congrats on your alpha release!
Thank you. It was a bit of a surprise to me that Sendy had no viable open source alternatives. phpList is okay, but is quite dated and has a number performance issues.
I've also been a Sendy user for a long time for non-critical applications, and I don't like the fact they hide their code so a tool like this is long overdue and much needed.
Projects like this seem like a great idea, but deliverability seems like a big concern that is hard to measure unless you have a reasonable amount of experience.
What are best practices for using/selecting an ESP if you were to use a project like this and want to ensure reasonable deliverability?
Author here. We've been using listmonk in production at our company (regulated financial business) to deliver e-mail updates including regulatory ones for over 6 months. We host our own SMTP instances using Postal[1] on EC2 instances and have never had any issues with deliverability. If it's legitimate e-mail, I don't think it's much of an issue.
>that aren't unilaterally marking you as spam for no reason
This sounds hopelessly naive to me. If the receiver marks it as spam, it's spam. By definition. Work harder to only send emails to people who want them.
> If the receiver marks it as spam, it's spam. By definition.
but that shouldn't effect the deliverability to everyone else, but thats a reality which has its own problem and has its own countermeasures.
when people asked OP if they had a solution to this, they hand waived it away. all closed sourced mailing list clients have solutions for this.
but anyway since somehow it is 2019 and people want to act like its 2009 on this matter I'll just skip ahead to echo the productive answer someone else left:
this could probably be run through Amazon SES with their DKIM settings
if I sign up on a site, click the 'yes email me' button,
and then mark the email as spam when I get it, I am the one acting incorrectly, not the sender who is literally doing what I instructed them to do..
Sure, but the end user is going to do whatever is most convenient for them.
Unsubscribing from email lists can be cumbersome and non-uniform, which is usually why people just hit that spam button instead. It always provides them a uniform interface and largely achieves the same goal from their perspective.
Actually, cold emails (without permission) aren't spam. What makes an email spam is: 1) misleading subject line 2) misleading message 3) lack of simple (not requiring login or extra steps) unsubscribe link, and a couple other factors.
People forget subscribing and hit "Spam", which has an impact on the sender reputation and affects future deliverability for that sender.
That's a very narrow definition of spam that you're only going to hear from spammers (your profile is utterly unsurprising after reading this comment). Unfortunately for you, end-users are going to mark as spam based on their own much more reasonable definition.
It's not my definition, that's how the CAN-SPAM Act of 2003 (applicable to US only) defines spam and most marketers are familiar with since it's been around for 15 years. Of course the laws are different in Europe and Canada.
One of the factors I didn't mention in previous comment is presence of physical mail address.
Not sure what "unfortunately for me" is in reference to - BigMailer isn't a platform for cold emails (there are plenty others that specialize in cold email campaigns) and Amazon SES doesn't give a free pass on the practice either.
The other comments here don't quite tell the whole story; "I never had any problems" is anecdotal - there are plenty of legitimate senders who do run into deliverability issues.
Using an ESP will generally simplify the amount of work necessary to achieve good delivery, but due to the complexities of domain reputation and fingerprinting, very occasionally they can also be the cause of delivery problems.
General guidelines:
- It's a bit harder to predict how deliverability will turn out at smaller ESPs, vs. well-established ones like Sendgrid, SES, Mailgun, and perhaps Mailchimp.
- Use the same root domain everywhere within your email: From, Return-Path, DKIM signature headers; URLs for links and images in the message body. (Separate subdomains are fine, e.g., img.example.com, bounce.example.com)
- Having your sending domain show up in third party email can cause problems. Try to avoid it if possible; use a separate domain as a last resort.
- Send consistently. Some variation in sending patterns will be tolerated, but you'll probably have trouble if you send 500 messages a day most of the time, then once a month you send 100,000 messages.
- Consider a dedicated IP if you're sending over 50,000-ish messages a month.
- You may want to split up transactional email from marketing, sending these from separate IPs and/or subdomains.
- Finally, make sure you're sending email your recipients find valuable and are interested in receiving. Too much disinterest/disengagement can cause delivery issues.
I agree with all your suggestions, but want to expand on a couple points:
1) The shared IP pool at cheaper providers like Sendgrid, SES, and MailGun will have lower reputation versus a platform with a guarded (via cost, but also on-boarding mechanisms) platforms like MailChimp, simply due to customer pool and quality of emails sent. I have been hearing from my (BigMailer.io) customers about having deliverability issues with platforms like SendInBlue and MailGun frequently in the last year or so.
2) If the bulk sending practices are healthy and engagement is good, then it's better to use the same sender/domain for both bulk and transactional email.
I would recommend to NOT send different email types from different providers, like use SendGrid or SES for transactional emails and then some other platform for marketing emails, especially using the same sender domain. That's because email header signature and tracking links will be different and email box providers like Gmail can find it suspicious and send emails to Spam. And then there is the issue of processing and syncing bounces/unsubscribes/complaints data.
Keeping all email campaign types in a single platform allows for centralized processing of bounces/unsubscribes/complaints date and protects future campaign engagement + sender reputation.
Right. Split up by IP/subdomain if possible, preferably within one provider.
That said, you may be surprised by the number of large organizations that have a dozen or more services that send mail on their behalf, for various different purposes... it's not optimal, from a deliverability perspective, but it's also not usually going to cause deliverability issues by itself as long as everything else is in order.
Yes, agreed. Lotsa factors in play when it comes to deliverability, like a puzzle. What a well established brand (with high quality emails) can get away with isn't something that businesses with less resources should try to copy though IMO.
Just saw an email from Starbucks in the spam folder - all because of a few words that no doubt triggered the spam filter.
Deliverability is a non-issue if you hook tools like this and Sendy up to Amazon SES with DKIM and verified MAIL FROM settings. I've been doing it for years and never once had a deliverability issue. I've also sent directly from EC2 instances without issues but SES is so cheap and reliable I've stuck with it.
Sounds like you have a good and established sender reputation, which is a big factor. When people start fresh (new sender identity) they tend to run into deliverability issues.
Also, the quality and size of the starting list matters too because a large list (like the OPs) or unengaged list can have a high bounce and low engagement rate, which factors into future deliverability.
I wonder if orgs using gApps can use it as an SMTP relay as a possible solution. My team did this (except we used our Office365 as a relay) when one of our clients brought in a new firewall and found that password reset emails were being blocked coming out of AWS.
Now before you mention how this should be a solveable problem inn AWS, we’re already well underway testing with the client. The problem presented itself as a manifestation of inheriting the last guy’s infrastructure and working to...unfuck a lot of his questionable design choices.
Looks great! Do you have any plans to support third-party sending APIs (e.g. SES)? I see the reference to SMTP.
I've been looking for a good solution for ad-hoc email campaigns (w/ templates) and for sending "systems" emails to an audience of customers.
I currently look after lists on both MailChimp and EmailOctopus and get frustrated by the flat pricing for the size of the list and not how you engage with the list (e.g. I'd love to see: store your list for $X/mo, or pay $X + (list size * $Y) in the months when you email the list).
You can use SES's SMTP interface with listmonk. With some added effort, you can self-host an SMTP server and achieve better throughput, and depending on the volume, cost reduction. We've been running our own Postal SMTP instances successfully.
As a business that sends emails infrequently, it's probably saved me hundreds/thousands and it has good instructions to set it up with SES. One-time (ish) fee, until you need to upgrade to the next major version.
Besides, that an email is produced by an AGPL software does not make it covered by the AGPL. That should be exactly the same legal situation as with open source text editors.
That's what I'm trying to understand, it was an honest question.
So, who does need to be provided with a copy? Just the 'administrators', the people using it to send email?
What about if I modify it and repackage it as monklist or whatever, and charge people to use it. I must give them the source code for listmonk too, and, since another requirement is same licence, the source for my modified version? i.e. monklist couldn't be closed source, but myapp that happens to use listmonk unmodified for sending emails can be?
> It requires the operator of a network server to provide the source code of the modified version running there to the users of that server. Therefore, public use of a modified version, on a publicly accessible server, gives the public access to the source code of the modified version.
So the license requirement is for the users of the software directly. They need to get access to the modified source.
> So, who does need to be provided with a copy? Just the 'administrators', the people using it to send email?
If that is a closed group said group. If it's the public the public. So in your case yes, the administrators.
You could make the argument (it has been tried before with editors) that the email is part of listmonk, but reasonably it would not stand. At first look there is more weight behind the argument that accessing the email is a form of network access between the user and listmonk, vie email, but I don't think that would hold up either, given the implications with regards of what is a software and what is a product.
> What about if I modify it and repackage it as monklist or whatever, and charge people to use it. I must give them the source code for listmonk too, and, since another requirement is same licence, the source for my modified version?
I don't think that you have to give the original source of listmonk, but you have to give users - including those who access the software on a server, like in a SaaS scenario - access to the source code of monklist, yes.
> i.e. monklist couldn't be closed source, but myapp that happens to use listmonk unmodified for sending emails can be?
There it gets murky. There are different understandings of when a software with different parts becomes one thing and thus the AGPL and GPL apply, and afaik that's not very well tested in court. That said, yes, if myapp just uses listmonk and is not a modified version of listmonk and listmonk is not an integral component of it, then myapp is not influenced at all by the AGPL. AGPL talks about "modified versions of software", myapp would be no such thing.
Pretty cool. Does the tool process bounces, complaints, and unsubscribes, e.g. auto removes from future mailings?
Have you considered using a hosted platform that uses Amazon SES rather then hosting Sendy or other tools yourself? I own BigMailer and our clients with established SES accounts (high daily rate limits > high limit per sec) see 1m emails go out in a little over 1 hour.
I always wonder why people choose to host email software in house and it seems like the requirement for direct integration with in-house system(s) is one of the biggest considerations when choosing in-house vs. off-the-shelf solution.
Try to implement bounce processing soon - an average list decays at 2% per month (3% for B2B lists) so in a few months, the bounce rate can be high enough for mail box providers to start blocking your emails due to attempts to send to large number of invalid email addresses. Especially sensitive issue with your list size and sending volume.
This looks great and is maybe exactly what I'm looking for! I have a couple questions:
I don't see anything in the docs about supplying an unsubscribe link. I don't see an endpoint in the API that could be used with an anchor tag in the footer of the email. Is this something that is supported?
And: my client is concerned about email/spam regulations, which I'm very unfamiliar with. Is there anything I'd need to look out for in this regard?
Thank you :) listmonk supports unsubscription and blacklisting. The default template that ships with it has an {{ UnsubscribeURL }} tag in the footer. I just realised that it's not in the documentation (work in progress). Will add.
- Performance. Unreasonably long amounts of time to send out e-mails. phpList degraded to the point of taking several days to process a campaign. listmonk can spawn N goroutines (~threads) and push e-mails to multiple SMTP servers. On a commodity ec2 instance, we're able to send 1.5mn+ e-mails in a couple hours.
- Subscriber imports were extremely slow. Direct integration to keep subscribers in sync with external CRMs was cumbersome. Direct DB inserts were complicated due to the complex table structures. listmonk imports 10k records/sec into a Postgres DB on a commidty ec2 instance.
- Segmentation. Often, we have to rapidly segment users by custom attributes and conditions and relay an update to them. listmonk supports SQL expressions to segment users on their attributes that are defined as arbitrary JSON maps (thanks to Postgres JSONB type).
- Unavailability of dynamic templates. listmonk templates support Go template expressions so it's possible to write logic in messages to make them dynamic.
I would like to add that listmonk is still work in progress and requires a number of essential features before it can come out of alpha.