ColdFusion MailSpoolService Performance
In my last article about the Adobe ColdFusion MailSpoolService I mentioned that I was going to try to get specifics on expected performance in the Standard Edition vs Enterprise edition of the MailSpoolService. Adobe has not respond to my requests with actual data. While attending the ColdFusion Summit 2017 I tried to get a clear answer from any of the Adobe ColdFusion engineering team members that were at the conference. They didn't know the answer. Because I didn't get the response I wanted from Adobe I decided to start testing.
My first test was to setup a Windows VM with ColdFusion 11 installed with a standard license. I also created a simple CFML page that uses CFMAIL to send an email with a CFLOOP to send that same email a lot of times. To make this a more realistic test I made up a new disposable email address on our mail server at CF Webtools and sent the emails from my email server on AWS. This means that the ColdFusion MailSpoolService has to actually communicate with a mail server. SMTP connections can at times take time. The emails I generated have several paragraphs of Lorem Ipsum text to simulate actual email sizes. My first test was to verify one email did indeed get sent. It did. The next test was to send 1000 emails while timing with my iPhone's stop watch. We also have ColdFusion 11 Enterprise which meant I was able to test the performance against the Enterprise Edition. Lastly, I was asked to test on the Developer Edition because it is often stated that the Developer Edition is essentially Enterprise Edition with a two connection limit. I ran this test a couple times each from ColdFusion 11 Standard, ColdFusion 11 Developer, and ColdFusion 11 Enterprise servers.
Standard Edition
It took approximately 23 minutes to process 1000 emails in the mail spool. This comes down to about 44/45 emails per minute. Which works out to about 11/12 emails per 15 second pooling interval or 2600 email an hour. Which is a little more that 60,000 emails per day processing 24 hours straight without any connection issues. That's not too shabby for being the single threaded version of the MailSpoolService.
Developer Edition
After running the same tests a couple times in Developer Edition I got the exact same results as I did for Standard Edition.
Enterprise Edition
This is where you can say "You get what you pay for!". Before I go into the numbers let me also remind everyone that the Enterprise Edition of the MailSpoolService is multi-threaded and you can specify the number of threads. I think the default is 10 threads. This setting is in the Mail section of the ColdFusion Administrator Enterprise Edition ONLY in the sub section "Mail Spool Settings". There is nothing that indicates that there is a maximum number of threads. My tests are with 10 threads.
I had to run this test several more times just to make sure I saw what I saw. All 1000 emails were sent in a single polling of the mailSpoolService. That's 1000 emails sent in under 15 seconds. I ramped it up a bit and sent 5000 emails. This time it took two polling intervals and sent 5000 emails in about 30 seconds. To get absurd I increased the test to 10,000 emails and the Enterprise Edition cleared those out in less than 60 seconds. This means it took 4 polling intervals to process 10,000 emails which comes out to 2,500 every 15 seconds with 10 MailSpoolThreads. I wanted to verify this exactly so I decreased the polling interval from 15 seconds to 30 seconds. I wanted to fill the mail spool completely beforehand and see how many emails were processed on each polling interval. What I saw is that I'm not nearly at the limit of what the Enterprise Edition MailSpoolService can handle. By slowing down the polling interval my CFML script was able to put all 10,000 emails into the mail spool folder before the MailSpoolService started processing. Then it happened, all 10,000 emails were process in one single polling interval of less than 15 seconds time. I'm not sure were the limit is, but it's fairly clear that the Enterprise edition can send more emails than most of us will ever need. Even if you're running a bulk mail service.
Summary
My results are not scientific. However, I do believe they are closer to what real people will see on real servers based on my experience with hundreds of servers. It would be really nice if Adobe would respond with some real numbers so we could help clients decide if this feature is worth buying Enterprise Edition instead of Standard Edition. However, based on my testing, if sending emails is your high priority and the amount of emails is going to be over 50,000 emails per day then you might want to weigh the cost of an Enterprise license.
Note:
The reason I was testing on ColdFusion 11 is this is the version that several different clients have that are having issues with the MailSpoolService. I think I know that for one client they really are trying to send near or over 50,000 emails per day and this is why they thought there was an issue with the MailSpoolService.
9 Comments
-
If you need to send large quantities of mail fast, then I recommend taking a look at mailgun or SendGrid. You can use their API to send out emails instead of SMTP, thus bypassing the cfmailspool and SMTP process entirely. This is significantly quicker if you are sending out the same email to all address as I believe that will be done with a single API call. Plus you have the advantage of their bounce processing, reporting and other useful tools that you don't get with cfmail. -
You should compare the perform of using a local SMTP service on the same server as ColdFusion. (We installed Microsoft IIS SMTP Service and sent email without having to use authentication when generated messages on the same box.) We've found this process to be a lot faster than repeatedly connecting to a third-party SMTP server using a single ColdFusion thread.
In the past, we also had issues with losing occasional messages w/both Standard & Enterprise. We would log the data, send and then... nothing. Messages wouldn't even end up in the CF SMTP log.
In the end, we migrated our sending to go through SparkPost (similar to SendGrid that Russ is recommending). It's a lot easier to generate one message and provide custom variables to be "mail merged" on their end than it is to personalize and generate them all using CF. SparkPost offers both SMTP & Rest API, but if you choose SMTP, you'll need to restart ColdFusion alot because their IPs change and ColdFusion tends to cache the IP forever. -
I believe Enterprise is capable of keeping the SMTP connections open between "sends". This would make it faster. I haven't monitored it, but I'm guessing Standard has to open a new connection, authenticate, send, close and repeat while sending via SMTP.
-
Sorry... I should have stated that we use Microsoft IIS SMTP Service to "relay" email to our dedicated mail server.
-
James, Actually for the server that prompted me to run these tests, it is using Microsoft IIS SMTP to relay mail and it is NOT getting any faster results sending mail from the Standard Edition. Also, I did review the SMTP logs on the server that was having 'issues' and found that a new connection was being made for each mail sent from ColdFusion 11 Standard Edition. It does appear that the mailSpoolService in Standard Edition is very restricted. My whole point of this has been to try to get actual facts from Adobe as to the mail spool performance levels in Standard vs Enterprise. That data has been elusive so I did basic testing to see what I could come up with.
Regards,
Wil -
Russ, I'm well aware of other resources for sending email outside of ColdFusion. My whole point of this has been to try to get actual facts from Adobe as to the mail spool performance levels in Standard vs Enterprise. That data has been elusive so I did basic testing to see what I could come up with.
Regards,
Wil -
It does seem like CFStandard is intentionally "hobbled". As long as CF has been out, you'd think a feature like SMTP connection caching would have become standard and not something that requires additional licensing. Modular pricing would be nice. I'd pay a little extra to enable non-hobbled SMTP & PDF features, but don't need many of the other "enterprise" features (API Manager, etc).
I recall that back in the CF5/6 days (~2002) that some developer blog about generating similar CFMail files and used a different program to monitor and perform the sending process. I did manage to find this article from 2002 that recommended writing text files directly to the mail server's outbound queue folder, but this isn't what I recall reading before. (Many CF-related blogs are no longer available.)
http://coldfusion.sys-con.com/node/41634 -
Wil, this is great stuff. Thanks so much.
Might you be in a position to try this again against CF2016 or 2018?
It would be very interesting to hear if you experienced the same seeming limitation in those, with their latest available updates. (Also, it may have been interesting to note what update of CF11 you were on when you did this test, in case that could have had an impact.)
That slowness in Standard is really something I was surprised to see. I thought I'd seen over the years people reporting faster processing, though I've not seen any other specific testing. While it's true that the ability to set or increase the number of mail threads is Enterprise-only, Adobe has never documented either how many mail threads the Std version has nor if there was a limit to how many emails were sent per spool interval.
In that regard, your research here is most helpful, and that's why I'd love to hear news relative to later versions (or perhaps a more updated CF11, even). There's always the chance that what you saw was the result of a problem that was subsequently fixed.
I was also really surprised to see that the Developer edition didn't fare as well as Enterprise, but instead operated like Standard in your test. We've always been told that Developer is "just Enterprise, with a limit of concurrent requests allowed", but it would seem that it might well have some "standard" limits, and it would be good to know, relative to this at least (and someday relative to all features that Std might throttle).
So do you think you might easily resurrect your test and run it again on later versions? :-) -
Charlie,
Adobe makes it public that the Enterprise version has a better email engine. The ColdFusion 2018 buying guide (https://www.adobe.com/products/coldfusion-family/b...) lists this item "High-scalability email engine" for Enterprise only. This has been true for all previous ColdFusion versions back to 10. Prior to 10 I can't remember. (BTW: I did this type of testing way back on ColdFusion 5, but I'm too old to remember those results ;-) I have also tried numerous times in comments, emails, messages on Slack, and in person at CFSummit 2017 to get a better definition of of the mail spool performance from the ColdFusion team at Adobe. They have been very elusive at best on the subject.
From what I have been able to figure out the email engine or mail spool is single threaded in Standard edition vs multi-threaded in Enterprise. This is evident in Enterprise edition because you can define the number of threads in the CF Administrator under "Mail Spool Settings".
I'll see if I can get some tests running for both editions of ColdFusion 2016. I don't have access (yet) to ColdFusion 2018 Enterprise. But I can test the Standard edition of ColdFusion 2018.
Regards,
Wil
Sites to Watch
Aggregators
RSS
