ColdFusion Muse

ColdFusion Debugging on Production

Today's short note is brought to you by "Don't Do That On Production!" At CF Webtools often times we get called in to help troubleshoot servers that are failing to perform well. We often hear the same sort of symptoms that goes like this. The server has been running fine for months then suddenly for no reason it's slow, CPU usage is high, and it hangs or crashes multiple times per day. This always prompts us to ask the same question. "What was changed just before these symptoms started?" And the answer is usually "Nothing was changed (as far as they knew)". In all reality the person we're talking to may not the be only person with access to make changes to the server. Or they may not in fact have access at all and they are relying on information provided to them by an IT team member. We take notes, assume nothing, and question everything (on the server).

We had this scenario play out a few times in the past few weeks with three servers from three different companies. The reason I'm writing this note is the same problem occurred on each server. The short answer is someone enabled ColdFusion Debugging on the production server. ColdFusion is a very powerful rapid development platform, but it has a few gotchas if you are not careful. Such as enabling debugging on a production server. Debugging output provides a massive amount of information and for obvious security reasons we never want this enabled on a production server. Yes, I know you can restrict debugging output to a certain IP address, but that does not prevent the debugging output from being generated. It's just not displayed. The generation of debugging output takes more CPU power and at times more JVM memory. On a low load web application you may not notice a difference. However, on a high load, high traffic production web application the extra resources needed to generate the debugging output may in fact cause all those symptoms described above.

In each of the cases we saw these past few weeks, we were reviewing the servers settings, looking at the results of Fusion Reactor, and reviewing ColdFusion settings. On the first server we almost missed the fact that debugging was enabled. By the time we were troubleshooting the third server with similar symptoms we were checking to see if debugging was enabled before we did anything else. Disabling debugging resolved the bulk of the performance issues. We then used this time to review each server and offered additional performance tuning recommendations based on each servers resources and application needs.

This falls into the category of "Don't Do That On Production!" Please leave debugging to your development and staging servers.

CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

A Whole Heaping Helping of Normal

When I'm called into a data center or large application to look at a ColdFusion performance problem I often find myself in a "war room" with highly skilled technicians and admins who may or may not know much about the JVM runtime. One of the things that require experience is knowing what normal memory patterns look like for a typical production heap. Windows admins will often go directly to the task manager or resource monitor to examine memory usage. If they are uninitiated the first thing they say is "Wow... Jrun is using a lot of memory." This often doesn't sit right to them, particularly if the site is idle. So this post is to help those windows folks (and Linux as well) to an understanding of what normal heap activity looks like.

Dynamic Compression on ColdFusion 9 and IIS7

Maybe your already know that web servers can compress outgoing content. Compressed content arrives at the browser which decompresses it and is able to render it. This is all generally seamless to the user and results in a more effective use of bandwidth. Now, compressing static files (like .html files) is a no brainer for web servers. They simply pre-compress the files and store them in a file cache somewhere. When the original file is called for the web server serves up the compressed file instead.

Dynamic files are more problematic. There's no correlation between the file name and the buffered output of a ColdFusion page for example. Consider search results. One user might receive 10 results and another user might receive 10 completely different results. Still another user might receive 100 results. How is the web server supposed to compress that data? Like your app server it does it "on the fly". It waits for ColdFusion to return the response buffer, compresses the file in memory (as I understand it) and then outputs the buffer to the browser. At least that's the way it works in theory. In practice you might find that ColdFusion 9 and IIS 7 don't quite have this figured out yet.

Before I give you the blow-by-blow (and thankfully a solution) I want to make it clear that this problem and solution come to me by way of my good friend and colleague Vlad Friedman of Edgeweb Hosting. EdgeWeb consistently receives the highest possible reviews from its customers and Vlad is one of the brightest folks I know in our corner of the IT world. Now let's talk about our little problem shall we?

Query Caching Run Amuck: Know Your FIFO Buffer

Query caching is one of those underutilized features of ColdFusion that can exponentially speed up your application. It is also one of those misunderstood features that, when used incorrectly can be very disappointing. Let me say at the start that the Muse believes you should use query caching. If you don't believe I'm a fan then check out my post titled, Good Developers Practice Safe Query Caching. It's not a panacea, but it definitely has it's uses. Almost every application has some queries that can be cached - and saving round trips to the database is the holy grail of application tuning. But in this post we want to talk about naughty developers who cache irresponsibly... developers who do not understand the nature of the FIFO buffer.

Webmaniacs Live - Mike Brunt and CF Server Tuning

CF Optimization Guru Mike Brunt gave an awesome presentation on tuning ColdFusion. In a short 50 minutes he covered such topics as JVM configuration, multi-server install and load testing. It was an excellent overview. One recommendation (that he made in is lyric British access) was to install See Fusion or Fusion Reactor instead relying on the built in monitor. His take was that, because the monitor is running under ColdFusion, it is too easily become affect itself by performance - or perhaps could become part of the problem instead of being part of the solution.

I had thought, based on a presentation by Adam Lehman on CF 8 a year ago, that the monitor ran in a different JVM space than CF. One of the things I have noticed with the monitor always shows a Flex gateway thread in the request monitoring. That thread is the monitor itself. So I'm thinking Mike might be right. Perhaps the server monitor does exist "inside" of ColdFusion. I like the new server monitor and I've found it useful. But I really like SeeFusion's default view which shows requests and visuals of the heap on one page. I think it is a better aggregate of the things I want to know immediately when I'm looking at a performance problem.

Verbose Garbage Collection

Another excellent tip is a "how to" on enabling verbose garbage collection. This is easy to do. You simply add the following arguments to the JVM.config file and restart ColdFusion.

The information is going to be stored in a log file in the default logs directory (like /runtime logs). How would you use this? One of the best ways to use it is to figure out when new generation collection is going on, when tenure (old "stop the world") collection is going on, and how much data is going from new to old. You can get a good picture of what is going on by examining this log information and using your imagination. Check out Mike's blog on GC tuning for a good outline of the process.

Scheduled Task Performance

So you have a template that runs fine when you are loading it into a browser but runs like a water buffalo on Percodan when it's run as a Coldfusion scheduled task? Here's a tip I saw on CF-Talk that serves as a good reminder. Take a look at the debug settings. By default, debugging includes the address 127.0.0.1 (often called the "loopback" or "local" address). Coincidentally this is the very IP address that the Coldfusion Scheduler will report as the calling address (cgi.remote_addr) - meaning the template in question will generate all of that lovely debug information. This can result in an otherwise acceptable level of performance turning painfully slow. Muse rule of thumb - disable debugging in production environments. Ok... so sometimes I enable it briefly to get some information when I'm troubleshooting a specific process, but it should always be off during normal operations.

Let me add as an anti-thesis, turn on debugging in a development environment. It's surprising how many developers work without it. Personally, I spend more time looking at the debug info than the actual output. One of the first things you should do when writing a new query for example, is check it out in the debug. How many records returned? How long did it take? Can I make it better? One of Coldfusion's big advantages (in my view) is the copious amount of debug information provided for you with each request. Use it!

Coldfusion Optimization and the Windows Legacy

Coldfusion has been navigating the Internet waters in the good ship Java for some time now (6 or 7 years I think). If you are old enough in Internet years to remember Coldfusion 5 you probably know that Coldfusion was originally written for windows in C++ and ported over to Linux, Solaris and UNIX. These ports were not reputed to be particularly good and Coldfusion was largely considered to be a windows server application (and probably justifiably so). In 2002 with "CFMX" Coldfusion offered the Web world an application that was not just a windows application with a port to other platforms. Instead it was a truly cross-platform effort. Since that time Coldfusion has found it's way onto Linux in increasing numbers. In fact we are seeing more Coldfusion on Linux than ever before - particularly users who need Coldfusion Enterprise.

And why not? Coldfusion runs splendidly on Linux. We use CentOS at CF Webtools and it sings along happily with very few issues. In fact, it is possible to allocate about 50 percent more memory to the heap on a Linux machine (using the 32 bit JVM). That is a significant advantage that CF on Linux has over CF on windows. Here's a blog post by CF Webtools own Linux guru Ryan Stille on that topic. Meanwhile, here's a take on the server optimization from the good old days to today:

Making String Comparisons Faster Using Javacast

In my Previous Post on the subject of variable comparison I tried to use JavaCast() to influece the comparison operator being used. I was attempting to keep Coldfusion from having to ask "Is it a float" or "Is it a string" or "why doesn't Ray Camden ever win People's 'sexiest man alive'". I thought that if I could pre-cast variables into Java objects, Coldfusion could use that information to cut to the chase and know exactly which operator to use without testing. Alas, that proved to be false hope. If you read the post you will already know that the difference between comparing typless and typed data in Coldfusion is not worth the effort.

Well as usual, smarter people than I (than me?) read this blog. The amazing Mark Mazelin (who's payment gateway API looks very interesting) suggested that I try to use the comparison operator directly, as in "object.compareTo()". I sometimes forget about all the nifty methods and properties attached to these objects when they are created. I set about testing his idea.