ColdFusion Muse

ColdFusion Exploit in the Wild

On September 11th of 2018 Adobe released a critical security patch to patch a very dangerous flaw (CVE-2018-15961) that could allow an attacker to upload a file that can be used to exploit and take control of the server. Adobe updated their security note to alert everyone that there are active exploits in the wild.

"UPDATE: As of September 28, Adobe is aware of a report that CVE-2018-15961 is being actively exploited in the wild. The updates for ColdFusion 2018 and ColdFusion 2016 announced in this bulletin have been elevated to Priority 1. Adobe recommends customers update to the latest version as soon as possible." - Adobe

Today it is being reported by multiple news outlets including ZDNet that the exploit is in the wild and being used by a nation-state cyber-espionage group.

"A nation-state cyber-espionage group is actively hacking into Adobe ColdFusion servers and planting backdoors for future operations, Volexity researchers have told ZDNet. The attacks have been taking place since late September and have targeted ColdFusion servers that were not updated with security patches that Adobe released two weeks before, on September 11." - ZDNet

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them or if they are not familiar or knowledgeable with ColdFusion contact us at CF Webtools to patch your servers. Our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to "operations at cfwebtools.com".

ColdFusion Debugging on Production

Today's short note is brought to you by "Don't Do That On Production!" At CF Webtools often times we get called in to help troubleshoot servers that are failing to perform well. We often hear the same sort of symptoms that goes like this. The server has been running fine for months then suddenly for no reason it's slow, CPU usage is high, and it hangs or crashes multiple times per day. This always prompts us to ask the same question. "What was changed just before these symptoms started?" And the answer is usually "Nothing was changed (as far as they knew)". In all reality the person we're talking to may not the be only person with access to make changes to the server. Or they may not in fact have access at all and they are relying on information provided to them by an IT team member. We take notes, assume nothing, and question everything (on the server).

We had this scenario play out a few times in the past few weeks with three servers from three different companies. The reason I'm writing this note is the same problem occurred on each server. The short answer is someone enabled ColdFusion Debugging on the production server. ColdFusion is a very powerful rapid development platform, but it has a few gotchas if you are not careful. Such as enabling debugging on a production server. Debugging output provides a massive amount of information and for obvious security reasons we never want this enabled on a production server. Yes, I know you can restrict debugging output to a certain IP address, but that does not prevent the debugging output from being generated. It's just not displayed. The generation of debugging output takes more CPU power and at times more JVM memory. On a low load web application you may not notice a difference. However, on a high load, high traffic production web application the extra resources needed to generate the debugging output may in fact cause all those symptoms described above.

In each of the cases we saw these past few weeks, we were reviewing the servers settings, looking at the results of Fusion Reactor, and reviewing ColdFusion settings. On the first server we almost missed the fact that debugging was enabled. By the time we were troubleshooting the third server with similar symptoms we were checking to see if debugging was enabled before we did anything else. Disabling debugging resolved the bulk of the performance issues. We then used this time to review each server and offered additional performance tuning recommendations based on each servers resources and application needs.

This falls into the category of "Don't Do That On Production!" Please leave debugging to your development and staging servers.

CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

ColdFusion Security updates for ColdFusion 2016 and ColdFusion 11

Adobe released important security updates and big fixes today, update 6 and update 14 for ColdFusion 2016 and ColdFusion 11 respectively.

These updates resolve an important insecure library loading vulnerability (CVE-2018-4938), an important cross-site scripting vulnerability that could lead to code injection (CVE-2018-4940) and an important cross-site scripting vulnerability that could lead to information disclosure (CVE-2018-4941). These updates also include a mitigation for a critical unsafe Java deserialization vulnerability (CVE-2018-4939) and a mitigation for a critical unsafe XML parsing vulnerability (CVE-2018-4942).

There is a bug of great importance to many that has finally been fixed. I've blogged about this before and I was able to create a work around to resolve this issue until it was fixed by Adobe. The SFTP/FTPS bug would not allow connections to secure FTP servers that utilized newer SSL protocols. When using CFFTP to connect to some S-FTP server, during connection, you can see an error message. This has been a growing issue as more and more companies replace plain text FTP servers with SFTP or FTPS servers that utilize stronger protocols.

For ColdFusion 2016 this update upgrades Tomcat to version 8.5.28 and OpenSSL to version 1.0.2n.

For ColdFusion 11 this update upgrades Tomcat to version 7.0.85 and OpenSSL to version 1.0.2n.

The security updates referenced in the above Tech Notes require JDK 8u121 or higher (for ColdFusion 2016) and JDK 7u131 or JDK 8u121 (for ColdFusion 11).

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them or if they are not familiar or knowledgeable with ColdFusion contact us at CF Webtools to patch your servers. Our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

*Note: ColdFusion 11 when it was first released came with a version of Java 1.7.0_nn. Adobe later re-released ColdFusion 11 with Java 1.8.0_25. If you have ColdFusion 11 still running on Java 1.7 I highly recommend that Java be upgraded to Java 1.8. Oracle is no longer supporting Java 1.7 and 1.7 is long past it's end of life. Even though the Adobe instructions for this current security update states that you can run Java 1.7.0_131, I highly recommend upgrading to Java 1.8. Personally I will not install Java 1.7 on a clients servers and sign off on it being 'secure'.

ColdFusion SFTP and FTPS Secure Connection Failure

I have seen a lot more people asking questions about making SFTP or FTPS secure connections from ColdFusion using the <CFFTP> tag. They are trying to figure out why they cannot make a connection. Often the error is "Algorithm negotiation fail" or "Connection Error". People are posting their questions on many support forums including Adobes forums and their new ColdFusion Community Portal. This is a problem people are experiencing in ColdFusion 10 and ColdFusion 11.

In the last few years we've seen a huge shift in SSL/TLS security including the removal of older less secure protocols and forcing secure connections to use the newer stronger protocols with stronger TLS certificates and stronger encryption cyphers. As such older systems need to be upgraded to handle the newer security protocols. More recently plain old unsecure FTP portals have been the focus of change to SFTP or FTPS.

At CF Webtools we've run into this same problem several times with multiple clients. It was so much of a problem that I needed to spend some dedicated time to see how we could resolve this issue.

The first thing I discovered is that this issue is a known "bug" that has been reported to Adobe. It's been a long known issue and somehow the fix which is in ColdFusion 2016 has not been included in an update for earlier ColdFusion versions. However, Adobe has affirmed to me that this fix is scheduled for an upcoming update.

Because it was fixed in ColdFusion 2016 I was able to inspect the included jar files to see if the one that handles CFFTP or secure communications was newer than the one(s) in ColdFusion 11. What I found is that jsch-0.1.44m.jar had been replaced by jsch-0.1.52m.jar. The JSCH jar library is the library that handles Java Secure Channel communications. "JSch allows you to connect to an sshd server and use port forwarding, X11 forwarding, file transfer, etc., and you can integrate its functionality into your own Java programs."

After seeing this was upgraded I had an ah-ha moment and figured it was worth a try to copy this newer version into my ColdFusion 11 test server and see what happened. The new version is in ./ColdFusion2016/cfusion/lib folder. You can download the free ColdFusion 2016 Developer Edition and install it anywhere so you can get access to the updated jar file. Once you have the new jar file copy it into ColdFusion 11. The proper way to do this is to remove or rename the old jar file version in your ColdFusion11/cfusion (or instance name)/lib folder then copy the new jar file version into the same folder. Then start or restart ColdFusion 11. That's it. You're done. The bug is fixed and you're good to go with SFTP or FTPS using <CFFTP> in ColdFusion 11.

This is not an approved fix from Adobe. I do not know if there is some unknown issue that could be created by doing this. However, I do know that everyone I've talked to that has tried this has had their secure FTP issues resolved. Additionally I have not tried this 'fix' in ColdFusion 10. However, if you are running into this issue with ColdFusion 10 it's worth the minimal effort to give it a try.

If you need someone to make this change on your ColdFusion server then contact us, we can help. CF Webtools is here to fill your needs and solve your problems. If you have a perplexing issue with ColdFusion servers, code, connections, or if you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations @ cfwebtools.com.

Cryptocurrency Miners Hacking Servers

Are your free CPU cycles making others rich? There's a chance they are and it's at your expense. A recent article at Vice.com states that "At Least 1.65 Million Computers Are Mining Cryptocurrency for Hackers So Far This Year". If this is to be believed then it's possible a server you are running has been compromised and is actually mining cyrptocurrency for the hackers.

Cyrptocurrency is an anonymous, digital currency that is supposed to be untraceable. It's used on the internet to purchase more and more products and services. One of the most common forms of cryptocurrency is Bitcoin. This is from the Wikipedia entry on Bitcoin.

Bitcoin is a worldwide cryptocurrency and digital payment system called the first decentralized digital currency, since the system works without a central repository or single administrator. It was invented by an unknown programmer, or a group of programmers, under the name Satoshi Nakamoto and released as open-source software in 2009. The system is peer-to-peer, and transactions take place between users directly, without an intermediary. These transactions are verified by network nodes and recorded in a public distributed ledger called a blockchain. Besides being created as a reward for mining, bitcoin can be exchanged for other currencies, products, and services. As of February 2015, over 100,000 merchants and vendors accepted bitcoin as payment. Bitcoin can also be held as an investment. According to research produced by Cambridge University in 2017, there are 2.9 to 5.8 million unique users using a cryptocurrency wallet, most of them using bitcoin. ...

Bitcoin Mining is a record-keeping service that runs on peoples computers, servers, or specialized Mining Devices, that are setup by individuals to help process Bitcoin transactions. As a reward for doing this you are given newly created bitcoins and transaction fees. ie. You can make money by mining for Bitcoin.

This reward is enough that hackers have taken it to the next level and started hacking servers around the world so they can install mining software and use YOUR computers and servers to make money for themselves. Just this week it was discovered that some of Showtime's web servers were mining cryptocurrency. This isn't a new thing either. Back in 2014 Iowa State University servers were also hacked for the purpose on mining Bitcoins. These are not isolated occurrences. They are happening regularly. This practice is free to the hackers an costly to the owners of the servers. Here's why.

Case Study

CF Webtools has seen this type of hack in the real world. We recently had a company come to us seeking our services for both Server Administration and ColdFusion programming. Part of taking this new company on as a client we performed a security review on all of their servers. They also had existing issues that we needed to look at in particular. One of their web servers was rebooting multiple times per day at what seemed like "random" intervals.

Upon review we found the web server was always running at 100% CPU usage with no services claiming to be using that much CPU power. Certainly not ColdFusion or IIS. After completing additional research we decided to install a malware removal tool and scanned for malware. It didn't take long to find that indeed there was malware running on the server. What we found surprised us only because we had not seen this in action before. It was a cryptocurrency miner and it was so intensive that it would crash the server. All attempts to remove the malware failed. It would end up back on the server in a short period of time. The fact is this server was compromised. To resolve the issue we sent one of our decommissioned, but powerful servers, preinstalled with a clean OS to their data center. Then our Operations Manager went on the road to install the new server as well as a physical firewall. We essentially rearchitected their entire server setup. Meanwhile the malware removal tool did it's best to keep the malware at bay while I recreated their web server on the new server. It was a busy week (or more), but we were able to clean the code on the clients server and put that on the new server. We also had to research and rebuild all the dependancies from scratch. When it was all said and done we replaced the compromised server with the new one and put all their servers behind a Cisco ASA.

This case of Hacking for Bitcoins proved costly in the end to the company who's systems were compromised all while providing a free profit to the hacker(s).

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com.

ColdFusion Security Patches ColdFusion 11 Update 13 and ColdFusion 2016 Update 5

Adobe just released security updates for ColdFusion 11 and ColdFusion 2016. This is a critical security update and you should be updating your ColdFusion servers. The information below is from the CF Webtools operations group. If you need help upgrading your VM or patching your server (or anything else) our operations group is standing by 24/7 - give us a call at 402-408-3733, or send a note to operations at cfwebtools.com. Meanwhile, this info below will help IT staff and DIY types get started.

With ColdFusion 11 Update 13 and ColdFusion 2016 Update 5 there are additional manual updates that are required to complete the security patch. The additional requirements are the same for both ColdFusion 11 and ColdFusion 2016 and the remaining information pertains to both versions. Both updates require that ColdFusion be running on Java version 1.8.0_121 or higher. For reference, ColdFusion 11 comes with Java version 1.8.0_25 and ColdFusion 2016 comes with Java version 1.8.0_72. The Java that needs to be installed is different from the "Windows User" Java client that may already be installed. The installer is available from Oracle. Once the new Java version installed, the jvm.config file for each ColdFusion instance needs to be updated to point to the new Java version installation path. If you're running the Enterprise version of ColdFusion, there's a likely chance there is more than one ColdFusion instance running.

Part of the instructions from Adobe says that if your ColdFusion server is installed as J2EE server then there is an addiction manual configuration that you ned to do. However, every installation of ColdFusion since the release of ColdFusion 10 is a J2EE or JEE installation. What Adobe really meant was that if you are using a third party JEE server and not the built-in Tomcat JEE server.

If your ColdFusion server is running on a third party JEE server such as WebLogic, Wildly, custom Apache Tomcat, etc (Not the built in Tomcat that comes with ColdFusion), then the following step needs to be completed.

Set the following JVM flag, "-Djdk.serialFilter=!org.mozilla.** ", in the respective startup file depending on the type of Application Server being used.

For example,

• On Apache Tomcat Application Server, edit JAVA_OPTS in the 'Catalina.bat/sh' file
• On WebLogic Application Server, edit JAVA_OPTIONS in the 'startWeblogic.cmd' file
• On a WildFly/EAP Application Server, edit JAVA_OPTS in the 'standalone.conf' file

This is one more friendly reminder to make sure your ColdFusion servers are patched! Either patch them yourself, have your hosting provider patch them or if they are not familiar or knowledgeable with ColdFusion contact us at CF Webtools to patch your servers.

As always, if you need help migrating to the next version, scanning your ColdFusion server for security vectors or installing this patch and new Java version, contact your project or account manager directly, or send an email to support@cfwebtools.com. You can also simply respond to this email (or call 403-408-3733).

*Note: ColdFusion11 when it was first released came with a version of Java 1.7.0_nn. Adobe later re-released ColdFusion 11 with Java 1.8.0_25. If you have ColdFusion 11 still running on Java 1.7 I highly recommend that Java be upgraded to Java 1.8. Oracle is no longer supporting Java 1.7 and 1.7 is long past it's end of life. Even though the Adobe instructions for this current security update states that you can run Java 1.7.0_131, I highly recommend upgrading to Java 1.8.

ColdFusion and JVM Versions and SSLv3-TLS Security Magic

This is the second entry by Wil Genovese (Trunkful.com) in our effort to provide a complete picture of how CF, Various versions of JVMs and various versions of SSL all work together. Wil's previous article on Surviving Poodle detailed a blow by blow description of how to troubleshoot a system broken due to the upgrading of SSL. This article includes some detailed technical information as well as the results of some painstaking tests. It is our hope that it will serve as a guide. It represents yet another reason to insure that you are upgrading to the latest JVM and CF version. Take it away Wil:

Read More

Surviving Poodle - ColdFusion and SSL 3

There's been a great deal of buzz about poodle. Poodle is an SSL exploit capable of highjacking a session using a browser's ability to "negotiate downward" the level of SSL it uses. It's recent prolifieration has put some urgency into the efforts to force existing applications and platforms to deny the use of any standard of SSL less than version 3.0. Super guru Wil Genovese (Trunkful.com) recently did some troubleshooting on a ColdFusion server with an issue related to this necessary configuration step. Wil writes:

We ran into an issue when a company contacted us at CF Webtools because ColdFusion was suddenly no longer able to connect to their email providers mail servers. One day ColdFusion was sending emails to their clients just fine and the next day it was failing. As you know these issues are usually best resolved by asking "What changed?" As far as the client knew, nothing had changed - but we knew enough not to stop digging.

Read More