While preparing to upgrade our production server from ColdFusion 9 to ColdFusion 11, I noticed some terribly poor performance under load testing. I’ve spent a considerable amount of time investigating the cause, and have tracked it down to the amount of time it was taking ColdFusion to open a connection to Oracle.
To be clear, this may not be a problem with ColdFusion itself. However the issue only became apparent when we started using CF11.
Our environment is ACF11 with Hotfix 3, deployed as a war file on Apache Tomcat 7.0.57. The Java version is 1.8.0_25 and the Operating System is RHEL 5.x (64 bit). Our database is Oracle 11g.
Here’s the basic test case to reproduce the issue. Thanks to @TransientAaron on Twitter for the assistance with this.
Assuming you have an Oracle datasource called “test”, create a new directory under your web root and add the following files:
Then use Apache JMeter to hit the page with multiple concurrent request (I gave it 2 iterations of 200 threads with a ramp up period of 25 seconds).
Here’s the initial results:
Only a handful of the requests actually completed before timing out after 60 seconds.
At this point I opened the CF monitor, and saw there were actually 25 Oracle connections open. So I took a look at what would happen if I ran it again:
Note that this second graph is cumulative. The bit to look at here is the almost solid black line near the bottom, showing the responses to the majority of requests now taking only a few hundred milliseconds.
So I think it’s fair to say that I’ve proved that establishing connections is bad (or much worse than bad). But once they’re established, things are fine.
So what is happening here? After a lot of fruitless searching for results related to ColdFusion and slow Oracle connections, I found a question on Stack Overflow relating to slow connections in Java, which provides us with the solution: http://stackoverflow.com/questions/5503063/oracle-getconnection-slow.
Apparently, creating Oracle connections depends on generating random numbers, and that is sloooooow. BUT, we can trade off (a bit) of randomness for (a lot) of speed by using an alternative generator. This is done by adding in the JVM parameter:
@elishdvorak also pointed out on twitter that the performance of cfhttp can similarly be affected:
So let’s repeat our tests with this new JVM parameter.
(note that the graph scale has changed considerably, because it is magnitudes faster)
I think we’re now ready to put CF11 into production.