Scalatra result announcer w/ various datastores

In previous post, i talked about an examination result announcer application in scalatra/scalate/scalaquery. I mentioned i would try the app with different datastores in another day and post the results. That day has arrived at last.

Since i’ve talked about the system before I’ll keep things short this time and jump straight to results.

			RPS	p90
In memory:		591	238
Voldemort (0.90):	552	256
Mongo (1.8.2):		548	255
Redis (2.2.12):		523	265
Cassandra (0.8.2):	504	273
HBase (0.90.3):		471	285
MySQL (5.5.15):		453	346

In memory storage means i used a scala.collection.mutable.Map object in my Controller to collect the results. The result above was measured with scala 2.9 parallel collections. Without them, the numbers were slightly smaller.

All 3rd party storage solutions were working on localhost with their default configurations. All of them have been accessed with preferred or well-known drivers. 

I did not try to optimize my code as it is another one of my goals to measure how easy it is to get best performance with little effort.

As seen in the chart, voldemort and mongo are virtually the same in terms of these simple performance measurements. I guess this is because both the storage systems work directly off memory. Since i everything is configured to defaults, voldemort was using in-process BDB-Java, and mongo was using memory-mapped files (i guess). On the other hand, while being an in-memory system too, redis missed their performance with a small amount. I feel some sort of configuration requirement there.

Cassandra and HBase with their BigTable like storage mechanics lag behind others with small margins. I don’t know much about Cassandra but running a pseudo-distributed HDFS and an HBase (and on the same computer as the application and jmeter) is highly discouraged. And, i guess i know enough HBase to say that it is not the perfect fit for this example application. Results simply reflected that.

Since my linux has updated itself several times in last two months, i ran mysql tests in order to keep things fair. You might have noticed its performance is also increased, but not to the point where it may compete with others.

All the code is on github. Switch branches for different stores.

NOTE: Those system updates effected my couchdb too, which upgraded it to 1.1.0. Long story short, my application running on it outperformed everything on the list with 649 requests-per-second and 90% of requests were under 201ms.

NOTE: Another thing i discovered was if i were to use rewrites in couchdb, it damages performance to 550 RPS and 230ms p90. Interesting…