February 2012
1 post
GPUs, Hadoop and Testing Scalability
As i told numerous times before, i am currently trying to get some GPU powered image processing application to run on Hadoop. In development phase we were using a cluster of 12 machines with one Nvidia GTX 480s each, but since we are launching in a few months, we had to do some tests on our production cluster of 25 machines with two Nvidia Tesla M2050s each. In this post, i’ll try to sum up...
December 2011
1 post
Java web services without (explicit) code...
Finally… it’s been some busy weeks which i constantly sat in front of the computer in first one and constantly moved around them in the others. Finally i found some space to finish what i started. But space is not too much so i’ll keep this short.
Previously i talked about some funny web services stuff, and finished with a problem concerning exception handling. In short form:...
November 2011
2 posts
Java web services without (explicit) code...
I don’t know you but i hate code generation. Bytecode generation may sometimes be useful, but kills debugging capabilities so should be avoided most of the time. Source-code generation on the other hand, i simply fail to understand the necessity. If some 3rd party library will write the codes i will run, why can’t i simply let the library do whatever it needs over some sort of an API?
...
Using ivy and maven together
It’s not logical, highly unnecessary and probably expensive. But anyhow we found ourselves in that environment no matter what. Problem stemmed from the fact that eclipse/RCP dependency system being incompatible with virtually everything out there. We were using ant/ivy and pretty happy with it but our UI side found no easy way of headless-building their application using it. Eclipse is...
October 2011
1 post
Apache ODE and CLOB issue
I took-over some responsibilities from a recently departed collegue, and with it, i was kinda forced to turn back to JEE world. Not exactly the same technologies and frameworks i am used to, but once you hate some part of something it is likely that you won’t enjoy the other parts.
Anyways, first assignment was to move some WS-BPEL processes from glassfish to Apache ODE. It sounds like it...
August 2011
2 posts
Hadoop MapReduce job statistics (a fraction of...
Well, this has been on my backlog for a while. The problem is extremely simple actually: when did a MapReduce job started processing? I need this info to report to my clients using my API, meaning redirecting them to the JobTracker’s web interface is not an option.
Everyone using hadoop for some time knows 0.20 is the version to use, and everyone developed something other than a WordCount...
Scalatra result announcer w/ various datastores
In previous post, i talked about an examination result announcer application in scalatra/scalate/scalaquery. I mentioned i would try the app with different datastores in another day and post the results. That day has arrived at last.
Since i’ve talked about the system before I’ll keep things short this time and jump straight to results.
RPS p90
In memory: 591 238
Voldemort...
June 2011
2 posts
Announcing results with scalatra
As other posts in the series mentioned, i am trying out some web frameworks and data stores with a small web application which would be used to announce the results of a hypothetical exam. You can find the details in the first post. Today, the example app will be of scalatra.
Scalatra is a sinatra-like lightweight web framework written in scala. For developers living under a rock for a few years,...
Resource synchronization on Hadoop clusters with...
Straight from where i left. GPUs are massively paralel in contrast to CPUs, hence for some parallel processes, they are damn fast. The benchmarks you see around showing performance increases over 100x are theoretically true. By theoretical i mean pure CPU vs GPU computing power. In other words, for an infinitely running computation, it is possible to get 100x more results with a GPU than you would...
May 2011
1 post
Resource synchronization on Hadoop clusters with...
“We need zookeeper to run HBase”. Until past week that was basically my view of zookeeper. It is a distributed configuration and coordination service, HBase requires it so we have to put it in cluster. For a size of our cluster 1 instance seems to be OK but we are running 3 instances. These four sentences pretty much summed up what i knew about it. Fortunately i had looked to its main...
April 2011
1 post
LZO vs Snappy vs LZF vs ZLIB, A comparison of...
Now and then, i talk about our usage of HBase and MapReduce. Although i am not able to discuss details further than what writes on my linkedin profile, i try to talk about general findings which may help others trying to achive similar goals. This post is about a recent research which tries to increase IO performance for our MapReduce jobs.
Why Compression?
HBase documentation and several posts...
March 2011
3 posts
And the winner is ...
It was harder than i thought to convince my sysadmin to do anything or to let me do anything, so i continued with my own machine, with some simpler tests. Since i am thinking about using a number of languages/frameworks, i tried to write the easiest possible example for each of them and measure performance.
The easiest example was a web page writing 3 bytes, “123”, as the output. One...
Announcing results with/of couchdb
I have already mentioned about the scenario in the previous post. Now its time for some business, with couchdb as it is my favorite nowadays. The application is actually dead-simple. One screen asks you your ID and password, next one lists the results you had. In order to look like some real-life counterpart (not that one exists), i added some styling to the page, but still kept the size minimum....
Examining the examination results - Warm up
Consider this: You took an examination and your result will place you in a university of your choosing, or not. Although the date of the result announcement has already been made public, on the evening of the day before it, reporter on TV tells us: “2021 university acceptance exam results will be announced tomorrow morning at 10am. [Enters a video showing students taking exams and zoom-ined...
February 2011
2 posts
Compiling gcc-4.5 on debian unstable
As of today, i have found no way to install gcc-4.5 on debian unstable without manually compiling it. It seems there is a deb package in ‘experimental’ but it does not install because of a bunch of dependency issues. So i was back in old days where we need to compile several versions of gcc in chain in order to run apache on AIX.
I had forgotten how hard to these things right so i...
Monitoring nVidia GPU metrics with Ganglia
Editor’s Note: As the first time in sleepcoding’s short history, i welcome our sysadmin H. Çağlar Bilir’s post on our recent experiences with graphic cards and ganglia. I mentioned about our new cluster but omitted the detail that the machines also have some GPUs on them. New toys mean new details to look for. So here it goes.
After a brief period of googling, we could not find...
January 2011
2 posts
Notes from past week
Not focused on anything specific, last week was one of those i had to multitask, so there are more than one subject that deserves a few words. I also am in a state where i both want and don’t want to talk about them, which usually means the post will be more like my weekly reports rather than an article.
Our new shiny cluster of 12 machines arrived at last, just in time to put my hands on...
Activiti 'Hello world' on play!
Previously, i told about my experiences with play module/plugin development. I tried to point out how it was easy to extend the framework to your needs. A primitive activiti plugin was the product of that effort, and i said there will be a simple application using this. Here goes.
I picked the “Financial Report” example from 10 minute tutorial part of the activiti documentation...
December 2010
4 posts
A prototype play! plugin for activiti integration
Nowadays, i am back on play! trying to do things with somewhat less-documented feature of plugins. There is a project on the horizon which needs tight integration with a BPM engine, namely activiti, so i thought it’s a good time to try and deal with it as a play plugin or play module.
I said “plugin or module” deliberately, because i wasn’t aware of a plugin system other...
Scala for a newbie
Finally had the chance to get my hands dirty with scala. I’ve been trying to learn the stuff for almost a year now but you know… it’s not easy without something to build. Reading and trying the examples in the book doesn’t get one far unless he has an objective.
Anyways, what i was trying to do was to run some statistics over the performance data in some csv files. The...
The art of bug hunting
This post was actually going to hbase-user mailing list as a reply to this thread but discarded at last minute. It is a tale that simply shows what we -opensource product users- are capable of, as opposed to propriety product users, and may contain pieces of information about why RTFM is a canned response in community support paradigm.
First the technical details: Last week i needed to force...
Notes from past week
Heard hudson is moving away from java.net. They better. I lost at least an hour trying to reach guice-managed. All i wanted to was their 50 lines code in 3 classes. They missed guice stage in that @GuiceManaged annotation, and i was planning to add it. If only it was on github. Got the codes from koders.com. Moving on.
Still learning about javadoc. It seems you can’t put a {@link} inside a...
November 2010
4 posts
Logging is for ...
As we get closer to production, it finally came to the point where we needed to build our logging system. I don’t know if it’s the right approach but we deferred anything log-related while developing the actual code by writing only a few log statements and using a standard log4j.properties. Now is the time the get those on the table.
I didn’t want to introduce another logging...
The odds of that failure
The scenario i mentioned before, … it happened. Today around 2:15 pm. I was onto some MapReduce jobs, and just after one finished, our RegionServer that holds -ROOT- went down, with a power failure!
I’ll list the events and let you decide the odds:
A general power problem happening today (happens every week)
UPS failing a machine in a rack with 4 machines (never happened...
Lessons from a failure attempt
It all started on October 26th when our sysadmin (Çağlar Bilir) came to me and said he would restart one of the machines of our cluster due to some weird hardware problem. He needed confirmation on i wasn’t doing something fragile on it. I said OK and i’d like to seize the opportunity and kill all of its services and monitor cluster status in order to simulate a power failure. That...
An advanced form control with evently/couchapp
Finally wrapping it up. My objective was to create an invoice-like screen where it would basically consist of two parts: above part for general details like name, address and phone number; the below part for an area to list orders along with prices. No need to mention i am using couchapp and evently to get this done, meaning application works directly on couchdb.
The above part is a simple form...
October 2010
6 posts
Recursive evently widgets
Hmm… where was i? Yeah, that remove button. Before i dive into the details of how i did this, i wanted to identify the parts of a single order as order-generic and type-specific. This means there may be orders for registrations and accommodations in a participant’s order list, but are there any common points to these elements? Of course there were. No matter what type of order it is,...
mustache experiences - adding elements the easy...
Straight from where i left before, to the complications of using mustache.js directly in <script> blocks. For a simple element, like the one i mentioned in the previous post, you have no problems with mustache.
$('.orders').evently({
newregistration:{
render: 'append',
mustache: '<input type="text" class="reg"/>',
}
});
But since it’s that simple, you don’t...
evently experiences - Adding elements
Another less-productive weekend passed by but this time i’ve made some progress cleaning the house so it wasn’t a total disaster. On a side note, i’ve conquered some cities and built some farms in civ5 if those count for productivity in any way.
Last week, on the other hand, i had time to delve into evently to create some widgets to improve plevsy UI functionality. What i needed...
Unit-testing HBase Applications
For the past six months i’ve been on the road with hadoop and related technologies. Although this wasn’t a long journey i finally got to realize some aspects of my fellow traveller. But each one of them deserves another blog post, and this one’s only about an integral part which is unit-testing.
First of all, here is the situation: We have some tables that we usually...
My docForm replacement for CRUD on couchdb
I told about a primitive form-to-object encoder in my last post, mentioning that i used it in a couchapp. Well, that primitive encoder helped me build yet another primitive docForm replacement.
docForm in a couchapp is supposed to be an easy way to bind a form to a document in couchdb but it seems it’s not that popular. From what i observed, it’s hard to understand how your form gets...
Serialize form data to json with jquery
I worked on new plevsy all day, and at some point i got in need of some function to serialize forms into deep json objects so that i can push whatever the form has, directly to couchdb. None of solutions i got from some googling satisfied me so i took a shot at my first jquery plugin. Well, it wasn’t a jquery plugin at the first place, but i thought it would be much re-usable.
(function( $...
