sleepcoding

GPUs, Hadoop and Testing Scalability

Tue, 14 Feb 2012 16:28:03 +0200

As i told numerous times before, i am currently trying to get some GPU powered image processing application to run on Hadoop. In development phase we were using a cluster of 12 machines with one Nvidia GTX 480s each, but since we are launching in a few months, we had to do some tests on our production cluster of 25 machines with two Nvidia Tesla M2050s each. In this post, i’ll try to sum up the process of testing, technical details will come later.

First some reminders about our architecture. Image processing application (IPA) receives an array of images and returns an array of results of doubles. A reduceless MapReduce application divides the images in HBase into chunks, and passes those chunks to IPA. Simply put, while it’s improbable for a single IPA to process thousands of images at once, whole system is able to process millions of images in parallel.

What matters (on our end) is the number of images IPA received and how much time did it take to return a resultset. Using those, we calculate a basic metric: speed in number of images processed per second (ipps). We also calculate the same speed for whole cluster, to see if we can reach a speed like nx ipps when our IPA runs at x ipps and cluster runs n IPAs in parallel (spoilers … we can!).

To show this in numbers, we measured base IPA speed on GTX 480. While the CPU on the system also effects it a bit, its runs at 19.46 ipps on average. On the other hand, our cluster with 12 GTX 480s runs at a total speed of 231 ipps which is extremely close to 12 x 19.46 = 233.52 ipps! Looking at this numbers we assumed our system scales linearly so when we increase the number of GPUs to, say 24 we’ll have 231 x 2 = 462 ipps.

With this assumption in mind, we measured base IPA on Tesla M2050, which is 14.80 ipps (yes, Tesla M2050 is about 24% slower than GTX 480) and expected to have a speed like 14.80 x 50 = 740 ipps on our production cluster with 50 Teslas. Our first results with 518 ipps was nowhere near that. We started investigating…

After some lousy ideas putting the blame on IPA folks and node configurations, we took a step back and started questioning our ways of testing. We knew there were IO and Hadoop task management overheads but they were omissible … for jobs containing large amounts of images. We missed that the definition of large would differ amongst clusters such as one with 12 GPUs and another with 50 (!). We were testing both using 100.000 images and it could’ve been a small number for the latter. We slowly increased the number of images to one million and…

we got close enough to expected speed of 740 ipps with 709 ipps. MapReduce jobs in our system will process millions of images in production which means the cluster will be fully utilized. If there were only a hundred thousand images a large portion of the investment would have been wasted.

Lesson learned in scalability: You have to cut your coat according to your cloth. Or you shouldn’t buy more cloth than what would be necessary to cut your coat. Or … Whatevs, you got the point.

Lesson learned in testing: Always test your systems, then test the hell out of them and when those don’t satisfy you change your tests and test again. It might cost some time but it will save money.

Java web services without (explicit) code generation - with exception handling

Mon, 05 Dec 2011 19:38:00 +0200

Finally… it’s been some busy weeks which i constantly sat in front of the computer in first one and constantly moved around them in the others. Finally i found some space to finish what i started. But space is not too much so i’ll keep this short.

Previously i talked about some funny web services stuff, and finished with a problem concerning exception handling. In short form: SOAP cannot transparently handle Java exceptions, so you cannot throw something in the server and expect the client to catch the same. You need some transformation.

In a longer form: SOAP has a thing called soapfault which is the closest thing you have to a java exception, but in order to use it you have to accept some rules. First of all your exception should be a checked one. Second, soapfault is basically XML so it can only transform things that can be parsed/rendered into XML. Which means you have to wrap your exception information in a JavaBean enabling it to be easily transformed into XML and back. Looks good with one little problem: what if you can’t get your exception information into a bean. Maybe you have errors in an enum or even worse, you are using an interface to describe your error codes. Well, JAX-WS has nothing to offer.

Another thing JAX-WS does not offer is a decent developer documentation. You have to waste some hours debugging to see which method gives you what. Except for typesafe exception handling, … because here goes:

MyException is the custom checked exception. For simplicity, it has two String parameters but it is possible for you to create your interface implementing custom object or enum or whatever you need. Just change the object instantiation in line 78 to suit you. Usage of this snippet will be simply like

MyService port = getPort(new QName(MY_NS, "MYServicePort"), MyService.class);
port = JaxWsExceptionCatcher.catchOn(port);

And you can catch and process your exceptions just as the service is in your classpath, still without code generation.

Java web services without (explicit) code generation

Thu, 10 Nov 2011 12:23:00 +0200

I don’t know you but i hate code generation. Bytecode generation may sometimes be useful, but kills debugging capabilities so should be avoided most of the time. Source-code generation on the other hand, i simply fail to understand the necessity. If some 3rd party library will write the codes i will run, why can’t i simply let the library do whatever it needs over some sort of an API?

Anyways, we all know the story. If you are making use of an external SOAP web service, you are kinda forced to generate (source) code. But most of us expand this approach and generate code for SOAP web services between modules of the same project. Which is extremely unnecessary, after JAX-WS 2.0 (i guess, not sure about the version). Instead, we can give plain-old-java-interface of our service and WSDL url to JAX-WS and make it work for us.

class MyService extends Service {
  public MyService() throws Exception {
    super(new URL("http://path/to/service?wsdl"),
        new QName("http://service.my.org/", "MyService"));
  }
  
  public My getMyPort() {
    return getPort(new QName("http://service.my.org/",
        "MyPort"), My.class);
  }
}

Above code shows the whats necessary on client side. Service we extend from is a class in JAX-WS framework. My is the interface of the service we are trying to use. This is the simplest example which you will find when you google JAX-WS without code generation. But as always noone’s trying to make a life with hello-world applications.

Every module uses custom beans (complex-types) in communication so a single interface will not be enough to work (It will be if there are no complex-types). JAX-WS will auto-generate transport classes but will not touch business specific beans. So what i come up with is to make the service providing module to publish a jar with necesseary beans and web service interface. Service consuming module defines a dependency to that artifact and goes along with its life. The jar actually contains the half of the stuff what JAX-WS would generate but now, its not ugly as in generated by some magic library, its ugly as some module developer wrote it ugly so you can push him/her around. Another upside is now that you have written the instantiator code (above) you can write it anyway you like and say dependency inject using guice.

Story does not end here though. Now that you have (almost) isolated yourself from SOAP-mechanics (using guice and all) you may want your service provider’s exceptions untouched. Hold tight for second post.

Using ivy and maven together

Wed, 02 Nov 2011 15:34:40 +0200

It’s not logical, highly unnecessary and probably expensive. But anyhow we found ourselves in that environment no matter what. Problem stemmed from the fact that eclipse/RCP dependency system being incompatible with virtually everything out there. We were using ant/ivy and pretty happy with it but our UI side found no easy way of headless-building their application using it. Eclipse is trying to make use of maven 3 with a thing called tycho, but that’s another story. Point is, they were practically forced to maven, and so was i (us).

The problem is, the eclipse project(M1), which is built using maven, depends on a project(I1) which is built using ivy. Since these projects are constantly evolving, dependency is for SNAPSHOT version. Add another oversight of choosing nexus as artifact repository manager, we ended up being unable to publish SNAPSHOTs with ivy and depending on them with maven.

We set M1’s updatePolicy value to always and expected it to re-download the snapshot artifact of I1 on every change but there is more than one way to do this. Ivy relies on the timestamps of those while maven uses external metadata to identify if a SNAPSHOT artifact has been changed or not. But, ivy has no idea about an external metadata file during publish/deploy so nothing to use for maven. (1)

Nexus can actually repair missing metadata files but i think (not sure) it requires the artifacts to be deployed with uniqueVersions (that funny timestamp-like things replacing SNAPSHOT). Of course, ivy has no idea about those neither. (2)

OK, we can disable uniqueVersions and get “SNAPSHOT” without funny timestamps. But no, because maven 3 got rid of the functionality and uniqueVersion is always on. (3)

Adding (1), (2) and (3); we had a huge incompatibility problem on our hands. Some researching came back negative (maven blaming nexus, nexus blaming ivy, ivy asking questions why maven?…) we fell back to disabling ivy-publish‘es and using mvn deploy:deploy-file’s instead. Reconfigured our jenkins accordingly and finally evaded problems.

Bottom line: don’t use ivy and maven together; it’s not logical, highly unnecessary and probably expensive.

Apache ODE and CLOB issue

Wed, 05 Oct 2011 20:51:00 +0300

I took-over some responsibilities from a recently departed collegue, and with it, i was kinda forced to turn back to JEE world. Not exactly the same technologies and frameworks i am used to, but once you hate some part of something it is likely that you won’t enjoy the other parts.

Anyways, first assignment was to move some WS-BPEL processes from glassfish to Apache ODE. It sounds like it should be easy since WS-BPEL is a standardized and well-acknowledged specification but only an inexperienced and/or naive developer believes that. Standards are never that standard. Only the simplest hello-world can be deployed to more than one (two at most) container without a problem, your JPA application will never port from hibernate to toplink and your standards-compliant webpage will never look like the same in IE. Without some unknown hours/days of hard work, that is.

But for that instance, i got lucky. The hard part was already done and documented (1|2|3) by Hilal Tarakci (still not twitting!), whom i’ve been working closer now. The last problem was the easiest one but helped me steal all the credits. ODE, by default, works using a derby database which doesn’t like CLOBs larger than some size and barfs like this when it encounters one:

java.sql.SQLException: An unexpected exception was thrown
	...
Caused by: java.sql.SQLException: An unexpected exception was thrown
	...
Caused by: java.sql.SQLException: Java exception: 'A truncation error 
  was encountered trying to shrink CLOB '' to length 1048576.:
  org.apache.derby.iapi.services.io.DerbyIOException'.
	...
Caused by: org.apache.derby.iapi.services.io.DerbyIOException:
  A truncation error was encountered trying to shrink CLOB ''
  to length 1048576.

I guess this was somewhat expected because there is a small tutorial in the installation docs of ode, showing how to configure it work on a mysql db. Distribution package also contains DDLs for Oracle, but if you’re already running a postgresql server and don’t want another link in the chain, you’re (not) alone. Without further ado, here are the things you should do.

Create the database you wish to use on the server (you wish to use).
Get this SQL piece and execute on it.
Take this context snippet and place it into your $TOMCAT_HOME/conf/server.xml in <Host> part after modifying as necessary.
Get a jdbc jar from postgre and place it into $TOMCAT_HOME/lib
Get this properties file and place it into $TOMCAT_HOME/webapps/ode/WEB-INF/conf
Start tomcat.

And yes, i mostly got this from the original tutorial. Only thing i did was to edit SQL into the form that postgre would understand. For those of you running something bigger than tomcat, it should be easier to define a JDBC connection on JNDI.

Hadoop MapReduce job statistics (a fraction of them)

Fri, 26 Aug 2011 09:25:39 +0300

Well, this has been on my backlog for a while. The problem is extremely simple actually: when did a MapReduce job started processing? I need this info to report to my clients using my API, meaning redirecting them to the JobTracker’s web interface is not an option.

Everyone using hadoop for some time knows 0.20 is the version to use, and everyone developed something other than a WordCount knows it’s a PITA. API is hard to use at best, misleading and incomplete most of the time. You might wonder how hard can it get to extract a basic (and easily accessible over the web interface) piece of information such as a start time of a job, all i can say is very.

Without further ado, while i expect something like Job.instance("JOB_ID").getStartTime() here is the piece of crappy code i found to be working:

long startTime(String jobID) {
  Configuration conf = new Configuration();
  JobClient jobClient = new JobClient(new JobConf(conf)); // deprecation WARN
  JobID jobID = JobID.forName(jobID);                     // deprecation WARN
  RunningJob runningJob = jobClient.getJob(jobID);
  Field field = runningJob.getClass()
      .getDeclaredField("status"); // reflection !!!
  field.setAccessible(true);
  JobStatus jobStatus = (JobStatus) field.get(runningJob);
  return jobStatus.getStartTime(); // finally
}

As noted above, JobConf and JobID are deprecated. But since there is no way of working with anything non-deprecated, we reluctantly accept that. What we may not accept is working with reflection, but well… I couldn’t find any other way (please point me if you know). It is actually funny to have that information in the status field of runningJob but not able to access with because of a lack of getStartTime() method which reads from it. (BTW v0.21 is closer to what i expect but it is largely unusable for various reasons.)

On the other hand, my requirement wasn’t that, exactly. There may be a delay between the time i have submitted a job and it started processing, highly because the cluster was busy. What i needed was when the job started actually processing, meaning the time the first task is fired on a task tracker. Now i expect something like Job.instance("JOB_ID").getTasksOrderedByStartDate().get(0).getStartTime() but i know i won’t get what i expect, instead:

long actualStartTime(String jobID) {
  Configuration conf = new Configuration();
  JobClient jobClient = new JobClient(new JobConf(conf)); // deprecation WARN
  JobID jobID = JobID.forName("job_201107011451_0001");   // deprecation WARN
  RunningJob runningJob = jobClient.getJob(jobID);
  TaskID firstCompletedTaskID =                           // deprecation WARN
      runningJob.getTaskCompletionEvents(0)[1].getTaskAttemptId().getTaskID();
  for (TaskReport tr : jobClient.getMapTaskReports(jobID)) {
    if (tr.getTaskID().equals(firstCompletedTaskID)) {
      return tr.getStartTime(); // search !!!
    }
  }
}

First task completion event belongs to the SETUP task which runs on the time of job submitting no matter what the cluster is busy with. That’s because i’m getting the second one in the array using [1].

One small problem is that i’m using task completion events, not task starting events, so i am assuming the first task to get finished is also the first task to get started. This is usually correct in my case but i know it will not apply to others.

I haven’t been able to find a way to get a job’s finish date yet, i’m using job.end.notification.url for that. Hadoop sends a GET to a servlet on finished jobs so i simply get the time the service was called. It may not be accurate but again works for me.

In the light of these difficulties, i am thinking about a simple application that serves easily parseable job information. It would probably be rendered obsolete when 0.22 is out but it might still be useful to be able to consume such info with other languages than Java.

Scalatra result announcer w/ various datastores

Wed, 10 Aug 2011 11:19:00 +0300

In previous post, i talked about an examination result announcer application in scalatra/scalate/scalaquery. I mentioned i would try the app with different datastores in another day and post the results. That day has arrived at last.

Since i’ve talked about the system before I’ll keep things short this time and jump straight to results.

			RPS	p90
In memory:		591	238
Voldemort (0.90):	552	256
Mongo (1.8.2):		548	255
Redis (2.2.12):		523	265
Cassandra (0.8.2):	504	273
HBase (0.90.3):		471	285
MySQL (5.5.15):		453	346

In memory storage means i used a scala.collection.mutable.Map object in my Controller to collect the results. The result above was measured with scala 2.9 parallel collections. Without them, the numbers were slightly smaller.

All 3rd party storage solutions were working on localhost with their default configurations. All of them have been accessed with preferred or well-known drivers.

Voldemort - Java API
Mongo - Casbah
Redis - scalaredis
Cassandra - Hector
HBase - Java API
MySQL - scalaquery/JDBC)

I did not try to optimize my code as it is another one of my goals to measure how easy it is to get best performance with little effort.

As seen in the chart, voldemort and mongo are virtually the same in terms of these simple performance measurements. I guess this is because both the storage systems work directly off memory. Since i everything is configured to defaults, voldemort was using in-process BDB-Java, and mongo was using memory-mapped files (i guess). On the other hand, while being an in-memory system too, redis missed their performance with a small amount. I feel some sort of configuration requirement there.

Cassandra and HBase with their BigTable like storage mechanics lag behind others with small margins. I don’t know much about Cassandra but running a pseudo-distributed HDFS and an HBase (and on the same computer as the application and jmeter) is highly discouraged. And, i guess i know enough HBase to say that it is not the perfect fit for this example application. Results simply reflected that.

Since my linux has updated itself several times in last two months, i ran mysql tests in order to keep things fair. You might have noticed its performance is also increased, but not to the point where it may compete with others.

All the code is on github. Switch branches for different stores.

NOTE: Those system updates effected my couchdb too, which upgraded it to 1.1.0. Long story short, my application running on it outperformed everything on the list with 649 requests-per-second and 90% of requests were under 201ms.

NOTE: Another thing i discovered was if i were to use rewrites in couchdb, it damages performance to 550 RPS and 230ms p90. Interesting…

Announcing results with scalatra

Thu, 30 Jun 2011 10:53:00 +0300

As other posts in the series mentioned, i am trying out some web frameworks and data stores with a small web application which would be used to announce the results of a hypothetical exam. You can find the details in the first post. Today, the example app will be of scalatra.

Scalatra is a sinatra-like lightweight web framework written in scala. For developers living under a rock for a few years, sinatra-like means just enough tools to map a URL to a method. No ORM, no templating, no authentication over LDAP. Just URLs and methods. Scala is a programming language which tries to leverage both object-oriented and functional concepts. Google is full of sites telling reasons why scala is a great language.

It is clear that a programming language and a simple web framework is far from being enough to develop a web application nowadays. In order to query results, they have to be stored in some database first. For this specific example, i’ll use mysql. A simple web-application like this one would not actually require another layer on top of the database but since i am evaluating the ecosystem rather than developing a real-life application, i’ll add scala-query into the mix. I hope it will ease the pain JDBC will induce. On UI side, mixing HTML with application logic is pretty much accepted as a bad practice, so i’ll use scalate as the templating engine. As all of the tools are leaning towards sbt, it will be the bowl holding the soup together.

Since scalatra is the core piece in the environment, it seemed logical to start with it and add other ingredients as i go. There are two ways to create a scalatra project skeleton. First way is simply cloning this repo. Other one requires installation of giter8 and sbt seperately. Both will come scalate included. Difference is former uses sbt 0.7 series while latter uses sbt 0.10. If you’re like me and have a tendency to walk on the edge, you’ll need to “g8 scalatra/scalatra-sbt” after installing these shiny tools. It will ask some questions about the project and create it.

$ g8 scalatra/scalatra-sbt
organization [com.example]:
name [scalatra-sbt-prototype]: resultannounce
servlet_name [MyScalatraFilter]: ResultAnnouncer
scala_version [2.9.0-1]: 2.9.0
version [1.0]:

After that is finished cd into your project and run “sbt”. It will download some files and give you the sbt shell. Run an “update” to get your dependencies followed by a “jetty-run” to see some Hello application running on localhost:8080… Now get coding!

First thing i did was changing the default Hello screen to the login page, written in template main using jade.

  get("/") {
    templateEngine.layout(root+"main.jade")
  }

After setting the form to simply POST to a url like /r/idnumber, it was easy to handle values in scalatra side.

  post("/r/:idn") {
    val result = Result of (params("idn"),params("pass"))
    result match {
      case Some(_) => templateEngine.layout(
        root+"result.jade", Map("result"->result))
      case _ => templateEngine.layout(
        root+"main.jade", Map("formErr"->"Wrong Details"))
    }
  }

The Result here is a DAO i have thrown with my limited scala and virtually non-existent scala-query knowledge.

object Result {
  val db = Database.forURL("jdbc:mysql:///ss?user=root", 
    driver="com.mysql.jdbc.Driver")
  def of(id:String, passwd:String) = {
    db withSession {
      val q = for (e <- Results if e.id === id) yield e
      q.first
    }
  }
}
object Results extends
    Table[(String, String, String, String)]("results") {
  def id = column[String]("id", O PrimaryKey)
  def passwd = column[String]("passwd", O NotNull)
  def name = column[String]("name", O NotNull)
  def result = column[String]("result", O NotNull)
  def * = id ~ passwd ~ name ~ result
}

There is a password-hash control step too but i omitted it for this post. The Results object defines the database table and enables us to query without writing any SQL.

After some manual testing to check everything is in working order, i loaded 3M results in that table and started hammering the application with jmeter. Simple stress tests on login screen which does not touch database or anything yielded 658 requests-per-second for 100 concurrent users and 90% of the requests were served within 180ms. Considering the test for couchdb on the same machine gave 500 requests-per-second with 290ms 90% line, scalatra and scalate seems like an improvement. But as i have mentioned before this is hardly a real-life scenario: Users generally won’t bounce off login screen, they will login using their details and try to see their exam results. Using a jmeter workbench for this scenario, requests-per-second dropped to 396 and 90% line increased to 413ms. Now that seems a like a little stepback from what i was able to achieve with couchdb. I’ll try to identify the pieces (scalate, scala-query, mysql) causing the slowdown but that’s for another day.

Scalability side is the same as the couch: application is totally stateless so running mirrors behind a load-balancer should be enough to increase that RPS to the point required. Since there are no writes there shouldn’t be any problem for scaling them.

The whole application took my whole day including the time needed to get to the default Hello screen. Besides mysql, all the tools here were new to me, yet i was able to make something that solves a problem. Although i am a newbie, scala is fun to work with and considering what i was able accomplish in a day, it is nowhere near complex. But couchdb still seems a better fit for this kind of problem. Results are documents and a document store with HTML rendering capabilities is all one can ask for.

UPDATE: Code is now on github

Resource synchronization on Hadoop clusters with ZooKeeper - Part II

Wed, 01 Jun 2011 19:55:00 +0300

Straight from where i left. GPUs are massively paralel in contrast to CPUs, hence for some parallel processes, they are damn fast. The benchmarks you see around showing performance increases over 100x are theoretically true. By theoretical i mean pure CPU vs GPU computing power. In other words, for an infinitely running computation, it is possible to get 100x more results with a GPU than you would with a CPU core given a constant amount of time. But as experienced GPGPU developers would undoubtedly know, in practice, things rarely happen that way.

First of all, only a small part of commercially meaningful computations are running infinitely. The first infinitely running computation coming to my mind is calculating the digits of pi. That surely is to make some money if you are into cryptoghrapy or something but i guess it is safe for me to say that is both a niche and a dominated market. Another computation may be fractal generation and i have yet to meet anyone making money out of generating colorful images. Businesses sell results and to get results, processes must end in some way.

Two of the well known facts of finite processes are that they need some data beforehand, and they output some data afterwards. That means the duration of any process will roughly be of IO time and computation time. A GPU may decrease computation time but since you cannot change IO time, it will eat into your 100x expectations. Bare computing time may decrease but when IO time stays the same, (actually it increases in GPGPU processes but that’s another story) depending on the type of the problem, you may settle for 3x performance or less.

So, as shown in Part I, if you configure your systems to have your resources (GPUs) occupied by only one process at any given time, you do not use them optimally. Meaning, if you configure your MapReduce TaskTrackers’ maximum simultaneous map tasks count to the number of your resources on the system, JobTracker will wait for each task to finish before starting another one, and your resources will sit idle in IO part of these tasks.

One solution is using more than one process. It is possible to start two processes for each resource and let one use the resource while the other one does its IO operations. After first one is done with the resource, it can signal the other one to start operation. So resources will never have to wait for IO to be done beforehand.

This process signaling mechanism fits perfectly with zookeeper’s watches. You can set a watch on a znode and zookeeper will notify you when there is a change on it. In this particular problem, second process may set a watch on a common znode. When the first process is done with the resource, all it has to do is modify the znode to let the second process know it has finished. This is the exact explanation of what ResourceSynchronizer does. When you call .request() it will return the next free resource or if there aren’t any free resources, it will wait for another process to call .release() to return anything. So the process will be blocked before using the resource.

You can set the same pool/resource list and spawn some processes to see the effect. Say you set your resources as [“res0”, “res1”], and your resource intensive procedure takes 10 seconds. If you run 3 processes in 10 seconds, first 2 will get “res0” and “res1” respectively for their rs.request() calls, while the last process will wait till any of the first two processes call rs.release().

ResourceSynchronizer rs = new ResourceSynchronizer(
    new ZooKeeper("zkensemble", 20000, null), 
    "/pool", new String[]{"res0", "res1"});
log.warn("Requesting resource...");
log.warn("Got resource : " + rs.request() + ". Working...");
Thread.sleep(10000);
log.warn("Process Done! Releasing Resource");
rs.close();

Output of process #1

19:01:16,799 WARN  Run - Requesting resource...
19:01:16,824 INFO  ResourceSynchronizer - ZK ensemble connected
19:01:16,846 INFO  ResourceSynchronizer - Resource '/pool/res0' allocated
19:01:16,846 WARN  Run - Got resource : res0. Working... 
19:01:26,846 WARN  Run - Process Done! Releasing Resource
19:01:26,853 INFO  ResourceSynchronizer - Resource released
19:01:26,861 INFO  ResourceSynchronizer - ZK connection closed

Output of process #2

19:01:22,504 WARN  Run - Requesting resource...
19:01:22,518 INFO  ResourceSynchronizer - ZK ensemble connected
19:01:22,541 INFO  ResourceSynchronizer - Resource '/pool/res1' allocated
19:01:22,541 WARN  Run - Got resource : res1. Working... 
19:01:32,541 WARN  Run - Process Done! Releasing Resource
19:01:32,556 INFO  ResourceSynchronizer - Resource released
19:01:32,564 INFO  ResourceSynchronizer - ZK connection closed

Output of process #3

19:01:23,967 WARN  Run - Requesting resource...
19:01:23,992 INFO  ResourceSynchronizer - ZK ensemble connected
19:01:24,003 INFO  ResourceSynchronizer - No available resource, waiting... 
19:01:26,854 INFO  ResourceSynchronizer - Retrying to get another resource
19:01:26,871 INFO  ResourceSynchronizer - Resource '/pool/res0' allocated
19:01:26,871 WARN  Run - Got resource : res0. Working... 
19:01:36,871 WARN  Run - Process Done! Releasing Resource
19:01:36,886 INFO  ResourceSynchronizer - Resource released
19:01:36,894 INFO  ResourceSynchronizer - ZK connection closed

Output of process #4

19:01:26,802 WARN  tool.Run - Requesting resource...
19:01:26,823 INFO  ResourceSynchronizer - ZK ensemble connected
19:01:26,835 INFO  ResourceSynchronizer - No available resource, waiting... 
19:01:26,854 INFO  ResourceSynchronizer - Retrying to get another resource
19:01:26,879 INFO  ResourceSynchronizer - No available resource, waiting... 
19:01:32,556 INFO  ResourceSynchronizer - Retrying to get another resource
19:01:32,573 INFO  ResourceSynchronizer - Resource '/pool/res1' allocated
19:01:32,573 WARN  Run - Got resource : res1. Working... 
19:01:42,573 WARN  Run - Process Done! Releasing Resource
19:01:42,581 INFO  ResourceSynchronizer - Resource released
19:01:42,590 INFO  ResourceSynchronizer - ZK connection closed

These 4 outputs belong to the same code piece running 4 times with a few seconds between them. If you check the timing on the logs 1st and 2nd processes got res0 and res1, just after they requested them. 3rd process took res0 just after 1st one released it. 4th process also tried for res0 after 1st one released it but couldn’t make it so waited for 2nd one to release. Timestamps show that the resources were left idle for only a few milliseconds.

I have also added a REUSE configurable in the code to let resources to be reused given amount of times. For the previous example if REUSE is set to 2, first 4 processes will get “res0”, “res1”, “res0”, “res1”. Fifth one will wait for a resource to be freed up. With small modifications i am sure it can be a solution to IO increasing properties of GPGPU processes too, but it will probably require you to change the way you access your application so i am not there yet.

UPDATE: A colleague advised me to use “A semaphore implementation using ZooKeeper” as the title, which would be appropriate but not entirely correct. As a careful reader might notice, the mechanism is not binary nor does it use a counter. Instead, it holds the names of the resources it is supposed to allocate.

Resource synchronization on Hadoop clusters with ZooKeeper - Part I

Wed, 18 May 2011 14:15:00 +0300

“We need zookeeper to run HBase”. Until past week that was basically my view of zookeeper. It is a distributed configuration and coordination service, HBase requires it so we have to put it in cluster. For a size of our cluster 1 instance seems to be OK but we are running 3 instances. These four sentences pretty much summed up what i knew about it. Fortunately i had looked to its main page previously and remember its somewhat abstract purpose : “Distributed coordination”.

First, some clarification: This post is generally about coordinating processes that requires access to some limited system resource. In order to keep things simple, i used an example resource throughout the post, which is GPU. Another example may be distributed CD publishing using Hadoop on machines with a number of CD writers. Or, controlling an array of arduino devices to simulate LHC. In short, post has no direct relationship with GPGPU programming nor GPU kernel thread synchronization. If you have arrived here googling that, i’m sorry.

So, the problem: is if you put a number of CPU cores in a single computer and start running processes, operating-system will place them to the cores accordingly. Say you have a 4-core machine and try to multiply 4 matrices simultenously, each multiplication will be done on another core. Well, i don’t know if there are any developments about it but that’s not the case for multi GPU systems. Say you have 4 GPUs on a machine and you spawned 4 processes wishing to multiply your matrices in each one of them, you need to explicitly tell those processes not to overlap with one another. If you leave it to OS, one or more of your GPUs may sit idle while others starve for resources. I heard Mac OS can manage this but they are not suitable to our environment.

In theory: there is no way of letting anything other than yourself decide which process (read: map task in a MapReduce job) should occupy which resource. Simplest solution is to supply the process the resource identifier it should operate on. Process may be executed with appropriate parameters. But this would mean manual control of all the processes which is not possible with MapReduce. Another solution is having processes ask to some daemon process, which resource to allocate. Daemon process may hold which process uses which resource, so answer other client processes accordingly. This is actually how we were working for a while now.

In practice: this daemon process would bring some maintenance issues as any other software components. It is just another service one needs to deploy to machines in the cluster and ensure it works properly. Because of this, we were hesitant to go production with this setup, looking for another solution. I am not sure how it happened but zookeeper seemed like it can do such a thing. Let me rephrase that, we thought we can do GPGPU process synchronization with zookeeper, without actually knowing what zookeeper does. It is a distributed coordination service right, how hard could it be?

After reading the “Getting Started” part of zookeeper documentation, i saw that my hand was blackjack. I can create some data points (called znodes), load some small data in them, and access those from any other process. An altogether solution to what keeps us scratching our heads. All i needed to do is, modify my mappers a little to talk with zookeeper before and after the process. Since GPUs should be coordinated per computer basis, mappers should know which computer they run on, and the GPUs on that node. I applied occam’s razor and got this.

  private String hostname() throws IOException {
    return InetAddress.getLocalHost().getHostName();
  }

  private static String[] discoverGpus() {
    File[] gpus = new File("/dev").listFiles(new FilenameFilter() {
      public boolean accept(File dir, String name) {
        return name.startsWith("nvidia") && ! name.endsWith("ctl");
      }
    });
    String[] ret = new String[gpus.length];
    for (int i = 0; i < gpus.length; i++) {
      ret[i] = gpus[i].getName();
    }
    return ret;
  }

On our nodes, GPUs are added as devices under /dev with sequential names nvidia0, nvidia1, … And there is one another device named nvidiactl which is not a GPU. Other additions to my mappers are those.

  // in setup()
  rs = new ResourceSynchronizer(
      new ZooKeeper(QUORUM_ADDRESS, TIMEOUT, null), 
      "/gpusync/"+hostname(), 
      discoverGpus());

  // in map()
  String gpu = rs.request();
  process(); // whatever
  rs.release();

  // in cleanup()
  rs.close();

Now i cheated a little bit here and didn’t include the core piece that is ResourceSynchronizer. That’s because in addition to holding GPU names and supplying names to process requiring them, it does one additional and somewhat more sophisticated task concerning GPGPU operations. Seasoned GPU developers may guess what it is but i left it for Part II of this posting.

LZO vs Snappy vs LZF vs ZLIB, A comparison of compression algorithms for fat cells in HBase

Thu, 14 Apr 2011 15:25:00 +0300

Now and then, i talk about our usage of HBase and MapReduce. Although i am not able to discuss details further than what writes on my linkedin profile, i try to talk about general findings which may help others trying to achive similar goals. This post is about a recent research which tries to increase IO performance for our MapReduce jobs.

Why Compression?

HBase documentation and several posts in hbase-user mailing list tell that using some form of compression for storing data may lead to an increase in IO performance. Considering hadoop clusters almost always work on commodity machines, the reason for that is simple to explain: disks are slow. Hadoop workloads i know about are generally data-intensive, thus making the data reads a bottleneck in overall application performance. By using some sort of compression we reduce the size of our data achieving faster reads. On the other hand we now need to uncompress that data so we use some CPU cycles. It is simply trading IO load for CPU load.

If the infrastructure starves on disk capacity but has no performance problems it may be logical to use an algorithm that give huge compression ratios, losing some time on CPU but that’s usually not the case. Large capacity disks are far cheaper than fast storage solutions (think SSDs) so it is better for a compression algorithm being faster than being able to give higher compression ratios. Because of that hadoop applications prefer LZO, a real-time fast compression library, to ZLIB variants. Of course these are general talks and to see real performance changes and compression ratios, one have to try those algorithms with his/her own data.

Which algorithm?

Our data is like 700kB per row and for testing purposes we have 100k rows. Each cell contains an image, more specifically a subset of an image so it is binary and supposedly not as compressable as some log file. Using no compression, our test data of 1000 items takes up 670MB and our MapReduce tasks are able to read a cell in 8.41ms.

First algorithm we tried was ZLIB, or java.util.Deflater/Inflater following this post by @jdcryans. It simply involves using Deflater just before “Put”ting data into HBase, and using Inflater just after reading data from “Result”s. The total size of our 1000 items decreased to 346MB meaning a compression ratio of 48%. But our reading performance suffered 16%, increasing the time per row to 9.73ms.

Second one was the famous LZO. Although we are unable to re-distribute it because of licensing issues, we still felt the urge test and see what we are missing. It is somewhat harder to use in hadoop (at least the recommended way), but i’ve managed. You can check here and here for instructions on how to set it up. On the other hand this complexity is sure to have a benefit. All other methods i talk about here compress data per item basis. LZO on the other hand will compress the whole file in HDFS, so in a regular setup it is expected to have better compression ratios since there will be similarities among the rows and it will exploit those. Anyways, our 1000 item set resulted in 398MB meaning a 41% compression ratio and we’ve seen a 5% increase in reading performance too: we read one item in 8.1ms compared to 8.41ms uncompressed. So it is starting to become a win-win.

Third test was a LZF implementation, ning-compress following Ferdy Galema’s response to previous Deflater tip. It works the same way as it does too, like using LZFEncoder.encode just before writing to HBase and using LZFDecoder.decode just after reading. At this test our data size was 400MB meaning a compression ratio of 40%. Reading performance increased 21% with 6.63ms passed for one item.

Last one was recently announced snappy of Google. The same compress-each-item-seperately mechanism applies here with Snappy.compress and Snappy.uncompress. Data size was 403MB which mean around 40% compression ratio and we read our data at 6.37ms per item which indicate 25% increase in IO performance.

Conclusion

Algorithm	Compression Ratio	IO performance increase
Snappy		40%			25%
LZF		40%			21%
LZO		41%			5%
ZLIB		48%			-16%

I am suspicious about something in LZO scores since I was expecting much better performance. But it doesn’t matter because of our inability to redistribute it. Snappy-java with its Apache license is a clear winner. It is way easy to use too.

I have to remind again YMMV. These are the scores for a data which consist of 700kB rows, each containing a binary image data. They probably won’t apply to things like numeric or text data.

And the winner is ...

Thu, 24 Mar 2011 15:07:35 +0200

It was harder than i thought to convince my sysadmin to do anything or to let me do anything, so i continued with my own machine, with some simpler tests. Since i am thinking about using a number of languages/frameworks, i tried to write the easiest possible example for each of them and measure performance.

The easiest example was a web page writing 3 bytes, “123”, as the output. One performance measure is number of requests processed per second. The other one is response time for 90% of the requests for 100 concurrency. I started with a little confusion, not knowing which ones i should test for, but as the number of frameworks i tested grew, i managed to classify them into two categories.

Networking Frameworks: These are the pieces that handle user requests first and passes it to our code for further processing. In my setup i simply returned that 3 bytes along with some HTTP headers.

Web Frameworks: These are the things that make our lives easier by handling all generic stuff and letting us focus on our logic. They also tend to have simpler programming models.

And there are some other tools which i couldn’t put into these categories but still had to be tested for this post to become more interesting. Actually, i will start with these. But before all, i should say that none of the tools had more than 15 minutes of my time [with the exception of Ruby tools]. Almost everything i did was a straight steal from their respective docs running in default configurations. That means two things: your mileage may and will vary, numbers here may indicate how much that tool is performance oriented.

First uncategorized tool is couchdb. It is not actually a web framework, definitely not a networking framework. But it acts as a web server and is able to process some sort of user request so it might fit the purpose of these tests. Besides i love it and don’t want to do anything that doesn’t involve a piece.
RPS: 1150, 90% RT: 120
I cannot compare the numbers to any other tool out there because there is no other one like it. It still is a winner.

Next uncategorized tool is php. Considering the popularity, i cannot leave it behind. Considering the options, it generates another category in itself. I am sure google knows about many blogs whose sole purpose is to compare php frameworks, which i left out. What i did was to get some measurements for a number different environments:

					RPS	90%
Apache prefork + mod_php 		7410	15
Apache prefork + mod_fcgi + php-cgi	5770	18
Apache prefork + php-fpm		5170	24
Apache worker + mod_fcgi + php-cgi	5440	22
Apache worker + php-fpm			5740	19
lighttpd + php-fpm			5000	24
nginx + php-fpm				7430	18

Seems like the old way with mod_php and prefork is pretty fast but make no mistake. I did not measure the memory performance of these environments, which i read in various places, prefork sucks. All other configurations with worker seem close to each other which again memory usage may make the difference. I expected lighttpd and nginx to generate similar results but that wasn’t the case.
My winner is : nginx + php-fpm

Networking Frameworks

					RPS	90%
twisted					2750	31
node.js					8150	18
gunicorn				11000	12
jboss netty				13000	6

I would love to have unicorn added here too but i wasn’t able to find a way to run it without sinatra. At least not in 15 minutes. I am specifically impressed by gunicorn performance. It was the fastest to run too. Their website just gave what i want straight from the landing site and it didn’t take me 2 minutes to hit ab. node.js was fast too but i had to compile it first so i finished in like 5 mins. Twisted was a little … well twisted. Programming model is somewhat different from others. So it took a little more while. No need to comment on netty. Java development cannot be faster than anything dynamic.
My winner is : gunicorn

Web frameworks

					RPS	90%
sinatra(unicorn)			734	140
express(node.js)			5660	30
django(gunicorn)			5270	20
play(netty)				7000	20
??					14220	6

I do not want to say anything on sinatra and unicorn here because i am almost sure that i did something wrong. I could not investigate further because of my 15 min rule. I believe all unicorn lovers have an explanation for this. I just don’t. On the other hand this comparison is somewhat unfair. play and django are full web frameworks with ORMs and caching and all. sinatra and express are lightweight and they aim performance. Oh, yet the numbers indicate otherwise.
My winner is : play

There actually is another web framework that totally blew my mind when i saw the output of the benchmark. But because of the unfairness i mentioned above, i did not want to include it in the list [but then i did]. It actually beat even netty and replied 14220 requests per second. 90% of the requests were replied in under 6 miliseconds. And ironically it was inspired by the worst performing tool of all:
My grand prize winner is scalatra. I will definitely use more of it in the future.

Announcing results with/of couchdb

Wed, 16 Mar 2011 12:04:00 +0200

I have already mentioned about the scenario in the previous post. Now its time for some business, with couchdb as it is my favorite nowadays. The application is actually dead-simple. One screen asks you your ID and password, next one lists the results you had. In order to look like some real-life counterpart (not that one exists), i added some styling to the page, but still kept the size minimum. You can see the screenshots in story-telling post.

These all are composed of a single show-function in order to keep the design-doc minimal. However in order to make it like a real-life application, with all maintainability issues and such, i used a mustache template to render that screen, and that means i used mustache.js too. User passwords should actually be on some other authentication system, but since we aim for high-performance, we cannot rely on some alien auth system. So every user/result doc also contains a md5’ed password. And that means we need to be able to generate md5 hashes out of user inputs so i used md5.js too. Whole design-doc (a special doc which you define your application) is like that:

{
   "_id": "_design/app",
   "_rev": "6-c80ec7e20e6ded17bf0e048fff596665",
   "templates": {
       "result": "[see below]"
   },
   "lib": {
       "mustache": "[mustache.js]",
       "md5": "[md5.js]"
   },
   "language": "javascript",
   "shows": {
       "result": "[see result.js below]"
   }
}

Show function result.js is :

function(doc, req) {
  var Mustache = require("lib/mustache");
  var md5 = require("lib/md5").hex;
  var ctx = {
    form: true
  };
  if (req.query.password) {
    if (!doc || doc.passwd != md5(req.query.password)) {
      ctx.formErr = "Wrong ID number and/or password";
    } else {
      ctx.form = false;
      ctx.doc = doc;
    }
  }
  return Mustache.to_html(this.templates.result, ctx);
}

For couchdb outsiders, this is a show-function which runs on couchdb when a request is received on some URL. If you add a doc-id to that URL, we receive the corresponding doc (a json object) in function’s doc parameter. req parameter contains details about http request. The result of the function is sent back to the user. Simple.

As you can see, both the login and the result screen spawns from the same function. It is simply a matter of query string of the request. More specifically, if user tries to log-in we check for auth and provide the doc to the template, if not we pass form:true to template so the login form gets rendered. The relevant parts of the template goes like that:

{{#form}}
<p>Please enter your identification number and password</p>
<form onsubmit="sb()">
  <table border="0" cellpadding="3" cellspacing="0">
    <tr><th><label for="idn">ID Number</label></th>
      <td><input type="text" id="idn"/></tr>
    <tr><th><label for="pass">Password</label></th>
      <td><input type="password" name="password" id="pass"/></tr>
  </table>
  <p><input type="submit"/></p>
</form>
{{/form}}

<p class="err">{{formErr}}</p>

{{#doc}}
...Result screen...
{{/doc}}

<script type="text/javascript">
  function sb() {
    document.forms[0].setAttribute(
      "action", "/"+document.getElementById("idn").value);
  }
</script>

That last script changes the form action to include the user’s id number in request path which means the show function will be run on the doc with that id. Short version: Document ids are user ids, and in order to ‘show’ that document we request the doc with the user’s id. An example user/result doc is like that:

{
   "_id": "1000002",
   "_rev": "1-72b2b17d3c46a69464c55c80373abc01",
   "name": "John Knife",
   "passwd": "877466ffd21fe26dd1b3366330b7b560"
   // result data
}

So if we request something like “_show/result/1000002” our result show function will be run with that doc.

I am omitting the result data and template because they may change for every exam. Also it is extremely easy to render json with mustache.

Application is done, but system isn’t. I need 3 million docs in that db, so i wrote a simple script to generate them all. It took me 15 minutes to prepate and around 20 minutes to run. In just over half an hour i had 3 million records ready to be read by the application. Add that duration to the 2.5 hour mostly passed in styling, i had completed single node system in about 3 hours. Isn’t that relaxing!

Speaking of single node deployment (my pc, a quad core machine with 4G ram), here are some numbers. Using ab on login screen: i got around 500 RPS. With 100 concurrent users, 90% of the requests served under 290ms. But that is hardly a simulation for the running system. Normally, a user lands on login screen, then asks for his/her results using credentials. Using jmeter to fit this scenario: i got 390 RPS and 360ms for 90%. In either case, single machine will take hours to finish 3 million results, so we need to scale.

Since the application is totally stateless, I should be able to run a number of couches behind some kind of a load balancer. Just put the data on all of them and assign an usher to show people where to sit. And since there are no writes i shouldn’t even need replication among them. Simple enough. The hard part is to convince some sysadmins to install couchdb on some servers. I need some time. I’ll update here when i’m done.

Examining the examination results - Warm up

Thu, 10 Mar 2011 17:13:02 +0200

Consider this: You took an examination and your result will place you in a university of your choosing, or not. Although the date of the result announcement has already been made public, on the evening of the day before it, reporter on TV tells us: “2021 university acceptance exam results will be announced tomorrow morning at 10am. [Enters a video showing students taking exams and zoom-ined test answer forms] 3 million candidates have to wait just another night to learn if they have made it to the school of their dreams, or those dreams are to be hold onto for another year. Head of the “student picking and planting institute” spoke today to inform that the candidates would be able learn their scores online at http://localhost:5984/ [Enters a monitor showing the app(below)] and official statements will be mailed till late July… [Continues with some interviews on the street]”

You got the picture. Only two strokes out of this picture really concerns me, which are 3 million and 10am. That means that poor server will start to get hammered at around 10am for results. We need a high-performance web-application to process all that. This’ll be my scenario for trying out web frameworks and datastores till i find a better one. Criteria will be requests-per-second on a single machine, on a four-machine cluster with some load-balancing and, how much did it take me to complete the running system, able to serve 3 million results.

Although not-final I have these tools in mind to experiment on

And besides couchdb, i’ll need a datastore for them to work such as

I’ve already finished the app with couchdb (screenshots) but, you know, the post needs to be cooked before serving. Just hold-on a little.

Compiling gcc-4.5 on debian unstable

Tue, 22 Feb 2011 15:56:00 +0200

As of today, i have found no way to install gcc-4.5 on debian unstable without manually compiling it. It seems there is a deb package in ‘experimental’ but it does not install because of a bunch of dependency issues. So i was back in old days where we need to compile several versions of gcc in chain in order to run apache on AIX.

I had forgotten how hard to these things right so i dived straight into ./configure && make cycle without RingTFM. My bad. On the other hand TFM is so Fed up it was not possible for me to R and understand anything. Anyways… this post is just for future reference.

First of all just as anywhere else mentions, you need GMP, MPFR and MPC to compile gcc. But these are not enough, you will need PPL, CLOOG and libelf too. You might try these to get from debian repositories but you may not be able to find them. Even if you do, you may not be able to install because of the same dependency conflicts. I compiled all of them manually and here are the versions used:

gmp-5.0.1
mpc-0.9
mpfr-3.0.0
ppl-0.11.1
cloog-ppl-0.15.10
libelf-0.8.13
and of course gcc-4.5.2

Any of those require any other library than the ones in the list, i installed using apt. After ./configure && make && make installing all dependencies i did those to get a build.

mkdir gcc-build
cd gcc-build
../gcc-4.5.2/./configure --disable-ppl-version-check --enable-languages=c,c++
make -j3
make install

Creating a dir like gcc-build and building there is the preferred way of doing things. I did not come across any, but docs say doing a build in the same directory as sources may yield unexpected results.

Configure script looks to /usr/local for manually installed libraries by default. But if you have changed it for dependencies you should show them with parameters —with-gmp, —with-mpfr, etc. I suggest you leave them default. No one needs any more complexity.

Here is the catch: gcc documentation says that it requires ppl-0.11 but the ./configure script somehow requires 0.10. So, if you provide 0.10 (say with apt-get install libppl0.10-dev) the config.log will not complain but ‘make’ will fail with:

configure: error: cannot compute suffix of object files: cannot compile

Per documentation, this tells you absolutely nothing. And if you give 0.11 (say by compiling manually) config.log complains with the stupidest error possible:

conftest.c:16: error: 'choke' undeclared (first use in this function)

It seems that “choke” is a non-keyword, non-variable statement, something like “foobar” whose sole purpose is to fail the compiler. Another problem which tells you absolutely nothing about the problem and worse, leads you to a completely irrelevant direction. I mean, if you need to fail the compiler can’t you just write something like “_PPL_version_is_not_0.10” so that i get an error like

conftest.c:16: error: '_PPL_version_is_not_0.10' undeclared (first use in this function)

Is it really that hard?! Anyways, i guess everyone but me use —disable-ppl-version-check by default in their build scripts so no one ever mentions it.

Another weird behaviour that cost me about half an hour is this line in config.log

conftest.c:10:28: error: ac_nonexistent.h: No such file or directory

Google this and get nothing again. Because it is nothing. It is a way to test the compiler if it loads nonexistent libraries where it should not. So this error is not an error but a success. But if you just look for error lines in config.log you are as silly as i am. How can you even think this is a configuration error!

I probably compiled dozens of different packages but i never seen successful ./configure output but a failed config.log. If there is a library missing ./configure output always told me whats missing. And if it did not complain about anything i went straight to make. Never needed to check config.log beyond some exceptional cases. This whole behaviour is default in gcc. You always need to check config.log for errors before make.

And remember, errors may not be errors! Even if they are, few words that is written right next to them are not error descriptions. Now i understand why there are so many C-flavored languages.

UPDATE: I received my official gcc-4.5 package with todays morning update. Guess i should’ve waited just a few more days.

Monitoring nVidia GPU metrics with Ganglia

Wed, 02 Feb 2011 23:51:00 +0200

Editor’s Note: As the first time in sleepcoding’s short history, i welcome our sysadmin H. Çağlar Bilir’s post on our recent experiences with graphic cards and ganglia. I mentioned about our new cluster but omitted the detail that the machines also have some GPUs on them. New toys mean new details to look for. So here it goes.

After a brief period of googling, we could not find any out-of-box solution for monitoring nVidia GPU’s with Ganglia, so we created our quick and dirty solution. This is the explanation, code and configuration. First some warm-up:

We have only gtx 480 cards, we have only one gpu per node. We only monitored temperature, fan speed and gpu/memory utilization.
We have linux as underlying OS. We used ganglia 3.1.7 compiled with python support.
nVidia has a command line tool called nvidia-smi (NVIDIA System Management Interface program) and it is in nvidia kernel module packages. (man nvidia-smi)
nvidia-smi tool may not be able to query your gpu cards so you may need to pass —gpu=0 parameter to get some sensible output. Or it may be just us since we don’t run X.

Actually Ganglia has a tool called gmetric, and if you have a command-line utility, you can send its output to Ganglia with the help of cron. To keep our OS installation clean, we did not want to use cron, we picked the harder way: Ganglia has a feature called metric modules which simply enables you to run your C or python code as a third party module (Ganglia README -> Extending Ganglia through metric modules). We used python ‘cos it is fast to develop.

Python pluggable module code (named as gpuwatch.py):

That ’ | tail -23’ is the result of trying to be quick-and-dirty. Standard output of the command has two lines starting with GPU. We needed the one in the last 23 lines.

To link gpuwatch.py with ganglia, a configuration file should be created and placed in appropriate folder. In our case the folder is /etc/ganglia/conf.d and the conf file is modpython.conf. The file is as follows:

For gmond to read the modpython.conf configuration file, there should be an include statement in gmond.conf file. Our gmond.conf is in /etc/ganglia and it contains the following configuration line:

include ('/etc/ganglia/conf.d/*.conf')

Ganglia must be compiled with python support and as a result, there should be modpython.so in the same place with the other metric module so’s.

Ex: In our system, modpython.so and others are in /usr/lib64/ganglia/.

Note: ldconfig must know and add these so’s in its cache.

After all of these, gmond must be restarted.

Here are some notes we took down during the time:

MultiGPU nodes may be processed in code.
Code could be cleaner/wiser.
Other parameters such as ECC errors etc. can be monitored.

Refs:

ganglia-3.1.x/README
http://tech.xlab.si/2008/10/writing-custom-python-metric-modules-for-ganglia/
Inspired by http://sourceforge.net/projects/nvchart/

To wrap things up, we have written that python piece in about half an hour with “pair programming” style, which we do not in our normal workdays, since, you know, he is a sysadmin. It was an enlightening experience to “pair program” with a man who writes shell scripts on-the-fly and avoids programming for a living. Pushed me to my edge of task-focusedness. Thanks man, for that experience and letting me publish this.

UPDATE: After a short while of running this, we ran into issues with nvidia driver itself, causing kernel panics and crashing machines. Although we were unable to pinpoint the problem, it went away after disabling this. In short, use it at your own risk.

Notes from past week

Mon, 31 Jan 2011 20:09:12 +0200

Not focused on anything specific, last week was one of those i had to multitask, so there are more than one subject that deserves a few words. I also am in a state where i both want and don’t want to talk about them, which usually means the post will be more like my weekly reports rather than an article.

Our new shiny cluster of 12 machines arrived at last, just in time to put my hands on HBase 0.90. As usual, I ran in pseudo-distributed mode first, then re-packaged it with our configuration details and sent it over for deploying. Almost nothing has changed in terms of configuration details or API, so everything went as smooth as it could get. Our sysadmin made me laugh when he warned me not to go guns and blazin on machines because they were not placed on rack and i might burn some CPUs. It would be fun though.

I regret to say that we had to go seam for that activiti integrated human-resources application (seam is an application framework that i do not want to link to. I guess that is my way of protesting it. Not that anyone cares). It was for certain even while i was drafting my last post, but i needed to ignore those facts and live in the peaceful world of play! just a little bit more. Now that dream has ended and i went back to JBoss workshop. Although it was a bit painful, i was able to pull of a maven and maven-jetty-plugin trick on seam 2. That saved me from the 40 sec startup of JBoss AS but i think i’m gonna have to live with the lump of unresolved-and-we-don’t-even-care-to-resolve-just-use-sth-else bugs of seam. I am not sure if i should write about the experience. Anyone in need of something like seam/maven/jetty integration may google it to find things that don’t work out of the box, but help to build a working pom.xml. In my opinion, anyone in need of seam should just use something else.

BTW, i know that it’s highly probable its just my allergies on this seam matter. @mozcelebi is using it with whole JBoss stack and is pretty happy with it. Moving on.

I am not an expert but i think we people prefer talking about the things we don’t like over talking about the things we like. I am trying to avoid doing that, so i will write a post about how i did integrate activiti with seam in order to raise my karma over the last paragraph. And this time i would be talking more about activiti, i need good things happening nowadays.

As a dessert, i tried to polish some details with plevsy. I added a few more CRUD screens and tweaked some UI to make use of those new data. There was nothing worthy to note on UI/Evently end past week, but i tried to discover couches on local network by sending JSONP requests all over it using web workers. It was fun. I can’t even begin to tell how delightful it is to be working with couchdb. They couldn’t have found a better motto than “Relax”. Note-to-self: should write about that web worker discovery experiment soon. On the other hand, plevsy is not without some factors that suck this fun and relaxed parts out of it. Note-to-self: should not write anything about these in order to, you know, keep my karma.

Activiti 'Hello world' on play!

Tue, 11 Jan 2011 21:15:10 +0200

Previously, i told about my experiences with play module/plugin development. I tried to point out how it was easy to extend the framework to your needs. A primitive activiti plugin was the product of that effort, and i said there will be a simple application using this. Here goes.

I picked the “Financial Report” example from 10 minute tutorial part of the activiti documentation because it seems the easiest way of demonstrating that it works. The BPMN file -which is used to define the business process- can be obtained from the site.

The use case is straightfoward: we have a company, let’s call it BPMCorp. In BPMCorp, a financial report needs to be written every month for the company shareholders. This is the responsibility of the accountancy department. When the report is finished, one of the members of the upper management needs to approve the document before it is sent to all the shareholders.

Process simply composes of two user actions (emphasized). It also mentions user groups (accountancy department and upper management) but i did not include them in my example in order to keep things simple. However, i used Secure module for user identification. Interface will be composed of three lists; one for processes deployed (registered) in activiti, one for tasks to start and another one for tasks of currently logged in user.

Since there is only one process definition deployed, first list will show only one item. A user will start the process by clicking [New] link next to it, and activiti will create a task per our process definition which will be listed in the second list. This list may contain arbitrary number of tasks from different process instances, which means there may be more than one instance of the financial report process at any given time. The user will start working on the task by clicking [Start] next to task. Now the task will move into the next list. It will wait there until the task is finished. User may logout and log back in to see his/her tasks waiting to be [Finish]ed. All of that information will be stored in activiti.

Other users may work on whatever tasks they should be working too.

Enough crappy screenshots. The code is simply one template and one controller to show lists and back actions.

#{extends 'main.html' /}
#{set title:'Home' /}

Welcome ${user}! <a href="@{Secure.logout()}">Logout</a>
<h3>Processes</h3>
#{list items:pdl, as:'pd'}
    [<a href="@{Application.start(pd.getId())}">New</a>]
    ${pd.getName()}<br/>
#{/list}
<h3>Tasks</h3>
#{list items:utl, as:'ut'}
    [<a href="@{Application.claim(ut.getId())}">Start</a>]
    ${ut.getName()}<br/>
#{/list}
<h3>My Tasks</h3>
#{list items:atl, as:'at'}
    [<a href="@{Application.complete(at.getId())}">Finish</a>]
    ${at.getName()}<br/>
#{/list}

@With(Secure.class)
public class Application extends Controller {
	
    @Inject static ProcessEngine pe;

    public static void index() {
        String user = Security.connected();
        List pdl = pe.getRepositoryService().
                createProcessDefinitionQuery().list();

        List utl = pe.getTaskService().createTaskQuery().
                taskUnnassigned().list();

    	List atl = pe.getTaskService().createTaskQuery().
                taskAssignee(Security.connected()).list();
        render(user, pdl, utl, atl);
    }
    
    public static void start(String pid){
    	String pdid = pe.getRepositoryService().
                createProcessDefinitionQuery().
                processDefinitionId(pid).
                singleResult().
                getId();
        pe.getRuntimeService().startProcessInstanceById(pdid);
        index();
    }
    
    public static void claim(String tid){
        pe.getTaskService().claim(tid, Security.connected());
        index();
    }

    public static void complete(String tid){
        pe.getTaskService().complete(tid);
        index();
    }
}

There is only one piece missing in this simple example: how did i deploy my process definition in the first place? Well, it is the dirtiest hack possible. I put it into the plugin where i initialized ProcessEngine. There are many ways to deploy a process definition into activiti and i didn’t want to get into the specifics. I used a mem database for the same reason, so none of these were persistent. I tried to see if the plugin would actually work and it did! On the other hand, if i wanted something in production i know it is a matter of configuration.

I said all i can about play’s plugin system in the previous post, it is great. But since it was about initializing activiti i didn’t say anything about it. AFAICT, it is one of the few libraries/frameworks that delivers what it advertises. It is light-weight, fast and simple. They say a BPM engine should be working in every Java environment, and this ‘helloworld’ is one example. API is clean, well documented and easy to work with. As a developer who correlates BPM engines to application servers that cannot start in under a minute, i am really pleased with what i was able to achieve. Overall experience with activiti is simply great.

A prototype play! plugin for activiti integration

Tue, 28 Dec 2010 22:20:00 +0200

Nowadays, i am back on play! trying to do things with somewhat less-documented feature of plugins. There is a project on the horizon which needs tight integration with a BPM engine, namely activiti, so i thought it’s a good time to try and deal with it as a play plugin or play module.

I said “plugin or module” deliberately, because i wasn’t aware of a plugin system other than the module system, which is basically described as just another play application. Actually i was wondering how it was possible to inject guice or spring dependencies into my application using just-another-application. Then i found out about the file play.plugins and saw that it was not magic at all (ability to code scala is still magic though).

I am also new to activiti and BPM, but i’ll try to explain what i made of it, at least the part that interested me. It is an application that uses a database to store some process definitions and ongoing processes, while executing everything according to the definition. Processes may be defined with a language called BPMN 2.0 which supposedly can be written by business-people so we shouldn’t care about the complexity or validity of them! We register these definitions into activiti which in turn may be instantiated into process instances using a Java API. Then, we may monitor and/or act on those instances using the same API. Activiti also provides a REST API which is marked as experimental so i didn’t touch it. Simple enough.

I still am not sure how will i use it in my future project but there is a point which will be needed anyway so i tried to cover that in this primitive plugin: initializing activiti. This means getting the hold of a ProcessEngine instance.

A play module may put any classpath dependency in its lib/ directory but its not created with $ play new-module. I mkdired it first and copied activiti related jars here. Activiti itself comes in a single jar named activiti-engine and this was the first jar to be copied. Because it has been developed using spring, it requires some spring jars to be on the classpath. It is possible to get them using spring module but i wasn’t sure if it works with 1.1 or is compatible with what activiti requires. I simply copied spring-core, spring-beans and spring-asm. It uses MyBatis to access the required database so its jar is to be included too. The last state of lib is like

activiti-engine-5.0.jar
mybatis-3.0.1.jar
spring-asm-3.0.3.RELEASE.jar
spring-beans-3.0.3.RELEASE.jar
spring-core-3.0.3.RELEASE.jar

$ play new-module created some unnecessary directories [for this example] like app/ and conf/ so i removed them before starting anything.

Activiti documentation states that ProcessEngine is thread-safe and it will be OK to initialize and destroy it upon container boot and shutdown. It gives an example for plain Servlet environments which seems easy to translate to a play plugin.

A play plugin is a class that extends PlayPlugin which is able to modify just about everything framework does, by overriding the methods. In order to function, it needs to have a default constructor and has to be enabled by a play.plugins file which should be somewhere in the root of the classpath. $ play new-module creates such a file in src/ so i edited it to put my plugin classname.

1100:play.modules.activiti.ActivitiPlugin

The number before the colon defines the priority of the plugin which simply defines the order that plugins are run. I’ll explain why i picked 1100 shortly. The class is as simple as it gets.

package play.modules.activiti;

import org.activiti.engine.ProcessEngines;
import play.PlayPlugin;

public class ActivitiPlugin extends PlayPlugin {
  @Override
  public void onApplicationStart() {
    ProcessEngines.init();
  }     
  @Override
  public void onApplicationStop() {
    ProcessEngines.destroy();
  }     
}

We have successfully initialized our ProcessEngines per activiti documentation. But it is obvious that this is less than useful. We should find a way to get a ProcessEngine instance in our controllers. I don’t know if it’s OK here but i am a huge fan of dependency injection so i would love to be able to just @Inject it. I knew that there is a @javax.inject.Inject support in play but i had to peek at the source code of guice module to figure out how to make it work. It turned out to be the simplest modification ever.

public class ActivitiPlugin extends PlayPlugin implements BeanSource {
  @Override
  public void onApplicationStart() {
    ProcessEngines.init();
    Injector.inject(this);
  }     
  @Override
  public  T getBeanOfType(Class clazz) {
    if (clazz.equals(ProcessEngine.class))
      return (T) ProcessEngines.getDefaultProcessEngine();
    return null;
  }     
  //...   
}

The Injector.inject() in second line crawles our classes for @Inject annotated static fields and applies getBeanOfType() method to its type when it finds one. And my methods gives the ProcessEngine instance if the type of the field is actually a ProcessEngine. Now it is possible to get it in any controller, mailer or job

public class Application extends Controller {
  @Inject
  static ProcessEngine pe;

  public static void index() {
    List pdl = pe.getRepositoryService()
      .createProcessDefinitionQuery().list();
    render(pdl);
  }     
}

That example gets the list of registered (deployed) process definitions. After getting that ProcessEngine, rest is details about how you want to use it which i will figure out after getting into the project.

On that priority 1100; since i am utilizing @Inject here, there is a huge possibility that guice or spring plugin will interfere. Both of the plugins’ priorities are 1000 so activiti plugin will be loaded after them and overwrite anything that has been injected by them. I am still not sure about this whole mechanism and i feel i’ll probably be forced to depend on spring plugin sooner or later.

For the completeness of the example, just in case anyone wants to work with it, there is one more step. Activiti offers you the easy way of defining your activiti database in a configuration file named activiti.cfg.xml. You should obtain one from the documentation and modify it under your conf/ directory. But be careful about it as you would probably need h2 or jdbc-driver jar in your lib/.

About that module/plugin confusion in the beginning i think it is more clear now. From what i understood, a module is just another play application, more precisely a web application, that you may want to include in several projects. Like a google-maps widget for a yabe. Or an authentication system that sits on top your application, saying hello to logged-in users and showing login controls for others. Plugins on the other hand, extend the framework for developer needs. Like my primitive activiti integration, or a hypothetical drools integration which may fire rules upon a valid request using a simple annotation.

Plugins also reminded me of a recent chat with @mozcelebi and @burakdalgic, about whether it is possible in play to do things like django middlewares are able to do. It seems they are the way to go for request pre/post processing and more. I am thinking about a middlewares plugin for example.

Next post ~~will be~~ is about a simple activiti application on this.

Scala for a newbie

Mon, 27 Dec 2010 10:31:00 +0200

Finally had the chance to get my hands dirty with scala. I’ve been trying to learn the stuff for almost a year now but you know… it’s not easy without something to build. Reading and trying the examples in the book doesn’t get one far unless he has an objective.

Anyways, what i was trying to do was to run some statistics over the performance data in some csv files. The files were the product of a python script which crawles our cluster for hadoop job logs and combines them with the timing data in job history. We ran 64 different test so there were 64 csv files. Each file contains a number of data sets which represents repeated tests.

I thought it will be a simple spreadsheet job but i didn’t find a way to import those files into one workbook (i need to learn one of these programs soon). I started to copy each file manually and got bored on the fifth. I decided it will be easier to run the statistics in python then pipe the data to google charts. On the third line of that python script, i saw the opportunity to get it done in scala.

Being an experienced scala newbie i knew that IDE plugins for scala were still not up to the rhythm so i skipped plugging anything to eclipse and went straight to vim. I also thought that it will be hard to go between write-code and compile cycle since this would be my first piece of scala code. After fixing some configuration problems with sbt and codefellow i had my first code running.

object Test { 
    def main(args: Array[String]) {
        println("TEST")
    }
}

Starting small, i had a fileReader first:

def lines(file: java.io.File) =
    io.Source.fromFile(file).getLines.toList

Little more verbose than pythonic file(filename).readlines() but way too concise for java counterpart. The most interesting part may be the lack of a return. Don’t expect any because last line actually produces a value which gets returned automatically.

Hadoop Job and TaskAttempt stats were in the same file so i had to filter them accordingly in order to generate different statistics for job times and task times. I have processed jobs with three simple functions (one of which being lines).

def jobTimes(file: java.io.File) = 
    for {
        l <- lines(file)
        if l.startsWith("job")
        jt = l.split(",")(1)
    } yield jt

def jobs(files: List[java.io.File]) = 
    for {
        f <- files
        jt = jobTimes(f)
        av = average(jt)
        sd = math.sqrt(variance(jt, av))
    } yield (f.getName, av, sd)

Doesn’t look scary right. Tell me about it. I have read the for part of the book at least twice but still didn’t remember the difference between {} and () for that for statements. yield is familiar from python but the placement is important. It can’t be in the code block. I am still trying to get the concept but this feels more correct than python. In the end yield is not related to the code so it should be outside.

Average function is actually as concise as it gets

def average(lst: List[Double]) =
    lst.sum / lst.length

But i didn’t like the looks of that length so i tried a pure approach. I was trying to learn after all

def average(lst: List[Double]) = {
    val mc = ((0.0, 0.0) /: lst.map(x => (x, 1.0))) {
        (a,b) => 
        (a._1 + b._1, a._2 + b._2)
    }
    mc._1 / mc._2
}

Now that is scarier, huh. You gotta trust me it’s not that complicated once you get to recognize it. It is a scala counterpart of a slightly modified version of a MR job that calcuate average. You emit a 1 with your values, and your reducers add each value in itself. In the end you have collected your values and those 1s which equals to the number of values. So you simply divide em in post-processing. In this code piece the first 0.0, 0.0 defines your initial values /: operator means “i am starting a fold (reduce) operation with this initial value over this list”. Since list contains only the values to collect, i had to map it to (value, 1.0) tuples in order to comply with the mechanic just described. x => (x, 1.0) is lambda x: (x, 1.0) in python, and i guess => is better than a keyword BTW. After our map phase, reduce phase in {} block simply collects these values. Weird looking _1 and _2 accessors are for tuples.

As a conclusion, after some gotchas in points just told, formatting an output was the easy part. I am aware of the fact that i probably used only 1 percent of what’s available in scala and there probably are better ways to do what i just did. But AFAIK scala is not a all-or-nothing language, if it was, no-one would be able to use it because it’s huge for an average developer to swallow as a whole. One should find similarities with other languages, do things in-effectively at first then slowly learn the scala way.