More speed comparisons of Ruby 1.8, Ruby 1.9, JRuby, Rubinius and MacRuby using Graticule

Thursday, April 24, 2008 1 comments

Here are some more benchmarks for a speed comparison between Ruby 1.8 (MRI), Ruby 1.9 (YARV), Rubinius and JRuby (brand new 1.1.1 release) and this time included in the package is also MacRuby 0.1 (r88 from the test branch to be precise) which I recently discovered (more on MacRuby soon in a seperate post ;) ). Also, this time is measured by the Benchmark lib and not from the time utility.

The benchmark is based on the Graticule library which apart from the geocoding it provides (including Google, Yahoo, etc...) it also has the ability to calculate distances with various methods. For this benchmark, I first got the cooordinates of the biggest European cities and saved them as Locations (Graticule's class) in a YAML file. Then (and this is where the benchmark begins) I loaded them from the YAML file and calculated each pairs' distance (=> all cities with all cities) using the default method that Graticule uses. Here is the code* :



require 'benchmark'
require 'yaml'
require File.dirname(__FILE__) + '/../../lib/graticule'

class DistanceCalculationBenchmarks
def self.european_cities(print_results = false)
Benchmark.bmbm do |x|
x.report("Largest european cities tour:") {
5.times do

locations = YAML.load_file('cities.yml')

costs = {}
locations.each_index do |loc|
costs[loc] = {}
end

locations.each_index do |loc_a|
locations.each_index do |loc_b|

if loc_a != loc_b and costs[loc_a][loc_b] == nil
d = locations[loc_a].distance_to(locations[loc_b])
costs[loc_a][loc_b] = costs[loc_b][loc_a] = d
end
end
end

end
}
end

end
end

DistanceCalculationBenchmarks.european_cities


and here are the results, first in raw text:



Running Distance Calcutions using Graticule for Europe's 100 Largest cities with five iterations & rehearsal.
Current time: Thu Apr 24 02:26:39 +0300 2008

Ruby 1.8...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 1.270000 0.010000 1.280000 ( 1.288320)
-------------------------------------------------------- total: 1.280000sec

user system total real
Largest european cities tour: 1.270000 0.010000 1.280000 ( 1.279469)
---------------------------------------------------------------------------

Ruby 1.9...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 1.270000 0.020000 1.290000 ( 1.289113)
-------------------------------------------------------- total: 1.290000sec

user system total real
Largest european cities tour: 1.280000 0.010000 1.290000 ( 1.284230)
---------------------------------------------------------------------------

Rubinius (bbe450f22) (04/15/2008)...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 46.258145 0.000000 46.258145 ( 46.258139)
------------------------------------------------------- total: 46.258145sec

user system total real
Largest european cities tour: 46.594087 0.000000 46.594087 ( 46.594102)
---------------------------------------------------------------------------

JRuby 1.1.1 with no flags...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 3.717000 0.000000 3.717000 ( 3.718000)
-------------------------------------------------------- total: 3.717000sec

user system total real
Largest european cities tour: 2.949000 0.000000 2.949000 ( 2.950000)
---------------------------------------------------------------------------

JRuby 1.1.1 and flags -J-server ...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 4.820000 0.000000 4.820000 ( 4.821000)
-------------------------------------------------------- total: 4.820000sec

user system total real
Largest european cities tour: 2.191000 0.000000 2.191000 ( 2.191000)
---------------------------------------------------------------------------

JRuby 1.1.1 and flags -X+C...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 3.701000 0.000000 3.701000 ( 3.701000)
-------------------------------------------------------- total: 3.701000sec

user system total real
Largest european cities tour: 3.030000 0.000000 3.030000 ( 3.030000)
---------------------------------------------------------------------------

JRuby 1.1.1 and flags -J-server -X+C...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 5.601000 0.000000 5.601000 ( 5.601000)
-------------------------------------------------------- total: 5.601000sec

user system total real
Largest european cities tour: 2.647000 0.000000 2.647000 ( 2.647000)
---------------------------------------------------------------------------

MacRuby 0.1...
Rehearsal -----------------------------------------------------------------
Largest european cities tour: 3.480000 0.340000 3.820000 ( 5.454403)
-------------------------------------------------------- total: 3.820000sec

user system total real
Largest european cities tour: 3.450000 0.330000 3.780000 ( 5.241581)
---------------------------------------------------------------------------



and in an obviously more elegant way :





Ok, the first thing I noticed is that once again Rubinius is not really usable for this particular piece of code (which was also the case with my precious benchmark) :( JRuby was also slower by 50% (on average) which means that I am either a bless or a curse for the Jruby team (it's the second time I give them "bad" results from a benchmark). Finally, the newcomer, aka MacRuby, is somewhere near (on average again) with JRuby. Not bad for a 0.1 release but it's not really that usable yet (my previous benchmark doesn't work yet - I created a nice ticket in their tracker... :D This bug seems to be fixed on trunk, gonna check it out later).



The specs of the machine and the misc versions are the same with what they were before except the updated Rubinius.



NOTE: After I ploted the chart I noticed that in the code I was using a hash for the costs and not an array. I re-run the benchmark with an array instead of a hash and even though it didn't make any serious difference in Ruby 1.8, 1.9, Rubinius and MacRuby JRuby was half or a second faster than before. In case anyone wants the new results let me know and I will try to update the post (text and graph). Otherwise I leave it as it is with this note :)



* : This may be the code for the benchmark but I also have a custom patched Graticule lib because Graticule requires the ActiveSupport gem which for the time being (and until 2.1 is released) it's incompatible with MacRuby. I think it was ok with the rest BUT it only uses 3-4 things from ActiveSupport and I prefer to copy them inside the Graticule and get rid of the "require 'active_support'" which makes the execution time increase about 10%. More on this also on a upcoming post.

So, that's it for the time being. Have fun...


,,,,,,,

Migrating to Ruby 1.9 (YARV): Posts you have to read!

Monday, April 21, 2008 1 comments

So, you are reading everywhere that Ruby 1.9 is faster than 1.8 (even I say so! :D ) and you want to give it a try or even port your existing code to it. Where should you start from? Here's a collection of links to posts, pages, discussions, etc with useful information.

First of all, here is a video talk from Matz for Google Tech Talks:

Also, some presentations:

  1. Matz on Ruby 1.9 from his Rubyconf 2007 talk.
  2. Migrating to Ruby 1.9, by Bruce Williams from his Scotland on Rails and Austin on Rails talks.
  3. Wild & Weird Ideas: An overview of Ruby 1.9, by Hlame from his LRUG presentation.

Blog posts covering multiple differencies between the two versions of Ruby:

Upcoming or recently published books that cover Ruby 1.9:

And after you've finished reading all the above and want an almost complete listing of changes (!!!) you can find it on this post from eigenclass. Have fun :)

Bonus section:

And now, you also want some info on how to get Ruby 1.9 side-by-side with the good-old 1.8? Ok, here are some links for you too:

,,,,

Tiny (but maybe important) Rubinius tidbits

Saturday, April 19, 2008 1 comments
As you play with a tool you progressively discover various small tidbits about it. I have a few in mind and I thought it would helpful to share them in case you encounter them and get this weird expression on your face of "huh?".

1. The multiple assignment operator
If you are used to use it's return values you'll have to think again cause in Rubinius it doesn't return the array with the new values of the variables. So let's say that you were doing this in Ruby 1.8, Ruby 1.9, JRuby, etc... :
p((a,b = 'some value', 'and some other value'))
This would result in :
["some value", "and some other value"]

In Rubinius it's not the case:
true

Hmm... Pretty nasty if you depend on the result...

Of course, this is not something really common, but in my case I was (unnecessarily) returning an array of values from inside a method that could be later used from the calling one. As the people at #rubinius pointed out I could do it with a much more nice and elegant way. The excellent quote after my "dough! Why didn't I think this in the first place?" was:
"See? Rubinius makes you write better code!" - Nice one :D

So, keep that in mind for the time being. They are thinking to make this behave as in the rest VMs but it's not sure yet since from what I've heard it does require some additional effort. Why don't you try yourself?

In contrast, the JRuby team has an open ticket about a possible optimization where the interpreter will be able to understand whether the value is going to be used or not, thus avoiding the unnecessary Array allocation - cool!

2. The nasty Symbol#to_proc and the splat ( * ) operator.
I've given some links about this issue before but I think here is more appropriate and some more explanation would be nice.

First of all, for those who don't know what does this method do you can find a short introduction on this post from Dave Thomas and from plenty other resources (Dr Nic, Invisible Blocks and InfoQ) including a railcast episode for the more video-podcast-savvy people. Keep also in mind the possible downside of using this method as noted by Pratik Naik.

Shall we go on now? Nice.

There are various approaches on this implementation like the one from Ruby Facets, Rails and Rubinius one. There's even a different proposal that got rejected from the Rails core team. As I mentioned in the appropriate ticket in Lighthouse, the current implementation doesn't work as it should. I haven't managed to track the problem down but it seems to me that it has something to do with the splat operator. Since I'm not sure, further investigation on this weird behavior is needed. For the time being I found a temporary solution to this that works fine with my examples. For those of you who are lazy enough to go and see all these links here are some examples:

(1..100).map(&:to_s) => ["1", "2", "3", (....)
(1..100).inject(&:+) => 5050
These work just fine.
But:
[(1..10).to_a, (1..10).to_a].map(&:to_s)
ArgumentError: wrong number of arguments (got 9, required 1)

(1..100).inject(&:to_s)
ArgumentError: wrong number of arguments (got 1, required 0)

Are obviously not.

Now, my (not so pretty) fix to this problem is this:
class Symbol
def to_proc
Proc.new { |*args|
obj = args.shift
args.empty? ? obj.__send__(self) : obj.__send__(self, *args)
}
end
end
which only breaks on the last example used before:
(1..100).inject(&:to_s)
ArgumentError: wrong number of arguments (got 1, required 0)
and that's because the #inject method yields two parameters in the block (and so it makes sense for this to break I believe and that's why I didn't include a test case for something like this in the specs I wrote for the patch.).

Feel free to share your thoughts.

Speed comparison of Ruby 1.8, Ruby 1.9, JRuby and Rubinius using a genetic algorithm

Wednesday, April 16, 2008 6 comments
This week I've been playing with charlie (a Genetic Algorithms library) and I've been rewriting the TSP example that's included with it so I can have a very very simple "prototype" for the GRuVeR project. I must say that I kind of left it incomplete and that's because I wanted to test on which RVMs it's running properly and after fixing it for the ones that it didn't, what's the performance on each of them.

Compatibility:
It turned out that it was running just fine to all of them except Rubinius. I managed to figure out what the problem was and after fixing it I jumped right away to the testing part :)

Comparisons:
Okay, I must say that I was expecting quite the opposite results (ok, almost the opposite). And why's that? Let's take a look:

Let's start with Ruby 1.8 (MRI):
$ time ruby tsp_solver.rb
real 0m34.940s
user 0m34.398s
sys 0m0.125s

Ruby 1.9 :
$ time ruby19 tsp_solver.rb
real 0m21.533s
user 0m20.736s
sys 0m0.119s

Let's move to the alternatives now.

First JRuby 1.1:
$ time jruby tsp_solver.rb
real 0m57.815s
user 0m48.651s
sys 0m0.751s

JRuby 1.1 with -J-server flag:
$ time jruby -J-server tsp_solver.rb
real 0m34.834s
user 0m37.714s
sys 0m0.667s

And finally Rubinius:
$ time rbx tsp_solver.rb
real 12m29.830s
user 12m20.105s
sys 0m4.219s

Graphics are always better:

Wow! There's quite some difference amongst them! To be honest, I really expected the vanilla Ruby to be the slowest (at least I hoped so) but no-no! It may be slower than 1.9 but it's faster than the rest of them. Obviously, Rubinius is out of the competition for this one. I believe that this specific code I tested is really an edge; the time required for rubinius to finish is practically infinite. After all it's the only VM that hasn't reached a 1.0 milestone yet so a lot of things will eventually get way better.

Update: Charles told me that running this example with JRuby and Java 1.6 results in much faster execution, faster or close to Ruby 1.9 to be exact. Quoting his words: "ok...and if it's not as fast as YARV, file a bug for me :) " . Heh, so I guess he is probably right. I will try to test it as soon as I set up Java 6 in my machine (off-topic: it's a shame that Leopard doesn't ship with Java 1.6 included...). And also thanks a lot Vladimir for letting me know about the -J-server flag (I'm not that familiar with Java).

Now, for those who want more info on the code and the machine (updated to include more details):
The machine is a MacBook Pro with 2.2GHz Intel Core 2 Duo and 2GB Ram running Mac OS X Leopard 10.5.2. The source code for the tsp_solver.rb can be found here and it requires the charlie gem (of course you can grab the code from the repository and pack it under a lib folder or something and require that instead in the tsp_solver).

Now, the exact RVMs versions:
  • $ ruby -v
    ruby 1.8.6 (2007-09-24 patchlevel 111) [universal-darwin9.0]
  • rbx -v
    rubinius 0.8.0 (ruby 1.8.6 compatible) (bbe450f22) (04/15/2008) [i686-apple-darwin9.2.2]
  • $ ruby19 -v
    ruby 1.9.0 (2007-12-25 revision 14709) [i686-darwin9.2.2]
  • $ jruby -v
    ruby 1.8.6 (2008-03-28 rev 6360) [i386-jruby1.1]
    And: $ java -version
    java version "1.5.0_13"
    Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13-b05-237)
    Java HotSpot(TM) Client VM (build 1.5.0_13-119, mixed mode, sharing)

Please, please, please keep in mind that this code, although it has to do with an almost real-life situation, is really specific and targeted! Don't take this comparison as a measure of speed between different Ruby implementations! I did it because I was curious to see the results and I will do it again a) when new versions of these VMs come out and b) with most of the code that I'm going to write for this project.

Later today (or probably tomorrow) I'm also going to post the results of the profiler for each VM for those of you who are curious but lazy to run it on your machine :) In case someone wants to run it, the integer parameters of this call can be minimized (so you can reduce the time required for the code to finish) :
population = Population.new(@genotype,100).evolve_silent(1000)
The first one is about the size of the population of each generation and the second the number of the generations that it is required to be generated. The smaller, the faster but the bigger, the better results (as this algorithm finds an estimation and not the exact solution for the *randomly created* Traveling Salesman Problem). Another parameter that could be reduced is the number of cities/stops which is the parameter of TSPSolver::by_example with a default value of 100. Try to reduce them altogether for better balance ;)

Have fun!

Google Summer of Code Proposal: GRuVeR: A General Ruby library for solving Routing Vehicle Problem

3 comments
This year is the last that I'm going to be a student, at least a bachelor one, since i've already completed (almost) my thesis and pretty soon I will be saying bye-bye to my University. I haven't really had the chance to participate in Google Summer of Code partially because I had never really heard of it (in time) and also because I didn't believe that I was able to contribute something really good (lack of confidence here :) ). But this year and after messing around with others' people code (heh) and some of my own (including my thesis) and with the help/ideas given by some friends I did finally sent a proposal.

The main concept is to implement a library for solving Vehicle Routing Problems in Ruby. How is this going to happen? The ground basis will be the beautiful Charlie library, written by Sander Land, which makes it really easy to write programs with Genetic Algorithms built in and really fast (in terms of time required to write the code :D). James also pointed out Gecode/R which is frontend to Gecode and uses another technique, Constraint Programming and I'm trying to think if and how this can also be used (a discussion can be found on the Gecode/R mailing list). I'm not going to get into details here; if anyone is interested a part of the proposal can be found in the Ruby mailing list and the full version is available as pdf here. Feel free to send in any thoughts :).

I'm really waiting for the ranking results and I reeeeeally hope that it will get accepted even though it's a really specific and targeted project when compared with others like contributing to the alternative VMs, refactoring RDoc (although version 2 is out) and such. The good thing with implementing this, is that it will produce some really nice benchmarks between the different VMs and also create some feedback with suggestions/patches/fixes (here are some examples from this Monday's experiments I was doing). Regardless though of whether or not it gets accepted, I believe that I will finally write this thing cause after a chat I had with Sander, which by the way is my potential mentor to the project, he also seems interested in this and willing to help.

So, stay tuned for updates on this - later this day I'm going to post some benchmark results between almost all current Ruby VMs (Ruby 1.8, Ruby 1.9, JRuby 1.1 and Rubinius)* after running some example code I wrote for the Traveling Salesman Problem.

* I'm missing the C# implementation of Ruby here but I thought that it won't be able to run on my Mac so I didn't bother to try it. If I'm wrong please let me know and I will also set it up right away!

P.S.: Do you have a better name in mind for the project??? I'm not sure I like "GRuVeR". Wouldn't something like "delivery" sound better?

Not moved yet...

2 comments
It's been almost half year since my last post which was saying that I was going to move to nikosd.com. I haven't found the time and the money to pay a hosting server so i'm still here and since I have some interesting things to post I will do so here until I (finally) move to my own space.

Updates are coming soon with my Google Summer of Code proposal, some benchmarks between different Ruby VMs and misc other stuff.

Cheers