Nikos D.

Here are some more benchmarks for a speed comparison between Ruby 1.8 (MRI), Ruby 1.9 (YARV), Rubinius and JRuby (brand new 1.1.1 release) and this time included in the package is also MacRuby 0.1 (r88 from the test branch to be precise) which I recently discovered (more on MacRuby soon in a seperate post ;) ). Also, this time is measured by the Benchmark lib and not from the time utility.

The benchmark is based on the Graticule library which apart from the geocoding it provides (including Google, Yahoo, etc...) it also has the ability to calculate distances with various methods. For this benchmark, I first got the cooordinates of the biggest European cities and saved them as Locations (Graticule's class) in a YAML file. Then (and this is where the benchmark begins) I loaded them from the YAML file and calculated each pairs' distance (=> all cities with all cities) using the default method that Graticule uses. Here is the code* :

require 'benchmark'
require 'yaml'
require File.dirname(__FILE__) + '/../../lib/graticule'

class DistanceCalculationBenchmarks
def self.european_cities(print_results = false)
  Benchmark.bmbm do |x|
    x.report("Largest european cities tour:") {
     5.times do
     
       locations = YAML.load_file('cities.yml')
     
       costs = {}
       locations.each_index do |loc|
         costs[loc] = {}
       end
     
       locations.each_index do |loc_a|
         locations.each_index do |loc_b|

           if loc_a != loc_b and costs[loc_a][loc_b] == nil
             d = locations[loc_a].distance_to(locations[loc_b])
             costs[loc_a][loc_b] = costs[loc_b][loc_a] = d
           end
         end
       end
     
     end
    }
  end

end
end

DistanceCalculationBenchmarks.european_cities

and here are the results, first in raw text:

Running Distance Calcutions using Graticule for Europe's 100 Largest cities with five iterations & rehearsal.
Current time: Thu Apr 24 02:26:39 +0300 2008

Ruby 1.8...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:   1.270000   0.010000   1.280000 (  1.288320)
-------------------------------------------------------- total: 1.280000sec

                                  user     system      total        real
Largest european cities tour:   1.270000   0.010000   1.280000 (  1.279469)
---------------------------------------------------------------------------

Ruby 1.9...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:   1.270000   0.020000   1.290000 (  1.289113)
-------------------------------------------------------- total: 1.290000sec

                                  user     system      total        real
Largest european cities tour:   1.280000   0.010000   1.290000 (  1.284230)
---------------------------------------------------------------------------

Rubinius (bbe450f22) (04/15/2008)...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:  46.258145   0.000000  46.258145 ( 46.258139)
------------------------------------------------------- total: 46.258145sec

                                  user     system      total        real
Largest european cities tour:  46.594087   0.000000  46.594087 ( 46.594102)
---------------------------------------------------------------------------

JRuby 1.1.1 with no flags...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:   3.717000   0.000000   3.717000 (  3.718000)
-------------------------------------------------------- total: 3.717000sec

                                  user     system      total        real
Largest european cities tour:   2.949000   0.000000   2.949000 (  2.950000)
---------------------------------------------------------------------------

JRuby 1.1.1 and flags -J-server ...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:   4.820000   0.000000   4.820000 (  4.821000)
-------------------------------------------------------- total: 4.820000sec

                                  user     system      total        real
Largest european cities tour:   2.191000   0.000000   2.191000 (  2.191000)
---------------------------------------------------------------------------

JRuby 1.1.1 and flags -X+C...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:   3.701000   0.000000   3.701000 (  3.701000)
-------------------------------------------------------- total: 3.701000sec

                                  user     system      total        real
Largest european cities tour:   3.030000   0.000000   3.030000 (  3.030000)
---------------------------------------------------------------------------

JRuby 1.1.1 and flags -J-server -X+C...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:   5.601000   0.000000   5.601000 (  5.601000)
-------------------------------------------------------- total: 5.601000sec

                                  user     system      total        real
Largest european cities tour:   2.647000   0.000000   2.647000 (  2.647000)
---------------------------------------------------------------------------

MacRuby 0.1...
Rehearsal -----------------------------------------------------------------
Largest european cities tour:   3.480000   0.340000   3.820000 (  5.454403)
-------------------------------------------------------- total: 3.820000sec

                                  user     system      total        real
Largest european cities tour:   3.450000   0.330000   3.780000 (  5.241581)
---------------------------------------------------------------------------

and in an obviously more elegant way :

Ok, the first thing I noticed is that once again Rubinius is not really usable for this particular piece of code (which was also the case with my precious benchmark) :( JRuby was also slower by 50% (on average) which means that I am either a bless or a curse for the Jruby team (it's the second time I give them "bad" results from a benchmark). Finally, the newcomer, aka MacRuby, is somewhere near (on average again) with JRuby. Not bad for a 0.1 release but it's not really that usable yet (my previous benchmark doesn't work yet - I created a nice ticket in their tracker... :D This bug seems to be fixed on trunk, gonna check it out later).

The specs of the machine and the misc versions are the same with what they were before except the updated Rubinius.

NOTE: After I ploted the chart I noticed that in the code I was using a hash for the costs and not an array. I re-run the benchmark with an array instead of a hash and even though it didn't make any serious difference in Ruby 1.8, 1.9, Rubinius and MacRuby JRuby was half or a second faster than before. In case anyone wants the new results let me know and I will try to update the post (text and graph). Otherwise I leave it as it is with this note :)

* : This may be the code for the benchmark but I also have a custom patched Graticule lib because Graticule requires the ActiveSupport gem which for the time being (and until 2.1 is released) it's incompatible with MacRuby. I think it was ok with the rest BUT it only uses 3-4 things from ActiveSupport and I prefer to copy them inside the Graticule and get rid of the "require 'active_support'" which makes the execution time increase about 10%. More on this also on a upcoming post.

So, that's it for the time being. Have fun...

ruby,jruby,rubinius,macruby,benchmarks,programming,mri,yarv

This week I've been playing with charlie (a Genetic Algorithms library) and I've been rewriting the TSP example that's included with it so I can have a very very simple "prototype" for the GRuVeR project. I must say that I kind of left it incomplete and that's because I wanted to test on which RVMs it's running properly and after fixing it for the ones that it didn't, what's the performance on each of them.

Compatibility:
It turned out that it was running just fine to all of them except Rubinius. I managed to figure out what the problem was and after fixing it I jumped right away to the testing part :)

Comparisons:
Okay, I must say that I was expecting quite the opposite results (ok, almost the opposite). And why's that? Let's take a look:

Let's start with Ruby 1.8 (MRI):
$ time ruby tsp_solver.rb
real 0m34.940s
user 0m34.398s
sys 0m0.125s

Ruby 1.9 :
$ time ruby19 tsp_solver.rb
real 0m21.533s
user 0m20.736s
sys 0m0.119s

Let's move to the alternatives now.

First JRuby 1.1:
$ time jruby tsp_solver.rb
real 0m57.815s
user 0m48.651s
sys 0m0.751s

JRuby 1.1 with -J-server flag:
$ time jruby -J-server tsp_solver.rb
real 0m34.834s
user 0m37.714s
sys 0m0.667s

And finally Rubinius:
$ time rbx tsp_solver.rb
real 12m29.830s
user 12m20.105s
sys 0m4.219s

Graphics are always better:

Wow! There's quite some difference amongst them! To be honest, I really expected the vanilla Ruby to be the slowest (at least I hoped so) but no-no! It may be slower than 1.9 but it's faster than the rest of them. Obviously, Rubinius is out of the competition for this one. I believe that this specific code I tested is really an edge; the time required for rubinius to finish is practically infinite. After all it's the only VM that hasn't reached a 1.0 milestone yet so a lot of things will eventually get way better.

Update: Charles told me that running this example with JRuby and Java 1.6 results in much faster execution, faster or close to Ruby 1.9 to be exact. Quoting his words: "ok...and if it's not as fast as YARV, file a bug for me :) " . Heh, so I guess he is probably right. I will try to test it as soon as I set up Java 6 in my machine (off-topic: it's a shame that Leopard doesn't ship with Java 1.6 included...). And also thanks a lot Vladimir for letting me know about the -J-server flag (I'm not that familiar with Java).

Now, for those who want more info on the code and the machine (updated to include more details):
The machine is a MacBook Pro with 2.2GHz Intel Core 2 Duo and 2GB Ram running Mac OS X Leopard 10.5.2. The source code for the tsp_solver.rb can be found here and it requires the charlie gem (of course you can grab the code from the repository and pack it under a lib folder or something and require that instead in the tsp_solver).

Now, the exact RVMs versions:

• $ ruby -v
  ruby 1.8.6 (2007-09-24 patchlevel 111) [universal-darwin9.0]
• rbx -v
  rubinius 0.8.0 (ruby 1.8.6 compatible) (bbe450f22) (04/15/2008) [i686-apple-darwin9.2.2]
• $ ruby19 -v
  ruby 1.9.0 (2007-12-25 revision 14709) [i686-darwin9.2.2]
• $ jruby -v
  ruby 1.8.6 (2008-03-28 rev 6360) [i386-jruby1.1]
  And: $ java -version
  java version "1.5.0_13"
  Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_13-b05-237)
  Java HotSpot(TM) Client VM (build 1.5.0_13-119, mixed mode, sharing)
  

Please, please, please keep in mind that this code, although it has to do with an almost real-life situation, is really specific and targeted! Don't take this comparison as a measure of speed between different Ruby implementations! I did it because I was curious to see the results and I will do it again a) when new versions of these VMs come out and b) with most of the code that I'm going to write for this project.

Later today (or probably tomorrow) I'm also going to post the results of the profiler for each VM for those of you who are curious but lazy to run it on your machine :) In case someone wants to run it, the integer parameters of this call can be minimized (so you can reduce the time required for the code to finish) :

population = Population.new(@genotype,100).evolve_silent(1000)

The first one is about the size of the population of each generation and the second the number of the generations that it is required to be generated. The smaller, the faster but the bigger, the better results (as this algorithm finds an estimation and not the exact solution for the *randomly created* Traveling Salesman Problem). Another parameter that could be reduced is the number of cities/stops which is the parameter of TSPSolver::by_example with a default value of 100. Try to reduce them altogether for better balance ;)

Have fun!