Testing Rails 3.0.3 with Ruby 1.9.2 and REE 1.8.7

Posted on January 08, 2011

Upgrade: There have been some confusion because of the faulty way I wrote the conclusion. If you read it you will get under the impression that I ran the tests using Rails’ development mode, which is known to not be used for testing and profiling. I know that, of course. When I say “production environment”, I mean my VPS machine where my application is hosted. And when I say “development environment”, I mean my Macbook Pro, that I use for development. That’s the confusion :-)

This is not news but I tried to do a small test with my own Enki-modified blog upgraded to Rails 3.0.3 made compatible with both Ruby Enterprise Edition (REE) 1.8.7 2010.02 and Ruby 1.9.2-p136.

Many micro-benchmarks show that Ruby 1.9.2 eats 1.8.7 for lunch. But the reality of a full featured Rails 3 application may be different.

I tested both using the Passenger 3 Standalone and simple local httperf. This is not an optimum environment for benchmarking but we can have a broad idea.

In terms of memory usage, 1.9.2 is a bit lighter. passenger-memory-stats shows that each Ruby process is consuming an average of 74MB of RAM. The REE version consumes around 106MB. So 1.9.2 is doing a better job in memory consumption.

But when I run httperf (httperf —server localhost —port 3000 —uri /2010/12/31/noticias-do-front-10 —num-conn 10 —num-call 2 —rate 2 —timeout 5) I get this result:

Ruby 1.9.2-p136

httperf --timeout=5 --client=0/1 --server=localhost --port=3000 --uri=/2010/12/31/noticias-do-front-10 --rate=2 --send-buffer=4096 --recv-buffer=16384 --num-conns=10 --num-calls=2
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 10 requests 20 replies 20 test-duration 6.249 s

Connection rate: 1.6 conn/s (624.9 ms/conn, <=6 concurrent connections)
Connection time [ms]: min 1509.8 avg 2086.8 max 3093.7 median 1788.5 stddev 562.5
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 2.000

Request rate: 3.2 req/s (312.5 ms/req)
Request size [B]: 93.0

Reply rate [replies/s]: min 2.8 avg 2.8 max 2.8 stddev 0.0 (1 samples)
Reply time [ms]: response 1035.8 transfer 7.5
Reply size [B]: header 652.0 content 38491.0 footer 2.0 (total 39145.0)
Reply status: 1xx=0 2xx=20 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.31 system 2.51 (user 5.0% system 40.1% total 45.1%)
Net I/O: 122.6 KB/s (1.0*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

REE 1.8.7 2010.02

httperf --timeout=5 --client=0/1 --server=localhost --port=3000 --uri=/2010/12/31/noticias-do-front-10 --rate=2 --send-buffer=4096 --recv-buffer=16384 --num-conns=10 --num-calls=2
httperf: warning: open file limit > FD_SETSIZE; limiting max. # of open files to FD_SETSIZE
Maximum connect burst length: 1

Total: connections 10 requests 20 replies 20 test-duration 4.899 s

Connection rate: 2.0 conn/s (489.9 ms/conn, <=2 concurrent connections)
Connection time [ms]: min 359.0 avg 399.5 max 510.9 median 376.5 stddev 45.2
Connection time [ms]: connect 0.1
Connection length [replies/conn]: 2.000

Request rate: 4.1 req/s (244.9 ms/req)
Request size [B]: 93.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 199.3 transfer 0.3
Reply size [B]: header 634.0 content 38491.0 footer 2.0 (total 39127.0)
Reply status: 1xx=0 2xx=20 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.68 system 3.67 (user 13.8% system 74.9% total 88.7%)
Net I/O: 156.4 KB/s (1.3*10^6 bps)

Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

Comparison

From these simple numbers we see that the 1.9.2 version is almost 30% slower overall. This is consistent with what I’ve been experiencing in my production environment (I don’t have production numbers right now).

I ran that test on the development environment, using sqlite3 underneath. But it is slower on MySQL as well. This is the Gemfile that I am using. All the gems are the same, but the database drivers.

I suspect one reason for that is the overhead dealing with Strings, but this is just speculating. Does anyone know of another Rails 3 benchmarking with both Rubies? I would be interested in learning a bit more about this behavior.