Rails on the Run

At work, we have an XML API that gets quite a lot of traffic. Last week I looked into improving its performance since we are expecting more traffic soon and want to make sure our response time is optimized.

My first thought was to make sure we had  an optimized ActiveSupport’s xmlmini backend. Rails 2.3.5 fixed some issues when using Nokogiri as a xmlmini so I switched to my favorite Ruby XML lib:

ActiveSupport::XmlMini.backend = 'Nokogiri'

I run some benchmarks using ab, httperf and jmeter but the results were not that great. I was so sure that switching from rexml to nokogiri would give me awesome results that I was very disappointed.

I was about to call Nick Adams (Nokogiri’s author) to insult him, blame him for _why’s disappearance and tell him that all his pro bono efforts were useless since my own app was not running much faster when switched to his library. As I was about to dial his number on my iPhone I had a crazy thought… maybe it was not Aaron’s fault, maybe it was mine.

So I took a break went to play some fuzzball and as I was being ridiculed by Italian coworker, Emanuele, I realized that most of our API calls were just simple HTTP requests with no XML payload, just some query params. However, we were generating a lot of XML to send back to the client and AS::XmlMini only takes care of the XML parsing, not the rendering.

The XML rendering is done by Jim Weirich’s pure Ruby builder library which is vendored in Rails. Builder does a good job, but I thought that maybe a C based library might improve the speed. A coworker of mine (James Bunch) recommended to look into faster-builder, a drop-in Builder replacement based on libxml. Unfortunately, the project doesn’t seem to be maintained and I decided to look into using Nokogiri XML builder instead. (Also, faster-builder’s author doesn’t like me very much while Aaron knows he’s one of my Ruby heroes so asking for help could be easier)

Some people reported having tried using Nokogiri as a XML builder but didn’t see much speed improvement. Because of the amount of magic required to render a rxml template, I was not really surprised but I decided to contact Aaron and ask him if he tried using his lib instead of builder in a Rails app. Aaron told me he gave it a try a while back and he helped me get my Rails app setup to render xml templates using Nokogiri.

The next step was simple, create a benchmark app and benchmark Builder vs Nokogiri using various templates. Here are the results I got using Ruby 1.9.1 (the Ruby version we use in production) and two sets of templates:

Builder small template, time per request: 15.507 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/builder_small
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/builder_small
Document Length:        216 bytes

Concurrency Level:      1
Time taken for tests:   3.101 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      114326 bytes
HTML transferred:       43200 bytes
Requests per second:    64.49 [#/sec] (mean)
Time per request:       15.507 [ms] (mean)
Time per request:       15.507 [ms] (mean, across all concurrent requests)
Transfer rate:          36.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    11   15   8.8     12      47
Waiting:        3   15   8.9     12      47
Total:         11   15   8.8     12      47

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     12
  75%     13
  80%     13
  90%     35
  95%     36
  98%     38
  99%     41
 100%     47 (longest request)

Nokogiri small template, time per request: 15.354 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/noko_small
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/noko_small
Document Length:        238 bytes

Concurrency Level:      1
Time taken for tests:   3.071 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      118717 bytes
HTML transferred:       47600 bytes
Requests per second:    65.13 [#/sec] (mean)
Time per request:       15.354 [ms] (mean)
Time per request:       15.354 [ms] (mean, across all concurrent requests)
Transfer rate:          37.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    11   15   8.6     12      39
Waiting:       11   15   8.6     12      39
Total:         11   15   8.6     12      39

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     12
  75%     12
  80%     13
  90%     35
  95%     36
  98%     37
  99%     38
 100%     39 (longest request)

Running the benchmarks many times showed that Nokogiri and Builder were taking more or less the same amount of time to builder a small template.

I then decided to try a bigger template, closer to what we have in production, here are the results:

Nokogiri longer template, time per request: 31.252 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/noko
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/noko
Document Length:        54398 bytes

Concurrency Level:      1
Time taken for tests:   6.250 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      10951200 bytes
HTML transferred:       10879600 bytes
Requests per second:    32.00 [#/sec] (mean)
Time per request:       31.252 [ms] (mean)
Time per request:       31.252 [ms] (mean, across all concurrent requests)
Transfer rate:          1711.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    24   31  11.3     26      62
Waiting:       23   30  11.3     24      61
Total:         24   31  11.3     26      62

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     27
  75%     27
  80%     29
  90%     54
  95%     55
  98%     58
  99%     59
 100%     62 (longest request)

Builder, longer template, Time per request: 140.725 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/builder
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/builder
Document Length:        54376 bytes

Concurrency Level:      1
Time taken for tests:   28.145 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      10947000 bytes
HTML transferred:       10875200 bytes
Requests per second:    7.11 [#/sec] (mean)
Time per request:       140.725 [ms] (mean)
Time per request:       140.725 [ms] (mean, across all concurrent requests)
Transfer rate:          379.83 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   127  141  24.6    132     331
Waiting:      126  139  23.6    130     328
Total:        127  141  24.6    132     331

Percentage of the requests served within a certain time (ms)
  50%    132
  66%    138
  75%    147
  80%    149
  90%    156
  95%    169
  98%    193
  99%    311
 100%    331 (longest request)

Wow, @tenderlove’s Nokogori just brought us a 4.5X speed improvement for this specific template.  100ms per request is probably not a big deal for most people and I have to say that Jim did a great job with Builder. However in my specific case, 100ms on a request being called thousands of times per hour is quite important.

(The benchmark app is available on github, feel free to fork it and benchmark your own templates)

Who would have thought that a man like this could save the day?!

The moral of the story is that adding a bit of tenderlove to your Ruby code can make it perform much much better!

Thank you Aaron ‘Tenderlove’ Patterson!

To celebrate the relaunch of this site and since we are waiting for Rails 3.0 beta to be released, I figured I should share with you what I worked on the other night.

I merged patches, refactored and released a new version of googlecharts, my Gem to create graphs using Google Chart API.

sudo gem install googlecharts

Here is a quick example of how the API works when dealing with a complex graph:

require 'gchart' # or require 'googlecharts' if you prefer to use the Googlecharts constant.
title = "Player Count"
size = "575x300"
data = [85,107,123,131,155,172,173,189,203,222,217,233,250,239,256,267,247,261,275,295,288,305,322,307,325,347,331,346,363,382,343,359,383,352,374,393,358,379,396,416,377,398,419,380,409,426,453,432,452,465,436,460,480,440,457,474,501,457,489,507,347,373,413,402,424,448,475,488,513,475,507,530,440,476,500,518,481,512,531,367,396,423,387,415,446,478,442,469,492,463,489,508,463,491,518,549,503,526,547,493,530,549,493,520,541,564,510,535,564,492,512,537,502,530,548,491,514,538,568,524,548,568,512,533,552,577,520,545,570,516,536,555,514,536,566,521,553,579,604,541,569,595,551,581,602,549,576,606,631,589,615,650,597,624,646,672,605,626,654,584,608,631,574,597,622,559,591,614,644,580,603,629,584,615,631,558,591,618,641,314,356,395,397,429,450,421,454,477,507,458,490,560,593]
Gchart.line(:title => title, :size => size, :data => data, :axis_with_labels => 'x,y', :line_color => '1e60cc', :axis_labels => [(1.upto(24).to_a << 1)], :max_value => 700, :custom => 'chg=10,15,1,0')

Which provides you with the url or image tag (or downloaded file) that produces the following graph:

This release works great with Ruby 1.9 and MacRuby, lots of bugs got fixed and some new features were added. Something a lot of people complained was that the gem was called googlecharts but that the main class was called Gchart. The problem was that I wrote my gem and called it Gchart and when I went to register the rubyforge project page, the name was already taken. In this release, I fixed this problem by allowing users to require and use the constant name they want, Gchart or Googlecharts. I also spent quite a lot of time cleaning up the old code which I was a bit ashamed of. Class variables are now removed and overall, the code should be a bit more sane.

The source code can be found in my GitHub accout and the documentation there.

This is huge!

While people still try to find some drama an in hypothetical war between rails and merb.

The Rails team and the Merb team announced working together on a joined version of the 2 frameworks. This is so exciting, nobody believed it could ever happen (I personally, had my serious doubt).

Yehuda had a great post laying down the plan and explaining things in details. Check out David’s post explaining why he wants us to work together and his vision of a better Ruby world.

I have to say that I have been impressed by the Rails core team and especially David (DHH).

I’ve known David for few years now and we had long/heated discussions on topics like i18n/l10n for Rails. David is known to be a very opinionated person. But if you come up with the right arguments, he can be convinced and when that happens, he is willing to move forward and shake things up.

This merge is a concrete example that David and the rest of the Rails team cares about Rails and the Ruby community more than we usually give them credit for. As a unified team, we are going to change the way web development in Ruby is done!

But what does it mean for you?

I put together a FAQ video here is the transcript:

Hi, I’m Matt Aimonetti from the merb core team and as you might have heard, a big announcement was made earlier today.

I did this video to hopefully answer the questions you might have.

Q: So what’s the big news?

• merb and rails team will work together on the next version of their frameworks
• merb 2.0 and rails 3.0 share the same common endpoint
• we realized we now have the same objectives and agreed on all the key principles.
• merb will provide Rails with agnosticism, modularity, improved performance and a public API.
• The end product will be called Rails 3.0 but what real matters is that it’s a huge gain for the entire community.

Q: What??? I thought there was a war going on between Rails and merb, what happened?

• There was no war between rails and merb
• We created merb because rails was not fitting us
• We wanted better performance, more choices/ more modularity and a Public API.
• The Rails team now embraces these concepts and want Rails to follow them, so why not work together?

Q: Wait, does that mean that merb is dead?

• Absolutely not!
• merb development won’t stop, we are going to keep on releasing updates until rails 3.0
• clear migration path, and upgrading to rails 3.0 will be as trivial as upgrading from rails 2.x to Rails 3.0

Q: What does the timeline look like?

We just started getting together and discuss the technical details. We are shooting for a release at RailsConf 2009. However it’s hard to estimate this kind of things so again, that’s just an estimate

Q: I just started a merb project, so what now?

I’m sure you had valid reasons to use merb, you needed modularity, performance and a solid API.

Keep on using Merb, we won’t let you down. The good news is that the next version of merb (rails 3.0) will be even awesomer!

Q: What about my client who was looking at using merb for his new project?

If your client is going to be using merb for valid reasons (and not, just because it’s not rails) he should still use merb, but with the full understanding that he/she will end up using Rails in 6 months or so. Again, Rails 3.0 will have what pushed you to use merb.

Q: I’ve been involved with the merb-book, what will happen with this project?

• rails 3.0 won’t get released right away
• still need awesome documentation
• if we look at rails 3.0 as merb 2.0, we can easily imagine how the current work can be extended to the new version.
• rails team will also include an evangelism team which I will be part of, so will be able to focus more on projects like the book.

Q: I’ve been working on a merb plugin, what should I do?

Keep working on it! We’ll assist you with the migration process and the new solid API.

Q: What if I still have questions?

Come talk with me, or any members from the new team. We’ll be open to hear your questions, worries, frustrations.

merb always valued its developers and we will continue to do so but at a bigger scale.

---

Concretely, nothing changes for Merb users. People loving Merb should not worry. The way you build merb apps won’t change until merb2.0/rails3.0. We will still work on your favorite framework and improve it.

Lori Holden worked on merb_sequel and we should release a 1.0 release of the plugin in a few days.

I’m sure this news comes as a shock for many of you, but try to not see Rails 3.0 as the way Rails is right now. Imagine a version of Rails with true modularity and agnosticism (sequel, DM and AR will still be supported) and the same type of performance as what you get with Merb. In other words, the rails world is about to discover the power of merb!

Here is what Yehuda explicitly says in his blog post:

• Rails will become more modular, starting with a rails-core, and including the ability to opt in or out of specific components. [...]
• We will port all of our performance improvements into Rails. This includes architectural decisions that are big performance wins.[..]
• As of Rails 3, Rails will have a defined public API with a test suite for that API. [..]
• Rails will still ship with the “stack” version as the default (just as Merb does since 1.0), but the goal is to make it easy to do with Rails what people do with Merb today.
• Rails will be modified to more easily support DataMapper or Sequel as first-class ORMs. [..]
• Rails will continue their recent embrace of Rack, which [..] will improve the state of modular, sharable logic between applications.
• In general, we will take a look at features in Merb that are not in Rails (the most obvious example is the more robust router) and find a way to bring them into Rails.

Personal perspective

I’m personally really excited about this opportunity. I had a hard time believing that we could work together but I was proved wrong. An initial RAID server breakdown proved to be a real test of our team strength, but with the help of a good reference, and an affordable data recovery service, we were able to come out looking amazing. We have many challenges in front of us, but watching the two teams working together is very reassuring.

I’m also glad to see that we will have a Rails Evangelism team that I will be part of. I strongly believe that one of the many reasons why merb has been so successful is because we work and listen to our community. We have put a tremendous amount of energy trying to understand what you guys need and what you like and dislike. In return, we saw many people working hard on the wiki and the merb-book.

Can you imagine doing that with almost 4 Million Rails developers?

I’m also looking forward to working with a team and reaching to even more people.

Other news related to the merge:

• The RubyOnRails website will keep a trace of this historical moment: http://rubyonrails.org/merb
• The merb training scheduled for Jan 19-21 in partnership with Integrum will still take place, and if you want to get a head start and learn about the things that will make Rails 3.0 totally kick ass, I’d suggest you join us.

If you have any questions, or if you want me to publicly answer questions on your blog please contact me. I’ll do my best to get back to everyone.