My first thought was to make sure we had  an optimized ActiveSupport’s xmlmini backend. Rails 2.3.5 fixed some issues when using Nokogiri as a xmlmini so I switched to my favorite Ruby XML lib:

ActiveSupport::XmlMini.backend = 'Nokogiri'

I run some benchmarks using ab, httperf and jmeter but the results were not that great. I was so sure that switching from rexml to nokogiri would give me awesome results that I was very disappointed.

I was about to call Aaron Patterson (Nokogiri’s author) to insult him, blame him for _why’s disappearance and tell him that all his pro bono efforts were useless since my own app was not running much faster when switched to his library. As I was about to dial his number on my iPhone I had a crazy thought… maybe it was not Aaron’s fault, maybe it was mine.

So I took a break went to play some fuzzball and as I was being ridiculed by Italian coworker, Emanuele, I realized that most of our API calls were just simple HTTP requests with no XML payload, just some query params. However, we were generating a lot of XML to send back to the client and AS::XmlMini only takes care of the XML parsing, not the rendering.

The XML rendering is done by Jim Weirich‘s pure Ruby builder library which is vendored in Rails. Builder does a good job, but I thought that maybe a C based library might improve the speed. A coworker of mine (James Bunch) recommended to look into faster-builder, a drop-in Builder replacement based on libxml. Unfortunately, the project doesn’t seem to be maintained and I decided to look into using Nokogiri XML builder instead. (Also, faster-builder’s author doesn’t like me very much while Aaron knows he’s one of my Ruby heroes so asking for help could be easier)

Some people reported having tried using Nokogiri as a XML builder but didn’t see much speed improvement. Because of the amount of magic required to render a rxml template, I was not really surprised but I decided to contact Aaron and ask him if he tried using his lib instead of builder in a Rails app. Aaron told me he gave it a try a while back and he helped me get my Rails app setup to render xml templates using Nokogiri.

The next step was simple, create a benchmark app and benchmark Builder vs Nokogiri using various templates. Here are the results I got using Ruby 1.9.1 (the Ruby version we use in production) and two sets of templates:

Builder small template, time per request: 15.507 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/builder_small
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/builder_small
Document Length:        216 bytes

Concurrency Level:      1
Time taken for tests:   3.101 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      114326 bytes
HTML transferred:       43200 bytes
Requests per second:    64.49 [#/sec] (mean)
Time per request:       15.507 [ms] (mean)
Time per request:       15.507 [ms] (mean, across all concurrent requests)
Transfer rate:          36.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    11   15   8.8     12      47
Waiting:        3   15   8.9     12      47
Total:         11   15   8.8     12      47

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     12
  75%     13
  80%     13
  90%     35
  95%     36
  98%     38
  99%     41
 100%     47 (longest request)

Nokogiri small template, time per request: 15.354 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/noko_small
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/noko_small
Document Length:        238 bytes

Concurrency Level:      1
Time taken for tests:   3.071 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      118717 bytes
HTML transferred:       47600 bytes
Requests per second:    65.13 [#/sec] (mean)
Time per request:       15.354 [ms] (mean)
Time per request:       15.354 [ms] (mean, across all concurrent requests)
Transfer rate:          37.75 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    11   15   8.6     12      39
Waiting:       11   15   8.6     12      39
Total:         11   15   8.6     12      39

Percentage of the requests served within a certain time (ms)
  50%     12
  66%     12
  75%     12
  80%     13
  90%     35
  95%     36
  98%     37
  99%     38
 100%     39 (longest request)

Running the benchmarks many times showed that Nokogiri and Builder were taking more or less the same amount of time to builder a small template.

I then decided to try a bigger template, closer to what we have in production, here are the results:

Nokogiri longer template, time per request: 31.252 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/noko
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/noko
Document Length:        54398 bytes

Concurrency Level:      1
Time taken for tests:   6.250 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      10951200 bytes
HTML transferred:       10879600 bytes
Requests per second:    32.00 [#/sec] (mean)
Time per request:       31.252 [ms] (mean)
Time per request:       31.252 [ms] (mean, across all concurrent requests)
Transfer rate:          1711.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.0      0       0
Processing:    24   31  11.3     26      62
Waiting:       23   30  11.3     24      61
Total:         24   31  11.3     26      62

Percentage of the requests served within a certain time (ms)
  50%     26
  66%     27
  75%     27
  80%     29
  90%     54
  95%     55
  98%     58
  99%     59
 100%     62 (longest request)

Builder, longer template, Time per request: 140.725 [ms] (mean)

$ ab -c 1 -n 200 http://127.0.0.1:3000/benchmarks/builder
This is ApacheBench, Version 2.3 <$Revision: 655654 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 100 requests
Completed 200 requests
Finished 200 requests

Server Software:        nginx/0.7.65
Server Hostname:        127.0.0.1
Server Port:            3000

Document Path:          /benchmarks/builder
Document Length:        54376 bytes

Concurrency Level:      1
Time taken for tests:   28.145 seconds
Complete requests:      200
Failed requests:        0
Write errors:           0
Total transferred:      10947000 bytes
HTML transferred:       10875200 bytes
Requests per second:    7.11 [#/sec] (mean)
Time per request:       140.725 [ms] (mean)
Time per request:       140.725 [ms] (mean, across all concurrent requests)
Transfer rate:          379.83 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    0   0.1      0       1
Processing:   127  141  24.6    132     331
Waiting:      126  139  23.6    130     328
Total:        127  141  24.6    132     331

Percentage of the requests served within a certain time (ms)
  50%    132
  66%    138
  75%    147
  80%    149
  90%    156
  95%    169
  98%    193
  99%    311
 100%    331 (longest request)

Wow, @tenderlove‘s Nokogori just brought us a 4.5X speed improvement for this specific template.  100ms per request is probably not a big deal for most people and I have to say that Jim did a great job with Builder. However in my specific case, 100ms on a request being called thousands of times per hour is quite important.

(The benchmark app is available on github, feel free to fork it and benchmark your own templates)

Who would have thought that a man like this could save the day?!

Aaron 'Tenderlove' Patterson

The moral of the story is that adding a bit of tenderlove to your Ruby code can make it perform much much better!

Thank you Aaron ‘Tenderlove’ Patterson!

Last time, in our ‘do it in less than 5 minutes’ series, we saw how to add quickly and simply add Ajax pagination.

This time we’ll see how to add some sexy/fancy charts to your rails app.

The goal is to end up with something like:

Various options

You might have heard or even tried solution such as Gruff or JFreeChart.

While these solutions are great, they are certainly a pain in the butt. Gruff requires RMagick (avoid RMagick as much as can) and creates static files (a real pain when your graphs change all the time) JFreeChart on the other hand requires Java, Java skills and I hate the way you create graphs:

1
2
3
4
5

  def CreateChart
         pipe = IO.popen "java -cp C:\\InstantRails\\rails_apps\\project\\jfree\\src;C:\\InstantRails\\rails_apps\\project\\jfree\\lib\\jcommon-1.0.0-rc1.jar;C:\\InstantRails\\rails_apps\\project\\jfree\\lib\\jfreechart-1.0.0-rc1.jar; CreateChart" 
         pipe.close
         redirect_to "/graph/report" 
      end

Anyway, none of these solutions would let us create our charts in less than 5 minutes so let’s cut the story short. The best solution IMHO is to use Flash. But wait, you don’t need to know ActionScript or to own a license of Flash or Flex, we have libraries available for us to use without any Flash knowledge

XML/SWF is cool Flash library which should fulfill our needs, you can even find a rails plugin to make things easier.

amCharts

But, to be honest I’d like to have something a bit “cleaner/sexy/fancy” and easier to setup. So we’re going to use amCharts Don’t get me wrong, XML/SWF is a great library and you can make your graphs look nice (but you have to pay for support).
Since we are running out of time let’s see how to implement a nice graph using *my* favorite library.

[DISCLAIMER: amCharts is NOT open source and NOT free. But, it's cheap (85 euros per site) especially when you think of how much time you will save. AND there is a FREE version. The Free version is the same as the full version but with a link back to amcharts.com]

Setup

Let’s go ahead and download one of the package: http://www.amcharts.com/column/download/ for instance.

Unpack the files and put them in their own folder in your public folder.
Make sure you have the .swf file (amcolumn.swf for instance), a XML settings file and the fonts folder.
(You might want to also create an empty amcharts_key.txt in the same folder since the plugin tries to load the key and you don’t want to pollute your logs.)

Usage

Now you need to understand how amCharts works.

After being loaded, amCharts expects a datastream. The datastream is then parsed and displayed as a chart.
You can modify the aspect of any chart by changing its settings.
Settings are set at runtime and/or in a setting file.

Great! I won’t cover the settings file. It’s a well documented XML file you just copied in your public folder. (or check the documentation)

What we want to focus on, is the datastream. Basically we just need to create a XML file that can be parsed by amCharts.

Let’s imagine that we have a reports_controller.rb file We want to display the population of the cities in California.

let’s add a new action to render our XML file:

1
2
3
4
5
6
7
8

  def population
    @cities = City.find(:all)
    @population_data_link = formatted_population_reports_url(  ml)
    respond_to do |format|
      format.html
      format.xml  { render :action => "population.xml.builder", :layout => false }
    end
  end

(notice that I’m using rails 2.0 and that’s why my XML template is not RXML)

As you can see, we have 2 values: @cities and @populationdatalink

@cities contains all the City records, including their population etc..

@populationdatalink contains the url to retrieve the datastream.

If you wonder how I got this url? I’m simply using a named route defined in my routes.rb:

  map.resources :reports, :collection => {:population => :get}

(note that you don’t need to create a restful route for that, a simple named route would have worked too)

Flash detection

Since we are going to use Flash, we want to make sure that people have the Flash plugin installed on their browser. For that we will use swfobject. Simply make sure to add swfobject.js (available in any amChart package) to your public/javascript folder. Then make sure you linked the javascript in your header:

  <%= javascript_include_tag 'swfobject' %>

We now need to create our 2 views: population.html.erb and population.xml.builder

population.html.erb

Basically, this view only loads amCharts and provides it with the details of the datastream:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

  <div id="population_chart" class='chart'>
    <strong>Text displayed when the user doesn't have Flash. You might want to display a simple table with the population, search engines and visitor without flash would love that.</strong>
    <p> To see this page properly, you need to upgrade your Flash Player, please visit the Adobe web site</p>
  </div>

  <script type="text/javascript">
    // <![CDATA[    
    var so = new SWFObject("/amcolumn/amcolumn.swf", "population_chart", "800", "380", "8", "#000000");
    so.addVariable("path", "/amcolumn/");
    so.addVariable("settings_file", escape("/amcolumn/column_settings.xml"));
    so.addVariable("data_file", escape("<%= @population_data_link %>"));
    so.addVariable("additional_chart_settings", "<settings><labels><label><x>250</x><y>25</y><text_size>18</text_size><text><![CDATA[<b>California Population as of <%= Time.now.to_s(:db) %></b>]]></text></label></labels></settings>");
    so.addVariable("preloader_color", "#000000");
    so.write("population_chart");
    // ]]>
  </script>

As you can see, we have a div called population_chart. This div is replaced at load time by the Flash object if the visitor has Flash setup locally. Think about providing some data in case the user doesn’t have Flash.

The rest is simple Javascript. I unpacked the amchart column lib in mypublic/amcolumn folder and that’s why I setup the path as “amcolumn”

  so.addVariable("path", "/amcolumn/");

My settings file is called column_settings.xml :

  so.addVariable("settings_file", escape("/amcolumn/column_settings.xml"));

and the most important part:

  so.addVariable("data_file", escape("<%= @population_data_link %>"));

Finally, I added some dynamic settings just to show you how easy it is:

1
2

  so.addVariable("additional_chart_settings",
  "<settings><labels><label><x>250</x><y>25</y><text_size>18</text_size><text><![CDATA[<b>California Population as of <%= Time.now.to_s(:db) %></b>]]></text></label></labels></settings>");

Ok, let’s now create our XML view:

population.xml.builder

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

  xml.instruct!   ml, :version=>"1.0", :encoding=>"UTF-8"
  xml.chart do
    # xml.message "You can broadcast any message to chart from data XML file", :bg_color => "#FFFFFF", :text_color => "#000000"
    xml.series do    
      @cities.each_with_index do |city, index|
        xml.value city.name,   id => index
      end
    end

    xml.graphs do
     #the gid is used in the settings file to set different settings just for this graph
      xml.graph :gid => 'population' do
        @cities.each_with_index do |city, index|
          population = city.population
          case population
            # When the population is > 1 million, show the bar in red/pink
            when > 100000
              xml.value value,   id => index, :color => "#ff43a8", :gradient_fill_colors => "#960040,#ff43a8", :description => level
            else
              xml.value value,   id => index, :color => "#00C3C6", :gradient_fill_colors => "#009c9d,#00C3C6", :description => level
            end
        end
      end
    end

  end

Nothing fancy, we first created a series with all the city names:

1
2
3
4
5

  xml.series do    
    @cities.each_with_index do |city, index|
      xml.value city.name,   id => index
    end
  end

Then we created another node with the values for each city.
Since it would be cool to display some bars in a different color, we used a case-switch statement:

1
2
3
4
5
6
7
8
9
10
11
12

    xml.graph :gid => 'population' do
      @cities.each_with_index do |city, index|
        population = city.population
        case population
          # When the population is > 1 million, show the bar in red/pink
          when > 100000
            xml.value value,   id => index, :color => "#ff43a8", :gradient_fill_colors => "#960040,#ff43a8", :description => level
          else
            xml.value value,   id => index, :color => "#00C3C6", :gradient_fill_colors => "#009c9d,#00C3C6", :description => level
          end
      end
    end

Depending on what you want to display, you might need to have different colors or a different tooltip text, or load an animation or image… and as you can see, it’s REALLY easy.

Got to http://yoursite.com/reports/population to enjoy your new fancy graph.

That’s it, you are done!

Time to tweak your settings file to make your graph look awesome.
Since you now have a lot of free time, you can start re-factoring your code and make sure you have a good test coverage.

Good luck!

The thing was that this time I had to interface with a model using attachmentfu. I’m a great fan of afu and it’s definitely the best way of dealing with uploads.

My model looked more or less like that:

class Photo < ActiveRecord::Base

belongs_to :user

  has_attachment(
    :content_type => :image,
    :storage => :file_system,
    :max_size => 10.megabytes,
    :resize_to => '640x480>',
    :thumbnails => { :thumb => '100x100>',
                              :preview => '300x200>',
                     }
  )
  validates_as_attachment
  # read more about validates_existence_of http://blog.hasmanythrough.com/2007/7/14/validate-your-existence
  validates_existence_of :user
end

My show action in my photo controller could have looked a bit like that:

respond_to do |format|
  format.html # show.html.erb
  format.xml  { render  ml => @photo }
end

That’s great, the problem is that we are displaying a lot of information that our Flash client doesn’t need to see, actually we are exposing a lot of information nobody should ever see and we are not displaying what we should. On top of being a waste of bandwidth and giving too much work to the client, we are not actually providing the user with the details of the thumbnail.

The first thing to do would be not to display some of the object attributes, the to_xml method lets you do that.

Note that in Edge, Rails will automatically try to convert your object using toxml, you don’t even need to mention it. However in our case, we want to use some _advanced features offered by to_xml, and here is how our code should look like:

format.xml do
  render  ml => @photo.to_xml( :except => [ :id, :filename, :updated_at, :user_id, :height, :width], :skip_types => true )
end

What I just did is very simple, we rendered our object as an xml object but we didn’t convert few attributes, :id, :filename, :updatedat, :userid, :height, :width. By default Rails also adds the object type, we don’t really need that right now, so let’s skip them.

(The reason why I don’t want to convert the filename is that I want to provide our Flash client with the photo thumbnail instead of the original picture.)

As far as I know, to_xml doesn’t let you create new attributes. (if I have some time, I’ll submit a patch to get that added).

What we are trying to do is to display the avatar of a user. We found the photo record using @user.photo but that’s the original photo and we want to provide Flash with the avatar info, not the original.

What we need to do is to simply add a new attribute called avatar:

format.xml do
  @photo[:avatar] = @photo.public_filename(:thumb)
  render  ml => @photo.to_xml( :except => [ :id, :filename, :updated_at, :user_id, :height, :width], :skip_types => true )
end

Simple enough, but it took me a little while to figure it out

Voila, we now have a clean, trimmed and safe XML returned object that you can be consumed by our Flash client. Ohh, and we added a new attribute that the original object didn’t have

However Yves aka Kadoudal was wondering what I did with Rails to return the event record:

Rails.get(‘events’, rails_events); how rails returns the event record as we don’t call a controller/action … ? I believe doing it RESTFul it’s depending upon your route ? is it not?

We are actually calling a controller/action If you look at the Restfulflash class, you’ll notice a function called get

public function get(controller, callback){
        var railsReply:XML = new XML();
        railsReply.ignoreWhite = true;
        railsReply.onLoad = function(success:Boolean){
            if (success) {
                    trace ('Rails responded: '+railsReply);
                    callback.text = railsReply;
            } else {
                    trace ('Error while waiting for Rails to reply');
               callback.text = 'error';
            }
        }
        var railsRequest:XML = new XML();
        railsRequest.contentType='application/xml';
        railsRequest.sendAndLoad(this.gateway+controller, railsReply);
        delete railsRequest;
    }

What’s really interesting are the following 2 lines:

railsRequest.contentType=”application/xml”;
railsRequest.sendAndLoad(this.gateway+controller, railsReply);

I’m preparing a request that I will send to my gateway mentioning the controller name. In the previous example I was calling the ‘events’ controller: http://mysite.fr/events because I’m using REST and because my request is a ‘get’ Rails will call the index action.

for more info on RESTful design check the nice series available from the softies on rails blog

Let’s just look at my code and see what’s up with rails magic

Here is my index action sending you back a different object based on the header of your request.

class EventsController < ApplicationController
  # GET /events
  # GET /events.xml
  def index
    @events = Event.find(:all)

    respond_to do |format|
      format.html # index.rhtml
      format.xml  { render  ml => @events.to_xml }
      format.json { render :json => @events.to_json }
    end
  end

That’s it, and it was automatically created for you by the script/generate scaffold_resource command

Since I’m answering Yves’ questions, let’s look at his final question:

problem : what if the xml returned is not a record, but it has to be prepared by Rails…
I explain, right now my rails view .. I have a javascript object passing all the info via JS
I would like to replace the datasource file by a direct call to rails from Flash, so this param will disappear… so (correct me if I am wrong)
1- I need to pass a user_id value in JS to Flash in the script…
2- Flash need to send a request to Rails (controller/action/id to prepare the xml… ) to get in return a correct xlm object to be displayed

1- I’m not sure why you need to pass the user_id from JS to Flash but I guess Flash doesn’t know what user to query?? (anyway that would work)

2- If you are using REST, Flash just needs make a get call to /controller/id Make sure to set the request header type as xml, like I did in my function railsRequest.contentType=”application/xml”; Otherwise I believe (but didn’t try) that you can call /events/1.xml where 1 id the id of the event you want to retrieve.

By the way, it will call the show action from the events controller which looks like that:

def show
  @event = Event.find(params[:id])

  respond_to do |format|
    format.html # show.rhtml
    format.xml  { render  ml => @event.to_xml }
  end
end