Anyways, I have been quite busy preparing courses for classes I gave to a bunch a great Engineers at one of the Fortune 100 companies based in San Diego. I was also planning my big vacation trip to Europe and wrapping up few projects.

However, during my exile overseas, I came to the conclusion that Rubyists don’t scale. Since Twitter became stable again, we don’t hear many people ranting about Rails not scaling anymore. With one of my clients’ app handling around 7 million requests/day I can tell you Ruby/Merb do scale quite well! But ruby developers don’t seem to scale for some reason.

Maybe saying that we(Rubyists) don’t scale isn’t technically correct but that’s basically what one of my client told me.

Let’s go back in time a little bit and follow my client who we will call clientX.

• ClientX has a great concept and wants to conquer the internet.
• ClientX hears that Rails is the way to go.
• ClientX hires a contractor/mercenary/freelancer/guns for hire/consultant (aka Me)

• Me builds a killer app using Merb (killing framework)

• ClientX raises loads of $$$

• ClientX wants to hire a team because Me doesn’t want to become a FTE

• ClientX and Me look for Rubyists wanting to relocate and get a decent salary

• ClientX *can’t find someone they consider good enough and who would accept their package

• Many JAVA guys are available on location and accept lower packages

• Ruby app gets ported over to JAVA
• Me sad

So is it really the Rubyists’ fault if we don’t want to relocate and only accept higher packages? Should I blame Obie for telling people to charge more and teaching how to hustle? Or should we just tell clients that it’s time to get used to working remotely?

Honestly, I don’t think any of the above explanations are valid. Ruby is the new/hot technology and very few people have the skills and experience to lead major projects. These people make a good living and enjoy their “freedom” and dream of building their own products. Most of them/us value their work environment, family and are reluctant to move.

At the same time, companies do need people locally(at least a core team) and can’t always afford the cool kids.

ClientX, quite frustrated by the whole hiring process told me once: “you Ruby folks are too unavailable and difficult to work with! We need a committed team that actually cares about the company/product.”

That hurts when you worked hard on a project and just can’t satisfy the client by finding guys willing to relocate and work for them. It gets even more painful when your code gets entirely ported over to JAVA!

But at the same time I understand ClientX’s motivation, PHP guys are cheaper, JAVA guys are more available, why in the word did we go with Ruby and are now struggling finding people?

Once again, there is positive and negative side in everything, by choosing Ruby and a “great contractor” ClientX was able to catch up with the competition and even pass them in no time. They quickly raised good money and got everything they needed to become #1. I don’t believe it would have been possible to do the same thing so quickly with JAVA for instance. However choosing a cutting edge technology means you need to look harder for talented people.

It’s too bad the code gets rewritten in a different language but at the same time, I do my best to facilitate the process and to keep a good relation with my client. There was nothing personal in the decision, it’s just too bad we were not able to keep on using the latest/coolest/awesomess technology available

To finish on a positive note, here is the solution to scale your Ruby task force provided to you by the #caboose wisdom:

Based on my conversations with other #caboosers who hire other devs, the word in the street is that you just need to get one or two great ruby guys (who will probably cost you a lot) and find a bunch of smart people to train. You’ll end up with an awesome team of scalable rubyists

My main points were that:

• you might make your code way slower if you don’t know what you are doing
• readability might drop considerably
• maintainability can become an issue

People left some very good comments about how to write the same module using metaprogramming and keep things fast.

Today Wycats pinged me about this post and told me that the issue was definemethod and that classeval is effectively the same as regular code, it gets evaluated in eval.c, just like regular Ruby code. On the other hand, defined_method has to marshall the proc.

I cleaned up my benchmarks using rbench, added some of the solutions provided to me and obtained the following results:

Here is the original/bad metaprogramming example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  module MetaTimeDSL

    {:second => 1, 
     :minute => 60, 
     :hour => 3600, 
     :day => [24,:hours], 
     :week => [7,:days], 
     :month => [30,:days], 
     :year => [364.25, :days]}.each do |meth, amount|
      define_method "#{meth}" do
        amount = amount.is_a?(Array) ? amount[0].send(amount[1]) : amount
        self * amount
      end
      alias_method "#{meth}s".intern, "#{meth}"
    end

  end
  Numeric.send :include, MetaTimeDSL

The no metaprogramming module is available there

Refactored:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

  module RefaMetaTimeDSL

  {:second => 1, 
   :minute => 60, 
   :hour => 3600, 
   :day => [24,:hours], 
   :week => [7,:days], 
   :month => [30,:days], 
   :year => [364.25, :days]}.each do |meth, amount|
    self.class_eval <<-RUBY
      def r_#{meth}
        #{amount.is_a?(Array) ? "#{amount[0]}.#{amount[1]}" : "#{amount}"}
      end
      alias_method :r_#{meth}s, :r_#{meth}
    RUBY
  end

end
Numeric.send :include, RefaMetaTimeDSL

the refactor 2 or eval based solution provided by Matt Jones which uses class_eval like the previous refactor.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

  module EvalMetaTimeDSL
    def self.included(base)
      base.class_eval do
        [ [:e_second, 1], 
          [:e_minute, 60], 
          [:e_hour, 3600], 
          [:e_day, [24,:e_hours]], 
          [:e_week, [7,:e_days]], 
          [:e_month, [30,:e_days]], 
          [:e_year, [365.25, :e_days]]].each do |meth, amount|
            amount = amount.is_a?(Array) ? amount[0].send(amount[1]) : amount
            eval "def #{meth}; self*#{amount}; end"
            alias_method "#{meth}s", meth
          end
      end
    end
  end
  Numeric.send :include, EvalMetaTimeDSL

and finally, the “better metaprogramming” version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

 module GoodMetaTimeDSL

  SECOND  = 1
  MINUTE  = SECOND * 60
  HOUR    = MINUTE * 60
  DAY     = HOUR * 24
  WEEK    = DAY * 7
  MONTH   = DAY * 30
  YEAR    = DAY * 364.25
  
  %w[SECOND MINUTE HOUR DAY WEEK MONTH YEAR].each do |const_name|
      meth = const_name.downcase
      class_eval <<-RUBY 
        def g_#{meth} 
          self * #{const_name} 
        end 
        alias g_#{meth}s g_#{meth} 
      RUBY
  end
end
Numeric.send :include, GoodMetaTimeDSL

Looking at the refactored version by Wycats, you can see he’s right and the major issue with the original version was definemethod. Using classeval does make things almost as fast and even faster than the no metaprogramming version.

Interesting enough, the benchmarks show that some methods from the meta modules are faster than the ones from the no meta module. Overall, an optimized metaprogramming can be more or else as fast as a non meta code. Of course, with the new VMs coming up, things might change a little bit depending on the language implementation.

In conclusion, metaprogramming can be as fast as no metaprogramming but that won’t help your code readability and maintainability, so make sure to only use this great trick when needed!

p.s: here is the benchmark file if you don’t believe me

• RailsConf 08. Great conference, probably my last Rails Conf though. I’ll be in Orlando for Ruby Conf 08 and I’ll focus on 1 or 2 local conferences (probably mountain west and another one).

• MerbCamp 08 in San Diego this Fall organized by SD Ruby. Details are not finalized yet but Yehuda Katz announced it during his Merb talk at RailsConf.

• Moved this blog to a new Joyent accelerator with git support and finally have the possibility to use Ambition! (planning on moving from Mephisto to Feather)

• Launched a client’s Merb app and getting around 3 million hits/day. Merb is just awesome. (more info when the client’s app gets out of beta)

• I’ll join Gregg Pollack from http://railsenvy.com/ during Qcon and take part in the Ruby for the Enterprise track. My talk will focus on Merb usage in real life.

• Renamed my github username, new repo url: http://github.com/mattetti (sorry about that)

word list.

I personally decided to use SQlite3 to store the words after parsing the text file if the database is empty.

It was a good exercise and it got me to play with SQLite and one of the Ruby adapter library. Once I was done, I decide to play with DrNic cool Gem generator.

http://random-word-gen.rubyforge.org/svn/

And the Rubyforge site

By the way, I did find an almost useful use for this gem. Activation code generator. You know, the kind of string your receive on by email or SMS to activate a feature or an account. It’s always a pain to type a MD5Hash string, while, when using the word generator, the string is made of existing words, making the process way more user friendly.

I also plan on adding some random copyleft text to the sqlite db so the Gem will be able to generate titles, paragraphs and random quotes. I’m just tired of reading lorem ipsum and on top of that, I get to it, I might had text in various languages so you check if your app breaks when using another char set, or if your layout can’t handle too much text.

Honestly, I don’t expect you to use this gem, but I jut wanted to encourage people to start writing their own gem, the process is super easy and rewarding. And actually, feel free to try the challenge and post a link to your implementation in the comments. (That’s seriously, the best way to learn)

p.s: I’m sorry about my RSS feed constantly being reset, it seems to be a problem with Mephisto, my blog engine and we are trying to figure out what’s going on.

Since I’ve started using RSpec I’ve started writing tests against my views. RSpec makes things really easy and I’ve been enjoying testing my views.

I’m not the only one having fun, check this great post from Mr Planet Argon aka Robby Russel

Recently I was working on implementing some Sexy Charts and I was using a XML builder to create an XML view of for a controller. Since I wanted to be a good Rails Ninja and obey the BDD rules, I figured I needed to test my XML view. Making sure that the nodes and the attributes were properly created. Turned out that is wasn’t too hard, there was many options but none were very well documented so I decided to write this quick tutorial.

UPDATE 31 Oct 2007: After a comment from Josh Knowles, I updated the tests to test with have_tags (built in RSpec) and hpricot.

Hpricot

hpricot is a awesome HTML parser perfect for screen scraping. But wait, there’s more to this awesome library, hpricot can also parse XML.

If you watched the excellent RSpec peepcasts you probably noticed that topfunky aka Geoffrey Grosenbach uses hpricot to test a remote API.

In our case, we’ll use hpricot to test that our generated XML follows our expectations.

XML Builder + RSpec

Let’s write a quick test to make sure our controller uses a XML builder view:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

  describe AveragesController, "handling GET /averages.xml" do

    before do
      Average.stub!(:find).and_return(@average)
    end
  
    def do_get
      @request.env["HTTP_ACCEPT"] = "application/xml"
      get :index
    end
  
    it "should render the action using the XML builder" do
      do_get
      response.should render_template('averages/index.xml.builder')
    end

  end

To make this example pass, we need to modify our rspec generated controller.

1
2
3
4
5
6
7
8

  def index
    @averages = Average.find(:all)
  
    respond_to do |format|
      format.html # index.html.erb
      format.xml  { render :action => "index.xml.builder", :layout => false }
    end
  end

(Please note that I’m using Rails 2.0 and that’s why I’m not using a .rxml view)

Here is what our XML file should end up looking like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54

  <?xml version="1.0" encoding="UTF-8"?>
  <chart>
    <series>
      <value xid="0">January</value>
      <value xid="1">February</value>
      <value xid="2">March</value>
      <value xid="3">April</value>
      <value xid="4">May</value>

      <value xid="5">June</value>
      <value xid="6">July</value>
      <value xid="7">August</value>
      <value xid="8">September</value>
      <value xid="9">October</value>
      <value xid="10">November</value>

      <value xid="11">December</value>
    </series>
    <graphs>
      <graph fill_alpha="50" color="#FF0000" fill_color="#CC0000" title="high">
        <value xid="0">65.1</value>
        <value xid="1">65.7</value>
        <value xid="2">64.9</value>

        <value xid="3">66.7</value>
        <value xid="4">67.1</value>
        <value xid="5">69.3</value>
        <value xid="6">73.0</value>
        <value xid="7">74.8</value>
        <value xid="8">75.4</value>

        <value xid="9">73.4</value>
        <value xid="10">68.9</value>
        <value xid="11">65.3</value>
      </graph>
      <graph fill_alpha="50" color="#0000CC" fill_color="#0000CC" title="low">
        <value xid="0">48.9</value>
        <value xid="1">50.7</value>

        <value xid="2">52.9</value>
        <value xid="3">55.6</value>
        <value xid="4">59.2</value>
        <value xid="5">61.9</value>
        <value xid="6">65.7</value>
        <value xid="7">67.3</value>

        <value xid="8">65.7</value>
        <value xid="9">61.0</value>
        <value xid="10">54.0</value>
        <value xid="11">48.7</value>
      </graph>
    </graphs>
  </chart>
  

Let’s write some tests to make sure our view is ok:

index.xml.builder_spec.rb

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81

require File.dirname(__FILE__) + '/../../spec_helper'
require 'hpricot'

describe "/averages/index.xml.builder" do
  include AveragesHelper
  
  before do
    average_1 = mock_model(Average)
    average_1.stub!(:month).and_return("January")
    average_1.stub!(:high).and_return("74.5")
    average_1.stub!(:low).and_return("61.5")
    average_2 = mock_model(Average)
    average_2.stub!(:month).and_return("February")
    average_2.stub!(:high).and_return("82.5")
    average_2.stub!(:low).and_return("71.5")

    assigns[:averages] = [average_1, average_2]
  end

  it "should render the months in the series" do
    render "/averages/index.xml.builder"
    response.should have_tag("value", 'January')
    response.should have_tag("value", 'February')
    # Same thing but with Hpricot
    doc = Hpricot.XML(response.body.to_s)
    (doc/:value).first.inner_html.should == 'January'
    (doc/:value)[1].inner_html.should == 'February'
  end
  
  it "should set the xid attributes for the series" do
    render "/averages/index.xml.builder"
    response.should have_tag("value[xid=0]:first-child")
    response.should have_tag("value[xid=1]:last-child")
    # Same thing but with Hpricot
    doc = Hpricot.XML(response.body.to_s)
    (doc/:value).first["xid"].should == '0'
    (doc/:value).last["xid"].should == '1'
  end
  
  it "should have 2 graphs and they should have a title" do
    render "/averages/index.xml.builder"
    response.should have_tag("graph[title=high]:first-child")
    response.should have_tag("graph[title=low]:last-child")
    # Same thing but with Hpricot
    doc = Hpricot.XML(response.body.to_s)
    (doc/:graph).size.should == 2
    (doc/:graph).first["title"].should == 'high'
    (doc/:graph).last["title"].should == 'low'
  end
  
  it "should have a color set by graph" do
    render "/averages/index.xml.builder"
    response.should have_tag("graph[color]:first-child")
    response.should have_tag("graph[color]:last-child")
    response.should have_tag("graph[fill_color]:last-child")
    response.should have_tag("graph[fill_alpha]:last-child")
    # Same thing but with Hpricot
    doc = Hpricot.XML(response.body.to_s)
    (doc/:graph).first["color"].should_not be_nil
    (doc/:graph).last["color"].should_not be_nil
    (doc/:graph).last["fill_color"].should_not be_nil
    (doc/:graph).last["fill_alpha"].should_not be_nil
  end
  
  it "should have an xid for each graph value" do
    render "/averages/index.xml.builder"
    response.should have_tag("graph > value[xid=0]:first-child")
    # Same thing but with Hpricot
    doc = Hpricot.XML(response.body.to_s)
    (doc/:graph/:value).first["xid"].should == "0"
  end
  
  it "should have the high average as values of the first graph" do
    render "/averages/index.xml.builder"
    response.should have_tag("graph > value:first-child", "74.5")
    # Same thing but with Hpricot
    doc = Hpricot.XML(response.body.to_s)
    (doc/:graph/:value).first.inner_html.should == "74.5"
  end
  
end

The first thing you must do (after installing the hpricot gem) is to require hpricot in your test:

  require 'hpricot'

Now that hpricot is created we can use it to parse the response and check against our expectations.

(we create mock objects to pass to the view so we know exactly what to expect and we separate Model/Controller/Views tests)

To check against our response we have to use hpricot parser syntax. It might look at bit funny at first, but believe me it’s really easy once you get it.

But first, let’s parse the view:

1
2
3
4

# Render the mocked up data using the xml view
render "/averages/index.xml.builder"
# Load and parse the view response body:
doc = Hpricot.XML(response.body.to_s)  

Let’s look at the first test:

1
2
3
4
5
6

  it "should render the months in the series" do
    render "/averages/index.xml.builder"
    doc = Hpricot.XML(response.body.to_s)
    (doc/:value).first.inner_html.should == 'January'
    (doc/:value)[1].inner_html.should == 'February'
  end

(doc/:value) returns all the value nodes, we take the first one and extract its content. We expect that it would match the name of the month for the first average.

Let’s now look at another test:

1
2
3
4
5
6
7

  it "should have 2 graphs and they should have a title" do
    render "/averages/index.xml.builder"
    doc = Hpricot.XML(response.body.to_s)
    (doc/:graph).size.should == 2
    (doc/:graph).first["title"].should == 'high'
    (doc/:graph).last["title"].should == 'low'
  end

The thing to look at here is the fact that we are checking on the node’s attribute “title”. Really simple syntax and clean test, isn’t it?

Finally let’s look at the last example:

1
2
3
4
5

  it "should have the high average as values of the first graph" do
    render "/averages/index.xml.builder"
    doc = Hpricot.XML(response.body.to_s)
    (doc/:graph/:value).first.inner_html.should == "74.5"
  end

We are checking that the content of the first value node nested inside a graph node is equal to 74.5 which is the high average for the first month.

In practice, you probably won’t write all these tests at once, but anyway, let’s look at our XML builder which will make all these tests pass:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23

xml.instruct!   ml, :version=>"1.0", :encoding=>"UTF-8"
xml.chart do
  xml.series do    
    @averages.each_with_index do |average, index|
      xml.value average.month,   id => index
    end
  end
  
  xml.graphs do
    xml.graph :title => 'high', :color => "#FF0000", :fill_alpha => "50", :fill_color => "#CC0000" do
      @averages.each_with_index do |average, index|
        xml.value average.high,   id => index
      end
    end
    
    xml.graph :title => 'low', :color => "#0000CC", :fill_alpha => "50", :fill_color => "#0000CC" do
      @averages.each_with_index do |average, index|
        xml.value average.low,   id => index
      end
    end
  end
  
end

Hpricot is a really nice tool which can make your BDD life much easier. And even if you don’t do BDD/TDD yet, it’s a great way to verify that any XML data you receive/generate is valid.

Happy testing

As many of you know, I’m a great RSpec and autotest fan.

Last night, I upgraded my MacBook to from Tiger to Leopard and I everything went very well… apart that Autotest stopped notifying me using Growl when my tests pass or fail

Growl itself works fine, even when using spaces.

However growlNotify, the Growl plugin used by autotest doesn’t seem to be Leopard compatible yet.

Just thought you guys should know that before upgrading.