The dreaded "wedged Mongrel" - your app server stuck on one request, with others piling up, waiting infinitely for it to come free - is a problem all production Rails apps face sooner or later. The solution most commonly used is to restart the app servers frequently, via something like Monit, or just on a cron job.

But such solutions are just a band-aids which hide the real problem, which is that your code is getting stuck in an infinite loop, or waiting on an IO request which never returns. A better solution is to wrap all your actions in a timeout:

class ApplicationController < ActionController::Base
  around_filter :timeout

  def timeout
    require 'timeout'
    Timeout.timeout(30) do
      yield
    end
  end
end

This prevents the wedged app server. And combined with an exception notifier, you'll be able to see which requests are getting wedged, so that you can fix your code. (Periodic app server restarts are still needed to combat memory leakage - another problem entirely.)

I'm surprised that request timeouts aren't a standard part of web frameworks like Rails, application servers like Mongrel, or both. (If you've seen the "timeout" parameter for Thin or Mongrel, don't be fooled - it's not that kind of timeout.) After all, web requests aren't supposed to be long-lasting. Nginx or Apache will time out the request after 90 seconds or so anyway, but this doesn't stop your app server from grinding away infinitely on the request.

But there's a catch with Timeout. It uses Ruby threads, which only works as long as it's Ruby code that's getting stuck or taking too long. The second case - a system call that's getting stuck - is often the problem. So this will time out:

Timeout.timeout(3) do
  sleep 4
  puts 'done'
end

...but this will not:

Timeout.timeout(3) do
  system 'sleep 4'
  puts 'done'
end

Good unix jockeys know that SIGALRM is the correct solution here. Back in my MUD days I encountered this technique in the CircleMUD server: it would detect infinite loops and abort with a log message, allowing the game to continue running. "Wow," I said the first time I saw it in action. "How does it know?" That's the magic of SIGALRM.

Philippe Hanrigou and David Vollbracht have implemented a SIGALRM solution for Ruby in the form of SystemTimer. (They also give a great description of green threads and why they don't play well with the underlying OS.) This is a nearly drop-in replacement for Timeout. Try it:

SystemTimer.timeout(3) do
  system 'sleep 4'
  puts 'done'
end

Woot! So now, your final solution for preventing wedged app servers in production:

class ApplicationController
  around_filter :timeout

  def timeout
    require 'system_timer'
    SystemTimer.timeout(30) do
      yield
    end
  end
end

Places you'll find me at Railsconf this year:

• Giving my talk about custom Nginx modules on Saturday afternoon. The talk has evolved quite a bit since I wrote the description, so expect some broader topics, like why I think HTTP is the critical enabling protocol in the era of Rails and cloud computing.
• Attending the Heroku product talk, in which we propose why you may never need to think about servers or hosting again. This is inconveniently scheduled immediately before my session talk, so I'll have to duck out partway through.
• Signing books along with the other recipe contributors to Mike Clark's Advanced Rails Recipes at the Powell's Books booth at the lunch break on Friday.
• Hanging around our booth, where I intend to hack on Heroku and my open source projects, listen in on Geoffrey Grosenbach's podcast interviews, and meet everyone that stops by. So... stop by! :)

Advanced Rails Recipes by Mike Clark is now shipping. I'm pleased to have contributed to the chapter on nested resources.

Most of the staple Rails books - like Agile Web Development and the original Rails Recipes - are a bit out of date now, so it's nice to see a new one. Advanced Rails Recipes is full of useful goodies, some of which are what I consider basics (authentication, restful resources, foreign keys) but quite a lot of which live up to the "advanced" label (dtrace, custom rspec matchers, process monitoring). Good stuff.

After Dreamhost's post that shared hosting with Rails is too difficult to implement, and DHH stated that he (mostly) agrees, Peter Cooper of RubyInside suggested that we need a mod_ruby. Now I'm going to chime in with a few thoughts of my own.

First, shared hosting of Rails apps is entirely possible with the tools that exist today. The proof: Heroku is doing it. Of course we're not much like a traditional shared host, and in fact we try to avoid calling ourselves "hosting" at all. But hosting is certainly a substantial part of the service we've created, and in terms of implementation, our free accounts are set up in what is basically a shared hosting environment.

How have we managed to accomplish this, especially given that Heroku as a product is less than six months old? First, there's the matter of focus. We're hosting Ruby, and Ruby on Rails, only. Being focused allows us to tailor the infrastructure to that setup exactly.

Second, Heroku is opinionated software. Even just saying "Ruby on Rails" only tells you two components of the stack. RoR can run on different operating systems, different databases, and be fronted by different webservers. When you put your app on Heroku, you don't get a choice of operating system, webserver, or database. Instead, we make those choices to create a unified stack that we can verify works together flawlessly. The idea is that individual app developers need not waste time making these decisions, and instead can dive straight in to the part that matters: coding their app.

You could compare this approach to an Apple computer vs. a PC built from OEM parts. An Apple has all the parts preselected and verified to work together, from the network chipset right up to the operating system and all the software that comes with it. Building a PC from parts, one has many choices to make, and sometimes the choices will be incompatible with each other. For DIY-types, building your own system can be a lot of fun, or let you build unusual configurations to fit specific needs, like creating you own MythTV box. Heck, I've built dozens of PCs, mostly servers, over the years. But if you're not looking for DIY fun, and you just want a computer that works as quickly and easily as possible, the Apple approach is clearly the better one.

Finally, I'd like to say that mod_ruby is probably not the right path for the Ruby world. It did work extremely well for Perl and PHP, true; but Ruby frameworks are different. (Mainly because, well, they are frameworks, not just a language. mod_ruby would probably be fine if people wanted to write raw Ruby scripts to generate HTML directly.) Perhaps mod_sinatra or mod_camping would be more appropriate. Still, it seems to me that Apache is probably near the end of its lifecycle, so investing time and energy in developing modules for it does not seem terribly worthwhile to me.

Perhaps the message here is not that we actually need a mod_ruby, but rather that we need a component that solves the same problem that mod_perl or mod_php did: drawing together several components in the toolchain to make the infrastructure easier to deploy. My personal feeling here is that something that makes use of the growing availability of virtual hosting is a better choice than trying to pursue the shared hosting model so popular in the era of PHP. Smarticus suggests that Rails users favor VPS hosts for deployment, and I wholeheartedly agree with that. But further, perhaps the equivalent of mod_php for the Ruby world is going to be some sort of tool that makes it easy to stamp out prebuilt VPS instances. EC2onRails seems to be one possibility (I haven't used it myself). I'd suggest to others looking to solve this problem that VPS hosting is going to be the key to creating a mod_ruby-alike solution for the Ruby world.

For a while, everything that we did in Heroku to extend the functionality of Rails, or interoperate with it, has been done through extension mechanisms. Monkeypatching via plugins, use of the somewhat obscure Mongrel GemPlugin, and tweaking of the user's Rails app files directly. (We try to avoid that last one whenever possible. Early on we did a lot of it, but more recently we've managed to avoid it almost entirely, much to my relief.)

All of this is quite a testament to the extensibility of Rails, Mongrel, Nginx, etc. I think it's safe to say that we're bending these tools in ways that go well outside the common use cases. But even the most supple reed can only bend so far. Certain areas (for example, script/generate) can't be monkey patched through standard mechanisms. And so earlier this week I decided it was time to fork Rails.

Since I'm now a fan of Git, this turned out to be a good way to maintain our fork. Steve or Pablo will tell you how. Maintaining a parallel branch has, so far, proven to be quite easy - even fun.

Packaging up the modified version for use is just a matter of running "rake package" in each module's subdirectory (i.e. activerecord, activesupport, railties, etc). The resulting gem is dropped into pkg/ under each module. The gem can then be copied to each user's app server as it boots and installed with gem install, which overwrites the standard gem if it already exists. (I'm still trying to figure out a way to run the package script without building the documentation, which takes ages. I ended up commenting out the body of the generate_rails_framework_doc task as a stopgap.)

One thing I still haven't decided on is how to maintain multiple versions. I wish that there was a syntax for environment.rb that looked something like:

RAILS_GEM_VERSION = '2.0.*'

Personally, I rarely care which minor rev I'm running - the latest version available on whatever box the app is running on is usually just fine. For Heroku apps, they should definitely use whatever the latest minor rev is. But note that setting one catch-all with an environment variable isn't adequate, because each app should be configured to use a particular major rev, and we need to respect that. Hopefully I'll come up with something better on this eventually.

Nested resources were introduced in Rails 1.2 and are touted as the Right Way to do REST with parent-child model associations. If your app has a url that reads something like /employees?company_id=1, a switch to nested resources would cause it to read /companies/1/employees.

Rails 2 introduced a few subtle but important syntax changes. So far I haven't seen any comprehensive guide to the new syntax, so I'm writing one.

I'll use the example that seems to be popular, which is tickets belonging to events:

class Event < ActiveRecord::Base
  has_many :tickets
end

class Ticket < ActiveRecord::Base
  belongs_to :event
end

Full code for the example app is available for browsing or download via svn. Most of the relevant code is in routes.rb, tickets_controller.rb, and the tickets views.

Routes

The first step (and the easiest one) is setting up the named route with :has_many syntax. (Note that this is a significant change from Rails 1.2 syntax of passing a block to map.resources.)

map.resources :events, :has_many => :tickets

Should there be a separate line reading map.resources :tickets? That depends on whether you want the tickets to be accessible in a non-nested form (e.g. //1). Given that a ticket will always have an event, I think it's more consistent not map tickets separately. The nesting information from the url isn't needed for some operations (show, edit, and destroy). But then you'd need to have fine-grained exceptions to the before_filter, deciding when to pull the event from the url and when not to. I don't think there's any consensus on this point yet, but for this post I'm going to use the always-nested approach.

Route Helpers

The next part I've found to be the hardest - or at least, rather time-consuming. It's somewhat difficult to remember the right syntax, since there are a dozen or so helpers to generate all the urls that are needed. Thankfully there's a rake task to show you all the named routes: rake routes. The output looks like this:

events GET    /events                                    {:controller=>"events", :action=>"index"}
           formatted_events GET    /events.:format                            {:controller=>"events", :action=>"index"}
                            POST   /events                                    {:controller=>"events", :action=>"create"}
                            POST   /events.:format                            {:controller=>"events", :action=>"create"}
                  new_event GET    /events/new                                {:controller=>"events", :action=>"new"}
        formatted_new_event GET    /events/new.:format                        {:controller=>"events", :action=>"new"}
...

Blood started squirted out of my eyes the first time I ran this task on an app with nested resources - it's not pretty if your terminal isn't wide enough to prevent the lines from wrapping. I suggest maximizing your window to prevent getting blood all over your keyboard.

The bit we're looking for is in the far lefthand column - the name of the route. For our nested resource, here's the interesting ones:

event_tickets      GET /events/:event_id/tickets
new_event_ticket   GET /events/:event_id/tickets/new
edit_event_ticket  GET /events/:event_id/tickets/:id/edit
event_ticket       GET /events/:event_id/tickets/:id

The naming scheme is: parent resource (singular), then child resource (plural). So where you might have used tickets_path before, you now use event_tickets_path. new_ticket_path becomes new_event_ticket, and so on.

Seem simple? Not so fast. You also need to include the event as a parameter. So tickets_path becomes event_tickets_path(event). In cases where the child resource already knows its parent, such as an edit link, you can use edit_event_ticket_path(ticket.event, ticket). (This last bit wouldn't be necessary if you chose to map.resource :tickets, in which case editticketpath(ticket) would still work. The downside is that then you have to remember when you need to use nested helpers and when you don't. As mentioned above, I prefer going the route of consistency - always nested.)

Forms

What else needs to change? form_for has this syntax with resources:

<% form_for(@event) do |f| %>

Which is wonderfully succinct compared to the way that non-resource form_fors usually look. But it gets a little funky with nested resources:

<% form_for([ @event, @ticket ]) do |f| %>

(Trevor Squires makes a good argument to why this syntax isn't too spiffy, and Codafoo makes a slightly less compelling argument as to why it is.)

Redirects

Redirects and XML locations should also use the form_for argument syntax:

if @ticket.save
    flash[:notice] = 'Ticket was successfully created.'
    format.html { redirect_to([ @event, @ticket ]) }
    format.xml  { render :xml => @ticket, :status => :created, :location => [ @event, @ticket ] }

In all cases, the parent resource always goes first - same as the url helper, i.e. event_ticket_path.

Before Filter

Since the controller is never called without nesting, the before_filter is simple:

before_filter :get_event

  def get_event
    @event = Event.find(params[:event_id])
  end

Thus, every action and view can always count on @event being set. In some cases you can access @ticket.event, but in the always-nested approach, @event can be used everywhere.

Scoping

Any place the controller makes an ActiveRecord call like find or new should be scoped:

def index
      @tickets = @event.tickets.find(:all)

   def show
      @ticket = @event.tickets.find(params[:id])

   def new
      @ticket = @event.tickets.new

One trick for making sure you've changed every reference is to search the file for the class name. The text "Ticket" should not appear in tickets_controller.rb, except on the first line as part of the controller name.

Conclusion

Getting this all set up is quite a bit of busywork if you start with two models generated with resource scaffolding. (Which reminds me: scaffold_resource from Rails 1.2 is gone, replaced by scaffold in Rails 2. The original scaffold is gone, which is good, because last I checked it had suffered some serious bitrot.)

It would be immensely convenient if there were a generator for this. Something like:

generate nested_resource Ticket belongs_to:event

However, this would be quite a bit more difficult to write than a typical generator, because it would need to modify existing code beyond just adding lines to a file. So although handy, don't count on seeing this anytime soon. Though if some enterprising soul wanted to put their mind to it, I'm sure the Rails community would be forever, or at least briefly, grateful.

The Rails 2 upgrade has been surprisingly painless on the half dozen or so production apps I've upgraded. One problem seems to be that not all the mirrors are updated, so you get "no such gem" and have to repeat the command a few times until it finds a mirror that has it. But the other one is that for some reason, the Rails 2.0.1 package doesn't find the newly-added activeresource dependency. So the full upgrade becomes:

   1  gem update --no-rdoc --no-ri -y update
   2  gem install --no-rdoc --no-ri -y activeresource

Note 1: I like to skip rdoc and ri since they take a lot of time to generate, and I never find myself using local docs. Especially on a server, which is where most of my upgrades were being executed.

Note 2: You can thankfully skip the -y option now if you're using the latest RubyGems, 0.9.5.

The activeresource dependency problem seems to happen everywhere - my Macbook, my Ubuntu workstation, and a handful of Debian (etch) servers I've updated. I'm surprised no one else has mentioned it.

When overloading an ActiveRecord setter, I've often done this:

   1  def name=(new_name)
   2    @old_name = name
   3    attributes['name'] = new_name
   4  end

However, this doesn't always seem to work (changes to attributes don't seem to stick). Recently I've started doing this:

   1  def name=(new_name)
   2    @old_name = name
   3    super
   4  end

Which seems cleaner anyway. I can't seem to find anyone talking about the correct convention - anyone know the right answer here?