rush 0.4 released. Some changes:

• Rush::Box#processes returns a ProcessSet, for syntax like this:

processes.filter(:cmdline => /mongrel_rails/).kill

• Daemonize shell commands: bash 'some_daemon', :background => true

• Pass args to rush on the command line to execute a one-off, like this:

$ rush 'processes.filter(:command => "ruby")'

Details and rdocs.

I've posted my slides from my talk rush, the Ruby shell and Unix Integration library, which I presented today at the Silicon Valley Ruby Conference.

You can read the full details, but here's a tantalizing sample:

box.bash 'mongrel_rails start', :user => 'www', :env => { :RAILS_ENV => 'production' }

file.access = { :user_can => :read_and_write, :group_and_other_can => :read }

Backticks are just so 2007.

New features include:

• an exceptions framework (communicated across the wire - an exception on the remote box raises the same exception locally)
• Box#bash to execute arbitrary bash command (returns stdout, raises an exception with stderr if shell command returned failure)
• FindBy extensions for processes.findby_pid(123) and processes.findallby_cmdline(/mongrelrails/)
• Process#parent and Process#children
• Process#kill doesn't return until the process is really dead (courtesy of the god code)
• Improved tab completion

Read the full announcement for details.

Programmers generally agree that working iteratively is a good idea. But sometimes, we'll say: you just can't. This particular problem has to be done in one big bite; there's no way to break it down into smaller pieces; we just have to take the plunge.

Poppycock, I say. (Or perhaps something stronger.)

I've begun the process of using rush in Heroku's infrastructure. But this is tough: it's an unproven and immature library, barely a month old. Worse, most of the things we want it for - system-level calls - are of critical importance. Misplaced files are not forgiveable the way that, say, a UI glitch might be.

I started by selecting the absolute least important bit of code it could be used for. This component is something that happens very infrequently: rsyncing config files when launching a particular kind of instance. Further, I launch these manually (unlike some other instance types which self-scale), so there will always be someone watching the logs when it boots up.

But I took it even a step further. I kept the old method around and, in the case of exception on the newly rushified method, call it as a fallback. Here's the code:

def sync_nginx_conf
    sync_nginx_conf_via_rush
  rescue Exception => e
    Log.error "Exception on sync_nginx_conf via rush, falling back to rsync: #{e.summary}", :addendum => e.full_display
    sync_nginx_conf_via_rsync
  end

This is not foolproof; something in the rush sync could write an incorrect file without throwing an exception, for example. I'll keeping a very close eye on this method when it runs in production. Once this has proven itself in action, I'll feel confident in taking a bigger step.

Mirko Stocker interviewed me for this InfoQ article on rush.

I've created a mailing list for rush, since discussion in the comments was getting a bit unwieldy.

rush now has RDocs and a gem.

The unix shell (bash) and remote login (ssh) are centerpieces of the server and app deployment process. While building Heroku, however, Orion and I became aware that these tools are pretty far out of step with modern, agile development practices.

I've wanted a Ruby-syntax replacement for the unix shell from almost the moment I began using Ruby. Whenever I can, I write shell scripts as Ruby scripts with lots of backticks. But the "everything is text" mechanism starts to show its age when you end up with Ruby code like this:

my_ip = `ifconfig | grep inet | grep -v 127.0.0.1 | grep -v inet6`.match(/inet ([\d.]+)/)[1]

Yergh. (If you've never had occasion to write code like this in the wild, just check out god's process lookup methods.)

What we really want - the modern way - is to query the unix system (filesystem, processes, network, services) as if they were a database. This avoids the fragility of text pipes, the complexities of firing up a complete new environment on each system call, and would allow unit tests of system-level code.

This is why I've created rush. It's a replacement for bash and ssh which uses Ruby syntax. More than that: it IS ruby. Imagine an irb shell in which you can do everything you can do at the unix command line, but without any backticks. That's the vision; what I've got so far is a good start in that direction.

I said it replaces ssh, so this isn't just a shell: you can use it to control an arbitrary number of remote boxes, using the exact same interface as you would locally. Copy a file, or grep through a logfile, or kill a process - whether the machine is remote or local, the interface is identical.

Unlike the character-based connection of ssh, the rush client connects to the rushd process on the remote server and passes discrete commands. This is very similar to connecting to a remote database. When you run a SQL query, it makes no difference to the programmer whether the connection is a remote box or a local one; the client handles this seamlessly. You can even connect to multiple databases from the same client. rush goes even a step further by allowing you to pass data seamlessly between any number of local and remote connections.

A quick example:

local = Rush::Box.new('localhost')
remote = Rush::Box.new('my.remote.server')
local['/etc/hosts'].copy_to remote['/etc/']

Check the rush website for more examples and to try it out.

One of the inspirations for rush as a shell was this preview of MSH, the Microsoft shell. I get the feeling that this is vaporware (though I don't really know, not being in the Microsoft world at all), but the concepts introduced in the preview really struck home. Treating data returned from shell commands - like file matches from grep or processes from ps - as discrete objecs, rather than text which can be parsed, is the obvious next evolution for shells.

There are some other deficiencies in the bash+ssh model:

• Consistency. Bash is a full-fledged programming language; more specifically, it's a DSL for managing a unix system. But it could also be considered a collection of smaller languages. Standard tools like cp, mv, ps, grep, sed, and sort all have their own unique syntax. You may combine several of these in a single command, which is a bit like mixing several different programming languages on one line. I've been using unix shells on a daily basis for well over a decade, and still I sometimes forget the syntax for a particular command. Compare this to Ruby, or any other modern scripting language, where just a few months of working with the language is enough to teach you 90% of the language's syntax.
• Quoting. Bash commands often have many layers of quoting. Consider:

  ssh remote "rm `grep '^class Thing' lib/* -l`"
  

  This has four layers of quoting: the bash command line on the client, the bash command line on the server, the backticks, and the regexp. This leads to both confusion (do I need one backslash or two to escape this quote character?) and is riddled with security holes.
• Quirks and limitations. Two that I frequently bump into are running out of space in the command line buffer space with backticks, such as:

  grep some_method `find . -name \*.rb`
  

  On a large project, you'll need to rewrite this with xargs:

  find . -name \*.rb | xargs grep some_method
  

  If the directory has filename with spaces in them, you have to use the null separator option on both find and xargs:

  find . -name \*.rb -print0 | xargs -0 grep some_method
  

  Ick. In rush, this would be:

  dir['**/*.rb'].search(/some_method/)
  

• Exceptions. Bash commands have three outputs: stdout, stderr, and the shell return value. Most of the time you're only interested in one and can ignore the others. But for more advanced uses, you need two, or perhaps all three. Explicitly checking for return values (or worse, pattern matching against stderr) is not a lot of fun. Exceptions are the modern way to handle errors.

Go give it a try, and then tell me what you think.