rush, the Ruby Shell
Posted by Adam Wiggins on February 19, 2008 at 01:07 PM
The unix shell (bash) and remote login (ssh) are centerpieces of the server and app deployment process. While building Heroku, however, Orion and I became aware that these tools are pretty far out of step with modern, agile development practices.
I've wanted a Ruby-syntax replacement for the unix shell from almost the moment I began using Ruby. Whenever I can, I write shell scripts as Ruby scripts with lots of backticks. But the "everything is text" mechanism starts to show its age when you end up with Ruby code like this:
my_ip = `ifconfig | grep inet | grep -v 127.0.0.1 | grep -v inet6`.match(/inet ([\d.]+)/)[1]
Yergh. (If you've never had occasion to write code like this in the wild, just check out god's process lookup methods.)
What we really want - the modern way - is to query the unix system (filesystem, processes, network, services) as if they were a database. This avoids the fragility of text pipes, the complexities of firing up a complete new environment on each system call, and would allow unit tests of system-level code.
This is why I've created rush. It's a replacement for bash and ssh which uses Ruby syntax. More than that: it IS ruby. Imagine an irb shell in which you can do everything you can do at the unix command line, but without any backticks. That's the vision; what I've got so far is a good start in that direction.
I said it replaces ssh, so this isn't just a shell: you can use it to control an arbitrary number of remote boxes, using the exact same interface as you would locally. Copy a file, or grep through a logfile, or kill a process - whether the machine is remote or local, the interface is identical.
Unlike the character-based connection of ssh, the rush client connects to the rushd process on the remote server and passes discrete commands. This is very similar to connecting to a remote database. When you run a SQL query, it makes no difference to the programmer whether the connection is a remote box or a local one; the client handles this seamlessly. You can even connect to multiple databases from the same client. rush goes even a step further by allowing you to pass data seamlessly between any number of local and remote connections.
A quick example:
local = Rush::Box.new('localhost') remote = Rush::Box.new('my.remote.server') local['/etc/hosts'].copy_to remote['/etc/']
Check the rush website for more examples and to try it out.
One of the inspirations for rush as a shell was this preview of MSH, the Microsoft shell. I get the feeling that this is vaporware (though I don't really know, not being in the Microsoft world at all), but the concepts introduced in the preview really struck home. Treating data returned from shell commands - like file matches from grep or processes from ps - as discrete objecs, rather than text which can be parsed, is the obvious next evolution for shells.
There are some other deficiencies in the bash+ssh model:
- Consistency. Bash is a full-fledged programming language; more specifically, it's a DSL for managing a unix system. But it could also be considered a collection of smaller languages. Standard tools like cp, mv, ps, grep, sed, and sort all have their own unique syntax. You may combine several of these in a single command, which is a bit like mixing several different programming languages on one line. I've been using unix shells on a daily basis for well over a decade, and still I sometimes forget the syntax for a particular command. Compare this to Ruby, or any other modern scripting language, where just a few months of working with the language is enough to teach you 90% of the language's syntax.
- Quoting. Bash commands often have many layers of quoting. Consider:
ssh remote "rm `grep '^class Thing' lib/* -l`"This has four layers of quoting: the bash command line on the client, the bash command line on the server, the backticks, and the regexp. This leads to both confusion (do I need one backslash or two to escape this quote character?) and is riddled with security holes. - Quirks and limitations. Two that I frequently bump into are running out of space in the command line buffer space with backticks, such as:
grep some_method `find . -name \*.rb`On a large project, you'll need to rewrite this with xargs:find . -name \*.rb | xargs grep some_method
If the directory has filename with spaces in them, you have to use the null separator option on both find and xargs:find . -name \*.rb -print0 | xargs -0 grep some_method
Ick. In rush, this would be:dir['**/*.rb'].search(/some_method/)
- Exceptions. Bash commands have three outputs: stdout, stderr, and the shell return value. Most of the time you're only interested in one and can ignore the others. But for more advanced uses, you need two, or perhaps all three. Explicitly checking for return values (or worse, pattern matching against stderr) is not a lot of fun. Exceptions are the modern way to handle errors.
Go give it a try, and then tell me what you think.
Comments
There are 31 comments on this post. Post yours →
A great, bold experiment. I'm pondering how much I like it, but I think I'm going to have to play with it a few days before I know.
BTW ack is a great replacement for your 'find' examples.
Very, very cool...
I haven't looked closely at the syntax but the "vision" is fantastic and well-put.
MSH became "Windows Powershell", and is currently in production in some of MSFT's big server products. Check the Wikipedia article for more info.
Awesome.. I really like the concept.. if nothing else, but as a way way to force myself to use ruby more.
I did notice on the mac that if I exec "something" .. that it closes rush, is there a better way to do that?
In particular on the mac, I use "open something" a lot since the mac does it's magic to use the right viewer/finder/whatever .. but that doesn't work with home['filename'].open
Great! You stole my idea ;-) Although I had the name "roosh" in mind (for Ruby OO Shell). The idea came to me exactly from the same Microsoft technology, now called Powershell (not vaporware anymore). My introductory article to Powershell may interest you, available at http://webmat.wordpress.com/discover-powershell/ Of course, as a Rubyist I pretty much find the Powershell console/scripting language irrelevant, but the idea of an OO shell is clearly the future. You might see me popping up in your mailing list soon, if you make this into a full-fledged open source project.
Fantastic! I look forward very much to trying this out. Is this a continuation of http://rubyforge.org/projects/rush/? I tried it for a while, but I recall it being somewhat incomplete, and it's not been updated in a few years.
This all makes sense for doing powerful things with a shell, things which I would have had to open up Dolphin for in the past. I might look into using it for this.
But it can't really be a bash replacement if typing entering firefox results in a method missing error.
Really I've always wanted a shell with ruby syntax. Bash syntax just sucks, like you I code systems scripts in ruby with lots of backticks. I shouldn't have to open up irb if I want to do some basic math.
But I think the basic concept of being in a directory and running commands from the path is valid.
I think I'm in love with you, Adam.
Sounds very cool. If you haven't already, check out Hotwire, which is also partly inspired by MSH/PowerShell, and is based on Python.
One of the other interesting ideas behind Hotwire is that since almost every terminal window now runs on a system with an advanced graphics system, a terminal app with a better understanding of the meaning of outputs can do clever things to format the display.
Very cool...and I completely agree on the need.
I think if rush is truly going to supplant a 'bash' then the typical Unix commands should work. For example, I typed in 'ls' expecting to get the listing of the directory I was in.
Might have to grab the source code and see what it will take to simply exec local commands.
Thanks dude for throwing this out there - and taking the criticism and praise.
Nice! Definitely worth a try. Would be nice if this gets incorporated into some distributions.
And so began a new era of shells ;)
I have thought exactly on this ideas quite a lot.
I've done dome experiments but mostly with custom ruby-bash scripts, for my own build system.
http://snipplr.com/view/3424/sheller-example/ http://snipplr.com/view/2669/sheller/
Perhaps I could join your effort?
Thanks Adam as the idea is fantastic and rush already usable.
I am not (yet ;-) ) a rubyst and I might have missed something but is there a possibility to execute a command remotely ? For example how would you call a perl script remotely (located remotely) ? If it isn't possible I miss that part.
Thanks in advance for the tip and let us know if you want contributors
Looks like a very good idea. Congratulations ... (Heroku is very good too.)
Having the same interface for local and remote is cool too.
A thought - not directly related to rush, but to the idea of shells in general, and specifically to the new process invocation overhead associated with invoking commands like rm and cp from the shell:
If we had operating systems like Oberon (created by Niklaus Wirth who also created the Pascal and Modula languages, among other things), that process invocation overhead would go away.
After I started writing this post, I browsed around for some related links (on Prof. Wirth and Oberon) - here they are:
http://www.oberon.ethz.ch/
http://en.wikipedia.org/wiki/Niklaus_Wirth
http://www.eptacom.net/pubblicazioni/pub_eng/wirth.html
That last link - "A Few Words with Niklaus Wirth" - is rather thought-provoking ...
Sorry, that should have read "After I started writing this comment", not "After I started writing this post" :-)
this is great! I was thinking of doing something like this, you beat me to it. I think I'm going to poke around in the source.
btw, why rush? would've gone for 'rash' way cooler i think :)
@Kit, Agreed. It would be nice to replace some unix commands, and make available the option to drop down to the unix metal whenever necessary.
An earlier rush (http://groups.google.com/group/comp.lang.ruby/browse_thread/thread/7f8d2b09f2c9613b/be519bd6bc7f1084?hl=en) allows doing this by prefixing commands with '!'. I can't tell if Adam is extending this abandoned project, or if the name is just a coincidence.
@Guillaume & Kit - There is a "bash" command on the Rush::Box class you can use to execute arbitrary commands. (I don't think this is in the gem yet, just the latest git repo.) So Rush::Box.new('www@deploy').bash('mongrel_rails restart'), for example.
The bang syntax sounds pretty good as a local shortcut, although I'll need to think about that a bit more. I really want to avoid polluting Ruby syntax with special cases. I really want the shell to be pure Ruby with access to some special libraries for manipulating the unix system, rather than a hybrid of Ruby and my own special syntax.
And no, this is not an extension of any of the earlier projects which used the name rush. All of those are defunct (or in some cases, never released any files); I thus felt it was ok to grab the name.
@Felipe - I'd love to have help - it's an open source project, after all. Email patches to me if you've got something you think is worthy of inclusion.
Hey - what a great tool. I was actually contemplating doing something like this and you've already done it - and seem to have done it well.
I had a few questions about using it on a windows client - maybe you could give me an idea of where to look - or maybe the git source works - I have the gem.
I love unix - but I do need to control some boxes that are win32.
I installed COPSSH - added a user rush, create the file /home/rush/.rush/passwords, shared the public keys - found the sleep executable for win32 and dropped it in - and now am stuck. The Post call to the server never completes. (remote.rb:96) I'm wondering if there is some catch with the passwords file that I had to 'touch' to get everything working.
Any help would be appreciated. Thanks a lot.
@Cary - Is rushd running on the server? Can you use telnet to check that it's listening on the port, and the ssh tunnel is working?
I suspect that getting it running on win32 will be a fairly substantial port. I'm hoping someone will step up and contribute some quality patches for this. I do like the idea of abstracting away the operating system - at least for common things like file manipulation.
I was playing around with rush, and i had a question. It says on the website that there is no notion of current directoy. why not? wouldn't this get inconvenient?
also do you have an issue tracker or forum or something? i'd like to help out with rush, and im guessing you must have a bug tracker or perhaps a todo list somewhere?
Hey Adam,
It may not be running rushd—though it thinks it is. I'll have to look into it.
If you have areas where you know/suspect there will need to be mods - I'm interested in helping out as I can. I was going to have to develop something like this anyway...
Maybe on win32 you just set an ENV var as 'rush_root' and do everything relative to that.
If you get a forum/bug tracker, etc we could go over what it might take.
@Adam -
To abstract file manipulation, use the built-in classes File, File::Stat, and FileTest, plus the included FileUtils.
Together, these should allow you to make most file operations operating-system independent.
My questions have to do with security (remote and local) and with permissions (remote and local). How does that work?
This does look uber cool, though!
Thanks!
@Mike - I do use FileUtils and friends whenever possible. But these tools are limited - that's part of why I wanted to write rush. So I have to drop back to shell calls or reading from /proc for many of the more advanced functions. Take a look at local.rb for the nitty-gritty.
As for security, the model is the unix one: you're constrained to whatever actions the user rushd is running as. If you choose root, you can do everything. (Soon I'd like to add a set of features for doing operations as other users that you can sudo to.)
@hellfeuer - There's no notion of current directory because it's designed to be stateless. If you want to do a series of operations on a directory, just store it in a variable.
You guys are right - discoursing in these comments is getting unwieldy. I'll go set up a mailing list.
I think (IMHO) rather than a '!ls' solution it would be cool to see (just like bash) the command get attempted straight off the "environment" path.
sleep command - you can replace sleep 9000 with
ruby -e 'sleep 9'
That helps you in moving to win32...
Just a thought.
very very exciting stuff... i've been wanting a ruby shell replacement but you've actually gone done built one. amazing, i'll be keeping an eye out
This is SUCH a good idea. I bash scripts, but I hate awkward bash syntax. You make an excellent point about how bash 'methods' (grep, find) all have a totally independent syntax.
I'm at a transition point for switching from bash, and this is what I've been looking for. I think I will rewrite my bash scripts in ruby just to get used to it, in conjunction with your efforts, which I will follow. I think it will give my scripts less of a 'glued together/one off solution' feeling.
Awesome idea. I've changed the default shell on my boxes and gonna try this out for a while. Following the mailing list. Intriguing!
I really like that all I had to do to get remote rush to work was to start rush once and close it. used ssh keys and everything. nice!
Hi, I'm the lead developer on the origional ruSH in 2005. Give it a try: https://rubyforge.org/frs/download.php/5752/rush-beta4.tar.gz
I'm sure the curses trickery will be a pleasant surprise to some (try tab completion). Check out the readme file. You dont need our custom curses binary anymore, my changes to ruby' curses lib was updated a while back.
AFAIK if you have a standard ruby installation, all you need is the latest version of open4.
The project suffered from some issues, some of which were curses related.
Sorry for the horrible background color, its easily changeable.
Thanks.
Wish this would work on Windows...
I am playing with heroku, and would like to push an app up, but I am dancing around all kinds of problems finding the right version of Git, installing CygWIn, etc.
Would like to use RUSH to do this alll.
Post a comment
Required fields in bold.