Adam @ Heroku

Dude, That Is So Fringe

Adam Wiggins — Tue, 22 Jul 2008 06:00:00 GMT

Rubyfringe was great. Off-the-beaten-path topics, small size, tons of style, and countless small touches made it stand apart from the typical tech conference. (They included a Rubyfringe-branded condom in the swag, for gawdsakes.)

Quick summaries of some of the talks follow. Warning, these probably reflect my own disposition as much as that of each speaker.

Dan Grigsby - Don't work for the man, be a programmer/entrepreneur instead. Treat each venture as an experiment and don't be shy about terminating the ones that don't work. You'll strike out a lot but eventually hit a home run.
Yehuda Katz - Living on the edge is fun, but also dangerous. Finding a balance between the cutting edge and the bleeding edge is the trick.
Luke Franci - There are methods for testing other than code-oriented ones like unit testing. RCov doesn't mean much. Use QA, usability tests, and code reviews for well-rounded coverage.
Obie Fernandez - To be successful at consulting, watch the balance of power in the relationship with your clients. Most Rails freelancers are charging too little - he recommends no less than $100/hr.
Matt Todd - Don't be afraid to dive in with both feet. Make mistakes, learn from them. Pick good problems to solve.
Jeremy McAnally - Frameworks are getting fat. Frameworks should be specific. Don't use Rails (or any other framework) outside of its domain.
Hampton Catlin - Javascript is a good general purpose language, but terrible in the browser. Add-on libraries like Prototype and JQuery are just band-aids on this problem. The client-side language should be tightly coupled to the DOM, like CSS. He proposes Jabl.
Giles Bowkett - Fuck the man, fuck the mainstream. Programming is a tool to do your art. His art is music, his tool is Ruby, and the result is Arcaeopteryx. (Giles stole the show with this talk. He ran like twice as long as his alloted time but we were all so into it that no one minded.)
Damien Katz - If you want to become the guy that gets paid to build cool things, take a risk: start by making something cool without any plan for how to get paid for it. (He did that, and now IBM pays him to work on CouchDB and contribute it to the Apache Foundation.) Also, Erlang is sweet.
Reginald Braithwaite - It's too bad Ruby isn't more like Lisp or Haskell.
Tom Preston-Warner - The scientific method rules, use it for coding (and life). Ruby 1.8 has weird memory leak bugs. Github now has a git-powered pastie site, gist.github.com. Also, Erlang is sweet.
Blake Mizerany - Sinatra is a framework for fast, small web services. Routers are unnecessary obfuscation; treat a web resource url like you would a function name, and address it directly.

The overarching theme of the conference seemed to be that Ruby's steady march into the mainstream means it's losing its luster for us, the early adopters. Ruby was once fringe, but now it's not. We're now all in seek of the fringes of Ruby, and of software tech in general.

You'll note the use of "fringe" as an adjective. A unique culture seemed to emerge from the attendees over the weekend, and using "fringe" this way - as in, "Dude, that is so fringe" - was one trait of that culture.

It's a great term. The fringe refers to all the places where the weird, interesting, chaotic experimentation goes on. This is the spawning ground for tomorrow's new hotness, but the fringe never looks like much when looking at it from the mainstream. Why do I need a new database paradigm? SQL serves just fine, thank you. What about a new web framework, a new object relational mapper, a new transport protocol, a new language? Rails, ActiveRecord, HTTP, and Ruby also serve just fine. CouchDB, Sinatra, DataMapper, XMPP, and Erlang are on the fringe, along with countless other even less well-known projects. These things are not just weird in how they work, they're weird in that they solve problems that most people don't even know they have. That's the fringe.

"Fringe" is more descriptive than the more commonly used "cutting edge." Cutting edge implies a one-dimensional graph, as if tech is on a single well-charted path toward an ultimate destination. Put that way, who wouldn't want to be on the cutting edge, meaning you're further down that inevitable path?

But that's not how it is. The state of tech, charted over time, is an N-dimensional graph. The fringe is the ragged edges of that graph, the weird bits hanging off the edge. Weirdos with weird visions doing weird stuff that few besides them understand. 99% of this weird stuff never turns into anything. But some tiny fraction turns out to be a new direction for tech, the next big thing, the new hotness, a revolution. In this, it has more in common with biological evolution than with the design and planning we associate with the works of man.

Early adopters crave being in the fringe. We love the chaos, the freedom for wild experimentation, the cognitive challenge of trying to predict which bits of this massively heterogeneous mess may turn into something world-changing. It's not the best place to be from a practical standpoint: proven tech from the mainstream is what you want for getting "real" work done. But there's a satisfaction you can get in the fringe that can't be found anywhere else.

rush 0.4

Adam Wiggins — Wed, 16 Jul 2008 04:55:00 GMT

rush 0.4 released. Some changes:

Rush::Box#processes returns a ProcessSet, for syntax like this:

processes.filter(:cmdline => /mongrel_rails/).kill

Daemonize shell commands: bash 'some_daemon', :background => true
Pass args to rush on the command line to execute a one-off, like this:

$ rush 'processes.filter(:command => "ruby")'

Details and rdocs.

Owning Up

Adam Wiggins — Sun, 13 Jul 2008 19:35:00 GMT

The healthcare industry is starting to see benefits from being honest when they make mistakes. Denying that doctors and hospitals ever screw up has been the historical approach to avoiding malpractice suits; yet by being more honest, hospitals are seeing a decrease in such lawsuits.

If doctors can do it, why not software authors? Own up to your mistakes. People will trust you more in the long run.

The End of Bugs?

Adam Wiggins — Sun, 06 Jul 2008 16:55:00 GMT

Although I've been a believer in TDD/BDD for quite a while, rush was the first time I started a project and said, ok, THIS time I'm going to get really serious about it. My very first commit was a bunch of empty specs, and since then, I don't think I've committed any new feature or even a bugfix without an accompanying spec.

A few weeks into the project, I had made really substantial progress and had quite a lot of functionality. One day, while using it to access a remote machine to do some file operations, something very surprising happened: I typed a certain command and it didn't behave the way I expected it to.

I figured I must have a typo or something, but double-checking it, I realized that, no: my command was correct, it was the program which was behaving incorrectly. I had found, it seemed, a bug.

I felt a sudden sense of disorientation and panic. The program wasn't behaving as expected? What should I do? Log things? Run a debugger? Just squint at the code for a while and see if I could spot the problem? These methods seemed so...crude.

And yet, I realized, these techniques are the very ones I've been using my whole career, that I use every single day to get work done. I don't give them a second thought. Yet, after two weeks of doing pure BDD, the idea of spending time dealing with a bug seemed foreign and painful.

I ended up tracking down the problem and discovered that, despite my nearly 100% code coverage (as reported by RCov), I had a fork in a single-line conditional that was not speced. Upon realizing this, another feeling washed over me: a sense of mistrust in the project's codebase, because a fragment of one line of code did not have spec coverage. I wrote the spec, confirmed that it failed / exposed the bug, and then fixed the bug itself ten seconds later. Relief flooded over me: things were right with the world again.

Nearly all developers, myself included, spend most of our time in that state: not quite trusting that the code all works. I only had the chance to realize how stressful and unpleasant this state is because I lived outside it for a little while.

This experience also told me something else: after several weeks of intense development, this was the first time I had encountered a bug. I feel absolutely certain that that's never happened to me before.

You could argue that rush was not then (and is still not now) a very big or complicated program, but I don't think that's significant. First, by this time I was already using it to control a remote server (i.e., it was communicating commands over the network via rushd); that's the sort of thing that is highly prone to bugs. But more importantly, even very short and simple programs often have bugs lurking in them.

My whole life I've assumed that bugs are a given in software, and that trying to eliminate them completely is a waste of time. And I think I was correct, for the traditional, non-BDD approach. ("Code-driven development," I guess?)

But it may be that a very disciplined approach to BDD means the opportunity to have truly bug-free software. Just imagine what it would be like to have all the time you spend debugging code go into writing it instead. Granted, a lot of that goes into writing specs; but for me, writing specs is far more fun than debugging. It's programming, not sleuthing.

Bug-free software is a very bold claim, though it does require a somewhat narrow definition of the word "bug." A bug is a situation where the code has been specified to behave one way, and instead it behaves another. It does not include things that users or developers may like the software to do, but that it is not specified to do currently. It also doesn't include external depedencies (libraries, web services..) behaving in an unexpected manner - good software should try to recover gracefully from failures in external services, but doing so is a type of feature.

Look back at that definition of bug again: "...the code has been specified to behave one way..." BDD is the very act of writing executable specifications. In non-BDD, there is no specification; so by definition, software written that way can be said to either have an infinite number of bugs, or perhaps just no functionality that you can rely on.

Read-Only Source Trees

Adam Wiggins — Wed, 02 Jul 2008 08:06:00 GMT

Cloud computing is on everyone's minds, because it offers the promise of infinite horizontal scalability. But to achieve this, we have to change how we build applications.

One such change is how we use the filesystem. The filesystem is unix's database. "Everything is a file" has served us well for decades, and that concept will continue to be critical at the systems layer. But at the application layer, it's time to stop treating the filesystem as a catch-all dumping ground, and start treating the data we store there in a more structured way.

An app's main use of the filesystem is sourcefiles. What qualifies as a sourcefile? Your code, sure - Ruby, ERB, HTML, Javascript, CSS, specs/tests, rake tasks. But also, small static assets that are part of the application's interface, like public/images/top_left_gradient.png and public/robots.txt. If you check it into revision control, then it is probably a sourcefile.

Other than sourcefiles, what do we stick on the filesystem? PIDs and logfiles come to mind. Anything that it is in tmp or log. This stuff is not source, which is probably why it's in your .gitignore. In my opinion it should not be in your application's directory structure at all.

How about user-uploaded assets, like profile pictures? attachment_fu offers a filesystem backend, which shoves files into your public/ dir. But these are not source - it's application data. It has more in common with the contents of the database: data specific to a particular installation of the app. Putting this data into your source tree is confusing.

More significantly, it greatly complicates the problem of scaling.

The correct solution, in my opinion, is to forbid access to the source tree by the web app. Temporary files can be offered through Ruby's Tempfile interface, with the understanding that files thus created are not accessible beyond the lifetime of the request being served.

Logs are a whole other challenge. I'm not a big fan of logfiles; there are better solutions to the logging problem, which I'll write about some other time. In the meantime, logs should go outside the code tree, some sort of /var-style location which can be cycled or thrown away as needed. This location could be write-only for the app; it pushes things in, but it can't read them back or otherwise access it once written. A one-way channel, ala syslog.

As for attachments, asset stores are the correct solution. attachment_fu's :storage => :s3 backend, for example. Storing in the database is reasonable, though I've always found a lot of frustration in trying to store large binary data in the database. Apps on Heroku can use the :storage => :heroku attachment_fu backend.

As we continue to explore the next generation of application deployment, I think we're going to bump into a number of ways to structure apps differently in order to make them scalable. There will be some transitionary pain with these changes, because structure implies restrictions. Many PHP developers coming to Rails have complained about not being able to access sessions from models, or write SQL in your view. MVC creates restrictions, yes, but those very restrictions are what provides the structure. Coming from an unstructured environment, those restrictions may seem cumbersome or arbitrary; but once you're in the habit, you come to appreciate the structure they create.

Rebasing is Editing Commits

Adam Wiggins — Mon, 30 Jun 2008 10:19:00 GMT

Rebase is one of Git's most alluring and yet most difficult-to-comprehend features. Rebasing is editing commits. When you rebase, you're rewriting history.

This is possible with Git because of the separation between a commit and a push. A commit is a changeset with some attached metadata like the commit message. A push publishes your commits to a remote repository. In Subversion, these steps are inseparable, both part of the commit.

Because of this separation, you can rewrite your commits before you push them. Just as you can edit your sourcefiles as many times as you want before you commit the results, so to can you edit your commits as many times as you want before you push the results.

Rebasing comes in two main forms. One is the interactive rebase. git rebase -i HEAD~5 pops you into an editor where you can change the order of the commits, delete entire commits, or squash commits together. You can also edit a commit, which will take you out of the editor and let you work on the commit in your working tree, and then commit with git commit —amend.

The other type of rebase is rebasing your local commits on top of some changes you're pulling from a remote source, or against a local branch. (Hence the term "rebase": you're creating a new base for your patches.) The action takes some new commits from a branch and slips them in underneath yours, at the point the two branches diverge. Commits prior to the divergence point are unaffected.

I use this kind of rebasing instead of git pull. You'll notice that pull almost always creates a merge commit, which is one of these things:

commit c4110e1fb1aa50c4f876716bde07f6a982a1f31c
Merge: 296a0db... cb6050b...
Author: Joe <joe@example.com>
Date:   Tue Jun 24 14:46:41 2008 -0700

    Merge branch 'master' of example.com:repo.git

You might wonder why a merge commit is needed. Subversion doesn't have that, after all. But that's because the merge commit in svn is always implicit. Did you ever find yourself working on an active project, and then when you went to commit, you needed to svn up and got a whole boatload of changes, including some conflicts? Sure you did. And then you had to sit there and perform the merge on your source, finally making a big commit which included both your changes and the merge.

Git encourages discrete changesets, so it makes sense to break apart regular changes (new feature, bugfix, etc) from merges. But on an active project with lots of contributors, there's always merging going on. So you end up with lots of ugly merge commits cluttering up your logs.

Rebasing lets us have and eat our cake. Now you can make atomic commits as you're working, regardless of whether you are ready to share those commits with your team. But when you want to pull down the latest work from your team and merge it with your work, you can instead use rebase to reapply your patches on top of theirs. If you're not working on the same areas of the code, then this takes almost no work. Just type git fetch && git rebase origin master and you're done. (Notice that the output of git log then shows your recent changes at the top, regardless of the timestamps.)

Occasionally there are merge conflicts during the rebase, and this will drop you out into a shell with some rather intimidating messages. Don't worry though, all you need to do is take a look at the conflicted files, choose the part that you want (just like resolving a regular merge conflict), and then git add them. When you're done, run git rebase —continue. It does this merge step separately for each commit, so if there are a lot of commits in the difference between you and the remote source, this could get time-consuming. In that case you may want to git rebase —abort and then run a regular git merge or git pull. Next time, resolve to rebase more frequently, to make the merge job less of a headache.

Since rebasing is editing of commits, it doesn't make much sense to rebase things that have already been pushed. You can do it, but as soon as you go to merge with another repo that had the unedited commit history, you'll bump into weirdness (and probably invalidate your whole reason for rebasing, which was to clean up the history). So as a general rule, I recommend never rebasing things that have already been pushed.

If you push something and then realize five seconds later that you shouldn't have, it is possible to rebase your local branch and then git push --force, which will obliterate the remote repo's history. This won't help if someone else has already pulled the commits, since the next time they push the commits will come back, so only use it when you're certain that no one else has pulled.

EVDO Rules

Adam Wiggins — Sat, 28 Jun 2008 10:21:00 GMT

EVDO cards rule. The speed and latency are good enough for working over ssh and other common development tasks like pulling up documentation on the web or installing a small gem or two. (Though I don't recommend doing a git clone of Rubinius over EVDO). At Railsconf I used my EVDO card instead of the spotty wireless and was tapping away happily as everyone else struggled to load google.com.

Mine is from Verizon, $50/mo, worked on my Mac without as soon as I plugged it in, no software installation or configuration of any kind. So far I've been able to use it everywhere I've traveled (all within the US so far, though I'll be racking up some of those $0.002/kb roaming charges in Canada on my way to Rubyfringe next month).

During a road trip to Santa Barbara last month, the pager went off right in the middle of farm country. I was able to crack open my laptop and take care of the problem without even pulling over. sshing at 75 mph is a whole new experience. (No, I wasn't driving.)

EVDO has changed the way I work and travel. Cutting the tether means I'm no longer afeared of getting stuck in a waiting room for an hour - that's just enough time to crank out a cool new feature.

Office Aesthetics

Adam Wiggins — Fri, 27 Jun 2008 08:26:00 GMT

Slicehost's new offices are mouth-wateringly gorgeous.

Service-Oriented Architectures

Adam Wiggins — Wed, 25 Jun 2008 18:56:00 GMT

I've always liked to build systems with a bunch of small apps that talk to each other through various protocols. Orion and I built TrustCommerce in this manner, and that gave it some pretty impressive fault-tolerance and scalability.

I had heard the term SOA (service-oriented architecture), but had always dismissed it as enterprisey talk. (Bland-yet-pompous three-letter acronyms make my brain turn off.)

At some point, it dawned on me that what I like to do - build small apps communicating with each other over the network - is exactly what SOA means. In its modern incarnation, service architectures use REST calls, which follows the unix tradition of small sharp tools, loosely coupled by a simple but extremely flexible protocol.

Heroku is no one app - even aside from all the server software and configs, our code is currently split among around two dozen apps, each with their own repository, and most with their own database. (Some have no database.) Most of these are Rails apps, though some are bare Ruby on a Mongrel or bare Rack on a Thin, and some are Sinatra apps.

This is why my Railsconf talk was about HTTP routing. When you've got dozens of apps, some of which respond to complex domains (for example, edit.*.heroku.com), a powerful http router outside your application VM becomes damned near indispensable.

Service architectures are the solution to the longtime problem of apps growing to monstrous proportions. Once you exceed a couple dozen models and/or controllers, it starts to be very hard for new developers to grock everything that's going on, and the barrier of entry becomes very high.

With a service architecture, each app has a simplicity that's reminiscent of the "make your own blog" sample apps you see in tutorials. Rails seems to better retain its beauty in this state. Probably everything does.

But it introduces new challenges. You've got dependencies between repositories - any time you change the interface, you have to be sure to roll out the server and client apps together. The relationship between internal apps is thus very similar to external ones - you need versioning and dependency management. Heroku has an app we call our architecture atlas which tracks all the components, dependencies between them, and documents their APIs.

Managing authentication becomes a big job. We do a lot with custom HTTP headers on this (again, one of the main topics in my Railsconf talk), but I've got my eye out for even more sophisticated solutions. OAuth is one that has piqued my interest.

Perhaps the hardest question is where to draw the dividing line between one app and the next. Does a given model go in app A or app B? And that requires a lot of hard thinking about your design. An app should do just one thing, and without having to touch other apps too much. This often means splitting an app apart, and occasionally fusing two apps back together. This is the same process as managing object classes within an app: as each item's responsibility within the architecture changes, code moves around.

I find it useful to think of each internal component as its own service that could potentially be spun off as its own company. If it has its own code repo, database, tests, API, and docs, then turning it into a standalone service would just be a matter of giving it a slick marketing name and putting up a website. It's not that you'd want to do this, mind you: but if your service architecture is designed well, it would be easy to.

Git Submodule

Adam Wiggins — Tue, 24 Jun 2008 17:09:00 GMT

Git submodules are pretty cool, except for kind of sucking. Things I don't recommend doing if you value your sanity:

Switching a submodule from one repository to another (i.e., editing .gitmodules and changing the repo url)
Switching a directory from a submodule to regular content
Switching a directory with regular content to a submodule (though this might help you)

It's a shame, because submodules are pretty handy. But you'll probably end up wanting to do one of these things during the lifetime of your project, and then you're screwed.