Cloud computing is on everyone's minds, because it offers the promise of infinite horizontal scalability. But to achieve this, we have to change how we build applications.

One such change is how we use the filesystem. The filesystem is unix's database. "Everything is a file" has served us well for decades, and that concept will continue to be critical at the systems layer. But at the application layer, it's time to stop treating the filesystem as a catch-all dumping ground, and start treating the data we store there in a more structured way.

An app's main use of the filesystem is sourcefiles. What qualifies as a sourcefile? Your code, sure - Ruby, ERB, HTML, Javascript, CSS, specs/tests, rake tasks. But also, small static assets that are part of the application's interface, like public/images/top_left_gradient.png and public/robots.txt. If you check it into revision control, then it is probably a sourcefile.

Other than sourcefiles, what do we stick on the filesystem? PIDs and logfiles come to mind. Anything that it is in tmp or log. This stuff is not source, which is probably why it's in your .gitignore. In my opinion it should not be in your application's directory structure at all.

How about user-uploaded assets, like profile pictures? attachment_fu offers a filesystem backend, which shoves files into your public/ dir. But these are not source - it's application data. It has more in common with the contents of the database: data specific to a particular installation of the app. Putting this data into your source tree is confusing.

More significantly, it greatly complicates the problem of scaling.

The correct solution, in my opinion, is to forbid access to the source tree by the web app. Temporary files can be offered through Ruby's Tempfile interface, with the understanding that files thus created are not accessible beyond the lifetime of the request being served.

Logs are a whole other challenge. I'm not a big fan of logfiles; there are better solutions to the logging problem, which I'll write about some other time. In the meantime, logs should go outside the code tree, some sort of /var-style location which can be cycled or thrown away as needed. This location could be write-only for the app; it pushes things in, but it can't read them back or otherwise access it once written. A one-way channel, ala syslog.

As for attachments, asset stores are the correct solution. attachment_fu's :storage => :s3 backend, for example. Storing in the database is reasonable, though I've always found a lot of frustration in trying to store large binary data in the database. Apps on Heroku can use the :storage => :heroku attachment_fu backend.

As we continue to explore the next generation of application deployment, I think we're going to bump into a number of ways to structure apps differently in order to make them scalable. There will be some transitionary pain with these changes, because structure implies restrictions. Many PHP developers coming to Rails have complained about not being able to access sessions from models, or write SQL in your view. MVC creates restrictions, yes, but those very restrictions are what provides the structure. Coming from an unstructured environment, those restrictions may seem cumbersome or arbitrary; but once you're in the habit, you come to appreciate the structure they create.

Programmers tend to overdesign. Rather than building a quick solution that works right now for the specific case, we want to build one that will solve all problems of that type, both now and for all time. Dreaming In Code shows a particularly egregious case of this: the developers spend years building a framework for the app, rather than the app itself. This type of story is quite common. So frequent is this pitfall that the agile methodology mantra for avoiding it if often referred to by its abbreviation: YAGNI (You Aren't Going To Need It).

On the opposite extreme, there's the quick-and-dirty hack. But this has its own problems: it's fragile and inflexible. (It's telling that Microsoft's first pass at an operating system was named QDOS, where the QD stood for Quick-and-Dirty.) Quick hacks don't lend themselves to being built upon. Contrived example: a method called sumtwoand_two that returns the constant four would not be nearly as useful as a method called sum that takes two integer arguments and returns their sum. Generalized solutions are important - hell, they're what software is about.

So where is the right balance? My parter Orion puts it well: "You want to build an architecture that will be somewhat flexible, but not infinitely so."

Here's an anti-example: at our first venture together, we needed some customer relationship management software which the whole company could access. (This was eight years ago, and there weren't any suitable web-based commercial or open source choices that we knew of.) We ended up writing something called the Super Nifty Node System. It didn't track companies, people, and leads, like you might expect. Instead it had a single object type - a Node - and the users could define node types and relationships between them. The idea was that we were building a system which was so flexible that we wouldn't need to do any programming when someone wanted to track something new. We thought we were solving both the original problem and many related problems, and would never need to touch the software again.

In practice, this worked out poorly. Our non-technical users didn't understand how to create new node types, and didn't really want to anyway. Things like getting the fields to go in a certain order was really difficult, leading to a lot of user frustration on entering addresses. Worse was that simple programmatic tasks we wanted to perform - like pulling out a list of email addresses for all our partner companies, or a list of all our customers who had been with us for a year or more - required SQL statements with fifteen complex joins that wrapped around the screen six times. In retrospect, we should have just made tables for companies, people, and leads - even though there would have been a fair bit of duplicated code, this being in the pre-Rails (and generally pre-framework) era.

Many people have asked us why we didn't build Heroku to support multiple frameworks and languages. Why not Heroku for Python/Django, my second favorite language/framework combo after Ruby/Rails? Why not for PHP and some of the excellent MVC frameworks that exist for it? Why limit ourselves, and our potential audience, by building to the more specific case?

Were I building Heroku earlier in my career, I might have designed it with this in mind. I might have created an abstract base class App, and from that inherited RailsApp. Or perhaps App hasone :framework and hasone :language, and then Framework and Language are abstract classes that can be inherited by Framework::Rails, Framework::Django, Language::Ruby, and Language::Python. The Django and Python classes would have sat empty for months or years as we put our energy into developing for Ruby and Rails.

And there's a cost to leaving that infrastructure laying around. Just because you're not actively developing on a particular part of the codebase doesn't mean its maintenance-free: you've got more abstractions to keep in your head, more training time for new members on the team, more specs to keep running. Even just the extra files hanging around in the code tree adds a tiny bit of overhead for your brain each time you do a directory listing or otherwise manage the code.

So don't start generalizing until you have a strong need for it. Generalizing too early is the death of many a project - almost as often as generalizing too late.

The Shaker Design Philosophy, paraphrased nicely by Josh Porter:

Don't make something unless it is both necessary and useful, but if it is both necessary and useful, don't hesitate to make it beautiful.

Tiny projects keep it new, for sure. It seems to tie in a little bit with proof-driven development and personal projects - in each case, you're escaping the mental cage of a monolithic, mature product to start something new. (Even if the "something new" is a component that's going to plug in to said mature product.)

Recently I've been dividing up the the Heroku architecture more finely. We've gone from about three repositories to over a dozen. Each time we carve off a new piece, it's a chance to freshen things up. A new revision control system, a new framework, or even just a new class hierarchy. Breaking free of the past is liberating.

There's a cycle to building software. First, you furiously write a bunch of code to get something working. It's complicated, a bit messy, a bit unfocused - but you manage to get it to work, just barely, and then you can start using it. Being used by one or more real users (even if that's just you, or you and a few others in your workgroup) then provides focus as to which features and functions are the most central.

Now the second half of the cycle begins: you start to trim it down, simplifying the code and data, trying to get the same functionality for the users with less code, fewer modules, less data complexity. At the end of this you come to a Zen-like state with your code, a minimalistic approach where you can subtract no more. From here, you now have a solid base to go back into the first cycle again, and start growing out your feature set.

Watch any good open source project to see this binge-and-purge cycle in action. Rails experienced this furious expansion of features in its early days, culminating around 1.2. Rails 2 was all about trimming down - doing more with less, focusing on core features that provide the most bang for the buck, moving everything else into external add-ons. Another one: Mozilla. After the open sourcing of Netscape, the codebase became a chaotic mess of building sprawling infrastructure components (Gecko, XUL, SpiderMonkey...). Once these had been built out, Phoenix Firebird Firefox showed up to trim, trim, trim down to a feature set that represented the core of what people really wanted out of a browser.

There's an apparent paradox here. If you don't let yourself write bad, messy, overly complex code you'll never make anything really cool or revolutionary. But bad code is unmaintainable over the long term. So if you don't take the time to factor down your bad, messy, overly complex code you'll never have a solid base to build the next set of killer features upon.

Which stage is your project in right now? And if you're on a team, do all the team members agree which stage the project is in?

Programmers generally agree that working iteratively is a good idea. But sometimes, we'll say: you just can't. This particular problem has to be done in one big bite; there's no way to break it down into smaller pieces; we just have to take the plunge.

Poppycock, I say. (Or perhaps something stronger.)

I've begun the process of using rush in Heroku's infrastructure. But this is tough: it's an unproven and immature library, barely a month old. Worse, most of the things we want it for - system-level calls - are of critical importance. Misplaced files are not forgiveable the way that, say, a UI glitch might be.

I started by selecting the absolute least important bit of code it could be used for. This component is something that happens very infrequently: rsyncing config files when launching a particular kind of instance. Further, I launch these manually (unlike some other instance types which self-scale), so there will always be someone watching the logs when it boots up.

But I took it even a step further. I kept the old method around and, in the case of exception on the newly rushified method, call it as a fallback. Here's the code:

def sync_nginx_conf
    sync_nginx_conf_via_rush
  rescue Exception => e
    Log.error "Exception on sync_nginx_conf via rush, falling back to rsync: #{e.summary}", :addendum => e.full_display
    sync_nginx_conf_via_rsync
  end

This is not foolproof; something in the rush sync could write an incorrect file without throwing an exception, for example. I'll keeping a very close eye on this method when it runs in production. Once this has proven itself in action, I'll feel confident in taking a bigger step.

I've spent much of my career straddling two worlds: systems adminstration and programming.

Sysadmining is the world of unix, processes, filesystems, daemons, networking, and servers. Sysadmins are all about getting hands-on: solving a specific instance of a problem, rather than the entire class of problems. You could call this ad-hoc and shortsighted. You could also say that it's just being pragmatic and realistic: solving a real problem, today, right now.

Programming is a world of design, theory, and elegance. Programmers are notorious for prefering solutions which are elegant over those that actually work. It's not about solving a particular instance of a problem: a good programmer wants to solve all the problems of that sort, now and forever.

People's personalities tend to lead them into one area or another. For some reason I always saw the value in both. Or actually, I never even realized that there was a divide. I thought of them as being different aspects of a more general category: how to make computers do your bidding. Only when I got my first jobs and started working on development teams it become clear that there were two camps.

There's no moral to this tale, it's just an observation.

Michael Easter puts it wonderfully:

"Clarity and brevity are often opposing forces, but there is a sweet-spot where they combine to form beauty. We spend our entire careers chasing that sweet-spot. It moves."

Specs (or tests) show that your code works in theory. A running app in production shows that your code works in practice.

Put this way, the obvious question becomes: what's the point of BDD? Working in practice is what matters.

Actually, both are equally important.

I've worked on (and created, in my less-enlightened past) lots of apps that are thrown together collections of PHP pages, ad-hoc daemons, and so on. These apps work in practice, but not in theory. They've been jury-rigged and duct-taped into working in practice, but when the first earthquake comes along they fall apart.

Another way to state this is:

• Code that works in theory is code that works by design.
• Code that works in practice only is code that works by accident.

Code that works in theory may not work in practice - only production deployment can tell you that. An app which has a large suite of running specs, but no users, is really no better than a non-speced, jury-rigged / it-works-if-you-just-don't-jiggle-it-too-hard app. Only when it has been proven in both theory and practice is your app truly sound.

In developing the early implementation of Heroku, I stumbled upon an interesting design/development technique: numerous throwaway proof-of-concepts for potential features. Heroku is an unusual product, and many of the features we envisioned were things we weren't even sure could be accomplished in a reasonable way with current browser technology. Writing these proofs let us explore the problem space without committing to a single product direction or internal architecture too early.

Each proof was something I threw together in a day or two, usually a single evening. I always made them as standalone project, showing the feature being proofed in isolation. One example I posted publicly was the proof of the javascript code editor.

You might argue that this is nothing new - they're called prototypes or tracer rounds. Although it does have much in common with prototype-driven development, proof-driven development actually comes in from the opposite side. Prototypes are supposed to be skeletal demonstrations of the entire application - as Dave Thomas puts it, to "show people how it's going to hang together." The hard implementation details are left as stubs; it's about the big picture, not details.

Proof-driven development is about the details. You prove that a specific feature can be implemented well, and that it will deliver useful value. It's a vertical slice of the application, instead of a horizontal one.

There are a few interesting effects that emerge from working this way. First, since you're starting from scratch each time, it means you're not constrained by any existing code or design. This gives a lot more room for creativity and experimentation. Normally you decide on a big-picture architecture up front, and then force features to fit into that architecture as they are built. Proof-driven development allows you to try out a bunch of cool ideas without worrying how they fit together. Then, you can select your favorites from the list, and think about what kind of high-level architecture would tie them all together.

Another effect of this approach is that it prevents you from getting to attached to any one idea or design choice. When working in an agile environment, developers often throw in experimental features somewhat on a whim to see how they'll work. That's great - often your best features come from this sort of spontaneous inspiration. But other times - probably most of the time - it adds to feature and code clutter, without adding much value. The problem is, once they are in the codebase, they tend to stay.

But since proofs are developed as standalone apps, you have to make a conscious choice to bring that code into your production app. The manual work of cut-and-paste as enough to deter you from bringing in some code unless it's really valuable. Which is a highly desirable effect, since it keeps your main codebase streamlined.

So now when our team is kicking around ideas for new components, the resolution on many points is "ok sounds cool, write a proof." The chief proponent for the idea can then go put together a standalone proof, spending at most a couple of hours, to demonstrate the value of the idea. When they're done it'll either be obvious that the new feature is really kick-ass and deserves to be included; or, perhaps it won't be too hot, in which case it can quietly be dropped without having dirtied our main code at all.

I do recommend keeping a git/svn/whatever repo for your proofs though. Even for ones you don't use, being able to come back and examine a failed technique can be useful in the future. In some cases, we've even written proofs that have not been deemed useful at the time, but later on serve as inspiration for something else.

When trying to block certain kinds of exceptions, it's tempting to write catch-all cases. Like:

string.match(/inet addr: ([\d.]+)/)[1] rescue ""

or:

File.delete('file_that_might_not_exist') rescue nil

In the first case we want no match to return nil. In the second we just want the file gone, and aren't concerned if it wasn't there initially. The problem is, both of these mask all exceptions, so we may not find out about other kinds of errors.

So we want to differentiate between user-generated errors and programatic errors. User-generated errors are a natural result of the imperfect data coming from a user or some other source outside our program. We can't force the world to always give us perfect data, so we handle these error gracefully.

Programatic errors are mistakes that we, as the developer, have made in crafting the business logic of the app. In this case, we want to hear about it as soon and as loudly as possible - so that we can find the flaw and fix it. (This is the Rule of Repair: "When you must fail, fail noisily and as soon as possible.")

So in our examples above, one option is to skip the rescue and test for the case that we're actually looking for. On the regexp match:

(m = string.match(/inet addr: ([\d.]+)/)) ? m[1] : nil

This will protect only the error we care about (no match) and let through any others.

The other option is to capture the specific exception you are looking for only. This might be the right choice on the delete file example:

begin
  File.delete('file_that_might_not_exist')
rescue Errno::ENOENT
end

Alas, not all on one line, but correctness beats succinctness when the two come in conflict. (Although you could also try FileUtils.rm('filename', :force => true) here.)

You're probably familiar with yak shaving: that is, an infinite recursion of tasks theoretically in support of your original task, but each of which is progressively less related to the original task. You know, you run into a bug with a Ruby gem that's fixed in the latest dot release, so you try to upgrade, but then you realize that you need to build a support library from source to do so, and then you remember that you hadn't get installed gcc after a from-scratch install of Leopard, but that reminds you that you left your OS disks at home, so you need to...

This, of course, can be a serious waste of time and loss of focus. Managers at big software companies are kept awake at night thinking about the fact that half their dev team might be off shaving yaks even though the deadline for their project is two weeks away. At small startups without any managers, it can simply lead to the death of the company. How many startups have spent years building some really impressive architecture or toolset with which they intend to build their product, but not the actual product? I can think of a few. In fact, my very first software job was at a company that did exactly this.

There's another side to yak-shaving, though. Sometimes you really do need to enhance underlying tools or architecture to enable your current task, and future related tasks. Whether it's rearranging your directory structure, streamlining the build process, or writing some diagnostic tools, a project can only stay agile if it has the proper infrastructure.

The counterargument to that (is that a counter-counterargument?) might be: sure, but work on the infrastructure some other time, when you're not focused on a particular task. This seems compelling, but experience tells us that it doesn't really work. You can't see what needs to be done to fix the problem unless you're in the thick of something that needs the fix.

For example: it's hard to do any kind of serious refactoring just 'cause. You can know that a codebase is due for some serious janitorial work, but without a specific goal in mind it'll be hard to focus your efforts.

Let's say you're trying to diagnose a bug. To understand what's going on you need to see it in isolation, so you need to write a test that triggers it. In trying to write a test, though, you realize that the way the object's methods are laid out can't be tested in isolation. Perhaps the methods need to be broken apart, and use parameters and return values instead of modifying object member variables directly. But to do that you may need to modify other places in the code which call the object. So here you are, changing code which has absolutely nothing to do with the bug you're trying to fix. Sound familiar? That's yak shaving, alright.

But in this case, now is the time to write that test and make those changes to that object. You could just struggle by and fix the bug without it, and then make a note to come back and write the test later, but you probably won't. Because you won't need it then. But the next time something in that same code breaks, suddenly it will all come into focus again...

So sometimes the yak DOES need a haircut. Just be mindful as you do it. Am I shaving this yak because it will be very valuable to my current goal and the project as a whole? Or am I doing it because I followed a rabbit trail of dependencies, and no one will care one way or the other whether the yak is shaved tomorrow?

In the few years I’ve been practicing atomic coding, its usefulness never ceases to astound me. I’m exaggerating only a little when I say that I’m not sure how anyone ever manages to write software any other way. I think that this single practice gives me more productivity than any other methodology, tool, or technique in my arsenal.

One of the more subtle benefits of atomic coding is the momentum it seems to produce, almost as a byproduct. For example, let’s say I take a component and plan to make my first (sub)iteration be something that I think will take me about 3 hours. Seems like a good chunk, right? It can be done in a day but is a big enough chunk to feel like I can really sink my teeth into the problem.

Except that we all know programmers are perpetual optimists, no matter how many times we’re proven wrong. An estimate of 3 hours will really turn out to be more like 4 or 5. (This is because we’re estimating for the best possible case where nothing goes wrong, which rarely happens.) Even 4 hours of work is a pretty good chunk of the workday. When you’re finished, you’re certainly not going to want to start on another 4 – 5 hour chunk.

On the other hand, if you’re using atomic coding correctly, you should be carving off subiterations of 20 – 30 minutes at a time. Which really means 45 minutes to an hour, of course, but that’s still short enough that when you finish, you’re not afraid to dive into the next chunk. Better yet, you’ve got that little serotonin rush resulting from having created a useful, working piece of functionality and committed it to the repository. You’re excited and invigorated to dive into the next chunk – which you estimate at a mere 20 or 30 minutes of work.

The momentum produced by this stream of small victories means you’ll produce more output. A 3 hour chunk turns into 5 hours and then you have to quit for the day (maybe just catch up on email or whatever till quitting time). A stream of one hour chunks means that you’ll keep working past the 5 hour mark, filling the time right up until when it’s time to leave.

A few years back, I found myself learning carpentry, as part of a large-scale Burning Man project I was working on at the time. (Carpentry, I should probably mention, is where you build things out of wood instead of bits.) I was lucky enough to have help from a friend, who was an expert on the subject. When we got to a part of the project that required drilling, he pointed out that the drill that I had wasn’t up to the task. The drill actually belonged to my girlfriend, which tells you how much of a handyman I was at the time. We went to Home Depot to remedy the situation.

Walking through the drill aisle in the tool shed, we looked at the options available. Each successive drill was a step up in price, strength, and features from the one proceeding it. Toward the far end he put his hands on a cordless Milwaukee hammer drill. “Oh hell yeah,” he said as he hefted the demo unit, “now we’re talking. 460 inch-pounds of torque.” His eyes lit with a fiery passion as he spoke. “This bad mother can drill through concrete!”

I have no idea what an inch-pound of torque is like, let alone 460 of them. Nor did I anticipate any need to drill through concrete in my future. But I bought the drill, even though a less expensive unit would have sufficed for the task at hand. Why? I recognized that look in his eyes. Even though this guy had never touched a computer a day in his life, it’s the very same look I see in the eyes of my fellow programmers as they talk about some amazing new software tool that they’re in love with.

The right tool for the job is a joy. More than just the right tool; a tool that solves the problem effortlessly, with power to spare. It makes you want to use it. It makes you look for problems you can solve with it.

This is what Rails is like for me and many others. People throw around numbers like 3x productivity gain, or 5x productivity gain. Are those numbers accurate? Well, yeah – but not for quite the reason you’d expect. Rails saves you time over PHP or JSP, sure, but that’s not the whole story. It’s that it makes the process of developing web application such a joy that you want to do, you’re happier doing it. You spend more time in a coding frenzy and less time sighing in frustration.

That’s a huge part of my motivation in creating Heroku. The one part of Rails development that has consistently been anything but joy is setting up development environments, and (worse) deployment. These aren’t programming, but they are necessary steps in the process to having a working, public application. My personal goal here is to bring the same level of joy to deployment as Rails bring to the rest of the process.

I still use my drill from time to time, and every time I do, it puts a smile on my face. I’ve not once drilled through concrete. But I’m happy with my purchase all these years later. A tool that makes me happy, makes me productive.