Adam @ Heroku
a tornado of razorblades

The End of Bugs?

Posted by Adam Wiggins on July 06, 2008 at 11:55 PM

Although I've been a believer in TDD/BDD for quite a while, rush was the first time I started a project and said, ok, THIS time I'm going to get really serious about it. My very first commit was a bunch of empty specs, and since then, I don't think I've committed any new feature or even a bugfix without an accompanying spec.

A few weeks into the project, I had made really substantial progress and had quite a lot of functionality. One day, while using it to access a remote machine to do some file operations, something very surprising happened: I typed a certain command and it didn't behave the way I expected it to.

I figured I must have a typo or something, but double-checking it, I realized that, no: my command was correct, it was the program which was behaving incorrectly. I had found, it seemed, a bug.

I felt a sudden sense of disorientation and panic. The program wasn't behaving as expected? What should I do? Log things? Run a debugger? Just squint at the code for a while and see if I could spot the problem? These methods seemed so...crude.

And yet, I realized, these techniques are the very ones I've been using my whole career, that I use every single day to get work done. I don't give them a second thought. Yet, after two weeks of doing pure BDD, the idea of spending time dealing with a bug seemed foreign and painful.

I ended up tracking down the problem and discovered that, despite my nearly 100% code coverage (as reported by RCov), I had a fork in a single-line conditional that was not speced. Upon realizing this, another feeling washed over me: a sense of mistrust in the project's codebase, because a fragment of one line of code did not have spec coverage. I wrote the spec, confirmed that it failed / exposed the bug, and then fixed the bug itself ten seconds later. Relief flooded over me: things were right with the world again.

Nearly all developers, myself included, spend most of our time in that state: not quite trusting that the code all works. I only had the chance to realize how stressful and unpleasant this state is because I lived outside it for a little while.

This experience also told me something else: after several weeks of intense development, this was the first time I had encountered a bug. I feel absolutely certain that that's never happened to me before.

You could argue that rush was not then (and is still not now) a very big or complicated program, but I don't think that's significant. First, by this time I was already using it to control a remote server (i.e., it was communicating commands over the network via rushd); that's the sort of thing that is highly prone to bugs. But more importantly, even very short and simple programs often have bugs lurking in them.

My whole life I've assumed that bugs are a given in software, and that trying to eliminate them completely is a waste of time. And I think I was correct, for the traditional, non-BDD approach. ("Code-driven development," I guess?)

But it may be that a very disciplined approach to BDD means the opportunity to have truly bug-free software. Just imagine what it would be like to have all the time you spend debugging code go into writing it instead. Granted, a lot of that goes into writing specs; but for me, writing specs is far more fun than debugging. It's programming, not sleuthing.

Bug-free software is a very bold claim, though it does require a somewhat narrow definition of the word "bug." A bug is a situation where the code has been specified to behave one way, and instead it behaves another. It does not include things that users or developers may like the software to do, but that it is not specified to do currently. It also doesn't include external depedencies (libraries, web services..) behaving in an unexpected manner - good software should try to recover gracefully from failures in external services, but doing so is a type of feature.

Look back at that definition of bug again: "...the code has been specified to behave one way..." BDD is the very act of writing executable specifications. In non-BDD, there is no specification; so by definition, software written that way can be said to either have an infinite number of bugs, or perhaps just no functionality that you can rely on.

Tags: bdd
Hierarchy: previous, next

Comments

There are 14 comments on this post. Post yours →

Harm

Splendid post! Today I refactored a large piece of code. Having specs (no complete coverage, but still) gave me a great sense of confidence in my code unknown to me before doing BDD (TDD). I'll never go back.

Although I've been a TDD supporter for years, I hadn't yet learned BDD. I've been thinking its something I should do, and now I'm convinced. Thanks.

Absolutely! This is why debugger support is a bad thing. Development practices which are very close to bug-proof exist.

Thanks guys. I thought this post would be more controversial - glad to hear I'm not alone on this. :)

Now, playing devil's advocate to myself here, specs can only extend so far - i.e. to the edges of the component they belong to. In a service architecture there's still the interaction between components. I'm starting to develop some techniques to cover that, but it's a very different sort of thing. No doubt I'll post about that stuff at some point.

Paul

Interessant ideas.

Ohh and THANK YOU for havinf formulating this : "Nearly all developers, myself included, spend most of our time in that state: not quite trusting that the code all works.". That's SO TRUE (IMHO too, at least)!

Stephen Waits

Who was testing your app all that time?

ngvrnd

If a bug is a variance from spec, and the behavior you observed was from code which wasn't spec'd, was it a bug? Maybe an additional definition is needed; any code which isn't covered by the spec is also a bug.

"Nearly all developers, myself included, spend most of our time in that state: not quite trusting that the code all works"

Fantastic quote. In ten years of development, no matter how amazing the code, or tests, this is our state of mind. I'm always amazed when things work the first time.

Harlan Mills at IBM worked on useful methods of software engineering called "cleanroom software engineering". By the mid-70s they basically worked out how to build software that is provably correct by design at no additional cost beyond conventional software development methods. Comparing that to the present day methods of programming, we've regressed considerably in the last 30 years across most of the industry. More info can be found here if you're interested: http://www.sei.cmu.edu/str/descriptions/cleanroom.html

marktucks

Is RSpec the de facto test framework i.e. what everyone is using, like Git?

Also, are there any really comprehensive test tutorials, including things like what to test, how to test, code coverage etc?

Thanks.

@marktucks - There are a lot of diverse opinions on the best way to write specs/tests, what framework to use, etc. I blog about this quite often, as you can tell: http://adam.blog.heroku.com/past/tags/bdd Reading through these should answer some of your questions, at least from one person's perspective.

Some other Ruby-centric resources on the subject I recommend:

http://blog.hasmanythrough.com/2008/6/1/the-great-test-framework-dance-off http://blog.jayfields.com/ http://blog.davidchelimsky.net/

marktucks

Great. Thanks, Adam.

This is why I wish Heckle was a little more robust/well-used. It seems like it would be a great boon to the test-driven workflow, if only it weren't so finicky.

http://ruby.sadi.st/Heckle.html

Post a comment

Required fields in bold.