Make sure code examples work out-of-the box!

TL;DR few things teach people better than working code examples.
automatically test them!

What's the deal?

When acquiring a new library many people start by reading the documentation and then pick one of the code examples as a starting point for their own experiments. But, too often the code examples simply don't work out of the box. You may now ask: okay is see that it's an inconvenience, but it's not a problem, is it? It is! The point is: this code example could have turned a passive documentation-reader into an active user. It could have been their entry point into the library, but the door was shut. Most programmers really wish to be active. If they are kept passive they grow impatient fast.

Why don't code examples work out of the box?

Most problems can be roughly code is ...

• ... is complete but incorrect.

  The usual source is that the maintainers either don't check their examples regularly or haven't checked them at all. Part of the problem is that some documentation tools provide a zillion ways to render code but not one to actually run it.

• ... is correct but incomplete or abstract.

  I generally agree that code examples should be as short as possible, but not shorter. A common practice is to leave away the code which sets up the objects used in the example or simply using placeholders. This indeed makes the code shorter. It however makes the time longer somebody has to look at the code to understand it, because it takes more time to figure out the correct preconditions. Please note further that "abstract example" is an oxymoron.

What can be done about it?

A first approach to achieve working code examples is to test them automatically. Many documentation tools already have support for this and others can be easily extended. See the example section for different approaches on that! My usual approach can be outlined as following:

• Incorrect code can usually simply fixed like a failing spec.

• Most abstract or incomplete examples can be fixed by adding two lines of initialization code.

• If that doesn't work. Think yourself: to whom is the documented feature relevant?

  If it's relevant for the average user, then most likely your API design is broken, because it obviously makes it too hard to archieve average things. Congratulations! By seeking for broken examples you have found a quirk in the API design. Fix that!

  If it's relevant for the heavy user, then put it in an external file an leave a note where it can be found. This way it doesn't clutter the documentation but it's still reacheable.

  If it's only relevant for the ultra-heavy user, then remove it! These guys won't trust the examples anyway and only the read the code.

Some examples

First an example how simple it can be to integrate validating the code examples together with the specs. This code is from one of my libraries and tests all code examples. Futhermore it transforms the comments in the example to simple rspec matchers.

An interesting approach is currently undertaken for the slim library to convert the specs into Markdown code and test them with a customized Markdown reader.

Finally relishapp.com made a business model of it! There is whole bunch of applications at relishapp.com which converted more or less their whole documentation to cucumber tests. Relishapp then tests and presents the documentation for them.