Repeated and reproducible systems research

Daniel Lemire (@lemire) recently posted on the Mythical Reproducibility of Science, noting that sharing code also makes it easier to spread your ideas:

The reproducibility that matters is getting people to use your ideas. Merely proving you are honest falls short of your potential!

He’s also written on this in the past with statistics on downloads of his software.

Here are some more writings on reproducibility:

• The case for repeated research in operating systems from ACM SIGOPS 2004.
• The Many Faces of Systems Research—And How to Evaluate Them from HotOS 2005.
• Tools for repeatable research, that I wrote four (!) years ago, has some pointers.

At PDOS, Frans Kaashoek and Robert Morris definitely encourage us to build real systems and make the system available. I definitely like this approach and never found that publishing the full source to Chord, including our work-in-progress/submission, caused any problems. It has also meant that a lot of people still play with Chord, even if I no longer actively maintain it.