Landing : Athabascau University

Given Enough Money, All Bugs Are Shallow

Jeff Atwood (StackExchange, Discourse) in typically brilliant form on the problem with the 'enough eyes' theory. He makes a strong point, citing the Heartbleed vulnerability, that the vast majority of open source developers do not actually review code - they use it, modify it, paste it, etc, but they don't normally look deeply at it except when it manifestly doesn't work. For things like encryption, 'doesn't work' is not such a simple thing: OpenSSL has always 'worked' - it just hasn't worked well enough. In other words, for some kinds of bug, there often simply aren't enough eyes. In such cases, open source is no better and no worse than closed source.

An increasingly popular solution, and the main subject of Atwood's critique, is to offer rewards (or to give in to blackmailers) but, as he very rightly notes, this has quite the opposite effect to the one intended because it crowds out intrinsic motivation to contribute. It's exactly the problem with grades and assessment in education, as it happens. So that doesn't work either.

Atwood has some good advice for bounty programs to mitigate some of the dangers, which focuses on social capital, cooperation and paying forward. To complete the motivational triangle, I'd also suggest a bit of non-competitive challenge (people love to solve puzzles). There need to be plenty of ways for people to fix problems because it's the right thing to do (I'd say also because it's the fun thing to do), not because of extrinsic rewards. Making bounties for bug hunting is just naive gamification. Like all behaviourist operant conditioning techniques, it achieves short-term local results but, systemically, it makes things worse.


  • Viorel Tabara April 7, 2015 - 9:05pm

    Some food for thought:

    • Yes, given enough money all bugs are shallow: use Coverity:
    Tom Lane pushed:

    - Fix multiple bugs and infelicities in pg_rewind.  Bugs all spotted
      by Coverity, including wrong realloc() size request and memory
      leaks.  Cosmetic improvements by me.  The usage of the global
      variable "filemap" here is still pretty awful, but at least I got
      rid of the gratuitous aliasing in several routines (which was
      helping to annoy Coverity, as well as being a bug risk).
    • Yes, given enough eyeballs are bugs are shallow: Trusting Tor