QA: On Errors, and Why Paying for Errors Pays Off

Published on Jan 20, 2007 (updated Apr 7, 2024), filed under development, quality, design (feed). (Share this on Mastodon or Bluesky?)

This and many other posts are also available as a pretty, well-behaved ebook: On Web Development. And speaking of which, here’s a short treatise just about managing the quality of websites: The Little Book of Website Quality Control (updated).

A pseudo-scientific approach to improve websites and services, and that is applicable almost anywhere:

Problems With Errors

1. They worsen the user experience.
   When a product stops working from time to time, when you notice an author not mastering their mother-tongue, when there is no clue in which format a date must be entered and error messages show up that aren’t helpful, or anything else goes wrong or works differently than expected, it’s not a great experience.

2. Consequentially, errors undermine your reputation.
   While there doesn’t seem to be much research on this issue, there is some. A few years ago, the Stanford Persuasive Technology Lab proved that “typographical errors and broken links hurt a site’s credibility […],” and recommended: “Avoid errors of all types, no matter how small they seem.” (This is one of ten Guidelines for Web Credibility.)

3. They’re a pain in the ___ for a perfectionist.

   Okay, that’s motivation rather than a problem. Errors suck.

Strategies

Dealing with errors may be simple:

• Ignore them. Or:

• Fix them. Invest in your own quality assurance measures (proof-reading, testing, &c.) and think about other ways how to get rid of mistakes, mistakes that will always occur and will always be there.

Beside these measures, I try to solve this problem by paying for errors people find on this website. To be more precise, I pay for errors, mistakes, and imperfections since November 2005, and I also throw in a little SEO incentive by pointing to all contributors’ websites. Because very few reward people that point out mistakes, here is why you should:

Benefits of Paying for Error Reports

1. It increases the probability that you get reports at all, and it increases the likelihood that these are specific.
   This is a nice two-in-one. Allow me to give sources later, but as far as I know, only every tenth customer complains when something goes wrong. (Alternatively, let’s agree on “a small percentage.”) Offering some benefits does make it more likely that you receive information on problems. From my experience with meiert.com, the resonance is almost tenfold, despite additional QA activities.

   By the way, this positive effect has already been described as an HCI factor: B.J. Fogg writes about “one hand washes the other” in human-computer context in his highly commendable book Persuasive Technology.

2. Service in return compensates for the user experience degradation caused by the error.

   This is a guess I place confidence in. Errors you find and that annoy you, but that get rewarded once you drop a line while the errors get fixed, have a different, more positive impact on the overall experience. It’s clear that errors, from spelling mistakes to service breakdowns, always occur, but it’s not clear that those who point them out are appreciated like this. Remember that your website or service needs your users and customers, not necessarily vice-versa.

3. It makes your product better, more quickly.

   Getting users aboard accelerates the process of shipping an even better site, an even better product. But it’s not only that: At the same time, you will start to think and act differently. You will take a closer look at what you do, because when you realize that errors and mistakes hurt you and your users, you take another step towards more focus on users. And finally, that’s what we strive for, a win/win for everyone.

Overall, error handling is just one aspect when it comes to better services. It’s an important one though. (So please, email me when something’s wrong here. Or add a comment.)

Update (January 22, 2007)

I feel honored to publish Donald Norman’s feedback on this:

I think your scheme works fine on small sites, but runs into complexity issues on large ones. […] this might not work on a newspaper site where there are hundreds of thousands of pages, so where it isn’t even clear to whom the error report should go, and, moreover, where the people maintaining the site often have a list of far more urgent things to fix (e.g., the next story, or pages that crash, or videos that don’t spool, or…).

And when it comes to physical products, errors might require months or even years to fix because of the complexity of the product and manufacturing process.

In my experience, errors are indeed taken seriously. But spelling and typography errors on small websites are one thing: complex errors are another. […]

Your scheme is innovative. Whether it is needed will depend upon the site. Thus, on my personal website (www.jnd.org) readers continually write to tell me about spelling, typographical, and factual answers. I try to respond immediately, both in fixing the pages they are talking about and in sending an email of thanks. For this site, the rapid response plus the thank-you seems to be enough—as subsequent emails confirm. Other sites might need monetary encouragement. But if my site were an order of magnitude or two larger (that is with 10 or 100 times more pages), I might be unable to keep up with the suggestions. At that point, monetary rewards wouldn’t help either: the limitation would be on the site side, not on the reader side.

So, although the idea is clever, I suspect the real problems lie elsewhere: in the ability of the site owners to maintain the pages.

About Me

I’m Jens (long: Jens Oliver Meiert), and I’m a web developer, manager, and author. I’ve been working as a technical lead and engineering manager for companies you’ve never heard of and companies you use every day, I’m an occasional contributor to web standards (like HTML, CSS, WCAG), and I write and review books for O’Reilly and Frontend Dogma.

I love trying things, not only in web development and engineering management, but also in other areas like philosophy. Here on meiert.com I share some of my experiences and views. (I value you being critical, interpreting charitably, and giving feedback.)