QA: On Errors, and Why Paying for Errors Pays Off

Published on January 20, 2007 (↻ April 7, 2024), filed under Web Development and Everything Else (RSS feed for all categories).

This and many other posts are also available as a pretty, well-behaved ebook: On Web Development. And speaking of which, here’s a short treatise just about managing the quality of websites: The Little Book of Website Quality Control (updated).

A pseudo-scientific approach to improve websites and services, and that is applicable almost anywhere:

Problems With Errors

They worsen the user experience.

When a product stops working from time to time, when you notice an author not mastering their mother-tongue, when there is no clue in which format a date must be entered and error messages show up that aren’t helpful, or anything else goes wrong or works differently than expected, it’s not a great experience.
Consequentially, errors undermine your reputation.

While there doesn’t seem to be much research on this issue, there is some. A few years ago, the Stanford Persuasive Technology Lab proved that “typographical errors and broken links hurt a site’s credibility ~~[…]~~,” and recommended: “Avoid errors of all types, no matter how small they seem.” (This is one of ten Guidelines for Web Credibility.)
They’re a pain in the *** for a perfectionist.

Okay, that’s motivation rather than a problem. Errors suck.

Strategies

Dealing with errors may be simple:

Ignore them. Or:
Fix them. Invest in your own quality assurance measures (proof-reading, testing, &c.) and think about other ways how to get rid of mistakes, mistakes that will always occur and will always be there.

Beside these measures, I try to solve this problem by paying for errors people find on this website. To be more precise, I pay for errors, mistakes, and imperfections since November 2005, and I also throw in a little SEO incentive by pointing to all contributors’ websites. Because very few reward people that point out mistakes, here is why you should:

Benefits of Paying for Error Reports

It increases the probability that you get reports at all, and it increases the likelihood that these are specific.

This is a nice two-in-one. Allow me to give sources later, but as far as I know, only every tenth customer complains when something goes wrong. (Alternatively, let’s agree on “a small percentage.”) Offering some benefits does make it more likely that you receive information on problems. From my experience with meiert.com, the resonance is almost tenfold, despite additional QA activities.

By the way, this positive effect has already been described as an HCI factor: B.J. Fogg writes about “one hand washes the other” in human-computer context in his highly commendable book Persuasive Technology.
Service in return compensates for the user experience degradation caused by the error.

This is a guess I place confidence in. Errors you find and that annoy you, but that get rewarded once you drop a line while the errors get fixed, have a different, more positive impact on the overall experience. It’s clear that errors, from spelling mistakes to service breakdowns, always occur, but it’s not clear that those who point them out are appreciated like this. Remember that your website or service needs your users and customers, not necessarily vice-versa.
It makes your product better, more quickly.

Getting users aboard accelerates the process of shipping an even better site, an even better product. But it’s not only that: At the same time, you will start to think and act differently. You will take a closer look at what you do, because when you realize that errors and mistakes hurt you and your users, you take another step towards more focus on users. And finally, that’s what we strive for, a win/win for everyone.

Overall, error handling is just one aspect when it comes to better services. It’s an important one though. (So please, email me when something’s wrong here. Or add a comment.)

Update (January 22, 2007)

I feel honored to publish Donald Norman’s feedback on this:

I think your scheme works fine on small sites, but runs into complexity issues on large ones. ~~[…]~~ this might not work on a newspaper site where there are hundreds of thousands of pages, so where it isn’t even clear to whom the error report should go, and, moreover, where the people maintaining the site often have a list of far more urgent things to fix (e.g., the next story, or pages that crash, or videos that don’t spool, or…).

And when it comes to physical products, errors might require months or even years to fix because of the complexity of the product and manufacturing process.

In my experience, errors are indeed taken seriously. But spelling and typography errors on small websites are one thing: complex errors are another. ~~[…]~~

Your scheme is innovative. Whether it is needed will depend upon the site. Thus, on my personal website (www.jnd.org) readers continually write to tell me about spelling, typographical, and factual answers. I try to respond immediately, both in fixing the pages they are talking about and in sending an email of thanks. For this site, the rapid response plus the thank-you seems to be enough—as subsequent emails confirm. Other sites might need monetary encouragement. But if my site were an order of magnitude or two larger (that is with 10 or 100 times more pages), I might be unable to keep up with the suggestions. At that point, monetary rewards wouldn’t help either: the limitation would be on the site side, not on the reader side.

So, although the idea is clever, I suspect the real problems lie elsewhere: in the ability of the site owners to maintain the pages.

About Me

Jens Oliver Meiert, on September 30, 2021.

I’m Jens, and I’m an engineering lead and author. I’ve worked as a technical lead for companies like Google and as an engineering manager for companies like Miro, I’m close to W3C and WHATWG, and I write and review books for O’Reilly and Frontend Dogma.

With my current move to Spain, I’m open to a new remote frontend leadership position. Feel free to review and refer my CV or LinkedIn profile.

I love trying things, not only in web development, but also in other areas like philosophy. Here on meiert.com I share some of my views and experiences.

Comments (Closed)

On January 20, 2007, 20:59 CET, Robert Nyman said:
I agree that end users should be encouraged to point out your errors . When I, for instance, see web sites or presentations full of typos, I automatically dismiss the one behind it.

Not that perfect spelling skills is necessary, per se, but that I think they should have done proper proof-reading and there are tools out there to help them out.
On January 21, 2007, 15:52 CET, Roger Johansson said:
Interesting approach! I’m not aware of anyone else doing this.

I agree with Robert that an article containing typos or bad grammar makes me trust the actual message slightly less, so I always appreciate it when readers make me aware of mistakes in my articles.
On January 21, 2007, 20:09 CET, Jens Oliver Meiert said:
Robert, Roger—thanks for stopping by!

I believe such incentives to help, so give it a try!
On August 3, 2007, 22:10 CEST, Robert Rother said:
Actually as I read your reward for errors I read your site more carefully and spend more time on it. Your content is great and this is a very smart marketing idea. The users want to get the reward or at least attention from your side if they find an error.

And of course your site will improve…

I will think about something similar.

Best regards,

Robert
On September 19, 2007, 16:07 CEST, Roman Schechtel said:
Your approach reminds me of Donald Knuth, the creator of TeX.

In the 80s, he claimed TeX to be bug-free and offered money for every bug to be found. This bold claim plus the moneraty incentive resulted in a) so many found bugs that he had later to lower the reward and b) a very much improved code base.

It certainly worked out for TeX, so Jens, why not claim meiert.com error-free in the next article? 😉
On September 19, 2007, 17:10 CEST, Jens Oliver Meiert said:
I didn’t know that Donald Knuth did this!

Claiming anything to be “error-free” on this site is certainly too early 😉
On February 25, 2009, 15:52 CET, Italie said:
How did you manage to have the great Donald Norman’s feedback? He seems such a distant guy…
On February 26, 2009, 17:25 CET, Fliesen said:
Jens, I like your article. But I have to agree to Donald, that it will work only well on smaller sites. I will test it on my own site and maybe report later!
On March 3, 2009, 20:16 CET, Josh said:
Really interesting idea, how do you handle payments of the “rewards”? I wonder if tthis make users assume their are mistakes and your website is relatively new?
On March 14, 2009, 22:30 CET, Jay Morris said:
Great post. I cant believe im just now seeing it. Great idea and approach. I am going to to try this out on my next site/blog i put up and see cross compare against another.

Thanks again!
On March 16, 2009, 14:14 CET, Jason Thorton said:
Wonderful idea thanks for this post going to try it on one of my slacking websites. Is paying with paypal good. or you recommend something else
On April 1, 2009, 4:36 CEST, Eric said:
Wonderful idea thanks for this post going to try it on one of my slacking websites. Is paying with paypal good. or you recommend something else
Thanks.
On April 18, 2009, 18:38 CEST, France Flights said:
Proper content is the key to the success of any websites. It is really annoying to see a site with full of typos and broken links. I agree to err is human but at this level you need to maintain your professionalism and have a good team to fix these silly errors.
On May 20, 2009, 10:30 CEST, armyshop said:
It is nearly impssible to create websites - which are fastidious - without some code-mistakes. To have a perfect result you need a code specialist. A normal user cant creat a perfekt code - in my opinion.
On October 8, 2009, 20:52 CEST, Florian said:
Great idea to reward people for pointin out mistakes!! only a few pages ask for errors to be reported to them ! This IS a great thing to do, because errors do undermine a perfect appearance. Greets
On December 3, 2009, 9:32 CET, John said:
Imagine you are working on some crucial work project and an error occurs abruptly which might lead to data loss and as a result this can disorganize you and disrupt all your plans. Errors can make you lose concentration since the messages can really get annoying each time they occur. Therefore, the remedy you are offering is set to be very helpful mostly for individuals who experience this problem regularly. From your website, I can find the possible techniques I will require to fix any errors that might occur on my system. I will surely point out any mistakes I observe with your website and remedies that you can offer, you can bet on that.