July 2, 2002
MarkBernstein.org
 

Statistical significance

Christina Wodtke (whose sidebar Victor Lombardi rightly praises for its exemplary elegance) gives Jakob Nielsen a pasting over his study of Usability Guideline Compliance.

And "International Websites lag behind"... he looked at six! Six sites? What if he looked at the wrong six sites? It's like saying I looked at 15 American women and 6 foreign women and American woman are much better looking. If someone said that to you, you'd wonder what women he was looking at... and what women he missed.

The irony is that Nielsen taught the field how important statistical significance really is. His 1989 paper on "Hypertext Usability: The Matters That Matter" looked at all the usability work done on the early systems and tossed out all the insignificant results. The discard pile was awesome. Nielsen knows methodology and he knows statistics; why is he doing this?

One reason might be that his audience changed. In the 80's, Nielsen was a scholar and his audience was the research community; they needed rigor and precision. Now, Nielsen's audience is top management; perhaps doing things right frightens them.