A recent Alertbox email from Jakob Nielsen included a quote about usability testing that I liked:
Many people who don’t understand statistics complain about usability studies with a small number of participants (for example, our recent test of the reading speed of tablets vs. books). The reason small studies work is that we are usually assessing huge effects. A lot of usability findings are big enough that websites almost double their business value after fixing them.
To study big effects, small samples are fine. (Especially if you don’t care about the precise size: whether your conversion rate will increase by 90% or 95% doesn’t matter – you should improve the design in either case, so all you need to know is what’s wrong with your site, not exactly how wrong it is.)
Carrying out qualitative testing, typically throughout the early stages of a project, will help us identify the big usability issues. In this case, as long as we are representing our different user groups, then the numbers of test participants is not so important.
When we need to carry out more exacting quantitative testing (often later on in the project, or as part of a comparison), then to produce results with more statistical significance, we need numbers, so more participants is good.