Do as they do, not as they say
George Olsen has some thoughtful commentary on the latest Alertbox, which is about watching what users do rather than listening to what users say. He points out that what Jakob is really taking to task are the surveys, questionaires, and other marketing-type approaches which simply test preference rather than user behavior. I couldn’t agree more.
Have you ever seen a marketing person conduct a usability test? Have you ever seen someone fishing for data? I have, and it’s not a pretty sight. I’m not against using anyone but a “usability specialist” to run a test - anyone with a couple hours of training can do a decent job - but it’s so easy to go fishing, to lead the participant, to come to your own conclusions. As George mentions, it’s fine to use well-designed, quantitative marketing approaches to get at attitudes, but asking eight participants which design they prefer doesn’t qualify as a usability test.