Steve Krug on DIY usability testing

This article first appeared in issue 215 of .net magazine - the world's best-selling magazine for web designers and developers.

Every self-respecting web designer knows about Don’t Make Me Think, Steve Krug’s best-selling ‘common sense approach to web usability’. However, just a fraction of its readers seem to be aware of his follow up, Rocket Surgery Made Easy, the ‘do-it-yourself guide to finding and fixing usability problems’, published last year. But Krug is on a mission. He doesn’t just want to pimp the book. He’s fixated on getting everyone doing their own usability testing.

“I’ve been doing usability consulting for a long time,” Krug explains. “I’ve found the most valuable thing you can do to improve a website or app is to have the people who are building it, paying for it or marketing it watch some people trying to use it."

Curing headaches

“I really do think that the web would be a lot better if everybody was conducting their own usability testing,” he continues. “We’d run into a lot fewer headaches. It inevitably produces great insights for everybody on the team.

“It’s more effective than focus groups and surveys. They don’t give you that ‘aha’ experience
you get when you realise what you thought people would think or do isn’t what they’re actually going to think or do. It gives you a chance to see what you’ve been building with other people’s eyes and that informs your design.”

If you can’t afford to hire a professional, it’s not a problem, says Krug. “You can do your own usability tests and do them fairly well. It’s going to cost a couple of hundred dollars as opposed to $5,000-$10,000. And you can do them more frequently, which is much more valuable.”

Timing is everything

“Most companies can only afford to pay for one round of usability testing, which they’ll do near the end of the development cycle, when the thing’s almost finished,” Krug continues. “Unfortunately, that’s the worst possible time to do a test. Yes, you’re going to find problems but you’re not going to be able to do anything about them any more. Some of them are going to be deep-seated problems. If you start testing at the very beginning of your design, though, you can pretty quickly uncover those problems.”

Krug recommends scheduling usability testing for one morning a month and recruiting only three test participants. “It’s very important to get as many people on the team as possible to come and observe,” he adds. “Don’t just run the tasks and tell people about the results.

“After the testing, over lunch, you debrief. An hour later you’ll have decided on the worst usability problems that you saw and what you’re going to fix in the next month.”

Krug admits that if you’re using just three test participants the results won’t be statistically valid. “However, people who have done this have shown time and time again that if you watch three people try and use your site, you’re going to discover a great many of the most serious problems that currently exist. It just works. If you’ve ever sat and watched one of those tests from behind the twoway glass, you know that by the time you get to the fifth person, you see the same problems again and again. You’re getting diminishing returns.”

Recruiting testers

While it’s good to have some people from your target audience, Krug says you don’t have to worry about that nearly as much as you’d think, especially if you’re starting out and getting the hang of it. His mantra is to recruit loosely and grade on a curve. “Honestly, when you do your own usability testing, what you’re looking for first of all are the problems that you built into the design that anybody is going to encounter, such as a confusing interface,” he says. “It doesn’t matter whether they’re from your target audience or not. Your grandmother could try using it and she would run into those problems.”

Each session typically takes an hour. Participants are given a few tasks and think aloud while they test various sections of your site such as the checkout or newsletter subscription system, and those can be clickable wireframes, paper prototypes and HTML pages.

It’s also a great idea to test competitors, sites that are in the same line of business or that use the same kind of formatting or functionality, he believes. “I always tell people that somebody else out there has gone to the trouble of building a full-scale working prototype of a site that solves the same problem that you’re trying to solve. And they leave it lying out there! We can use it!

“So do a round that’s just testing your competitor sites. Marketing people will love it because they care more about your competitors. They assume that your stuff works fine. It also takes the pressure off if it’s your first round of testing because it’s not your stuff.”

It’s crucial that anybody who’s involved with the production of the site comes to observe the tests. “It’s a transformative experience. People watch the tests and they suddenly get it, they understand what you’ve been trying to explain to them all this time. You see people using your stuff in ways you’d never have thought of. You realise not everybody is like you. It’s so valuable.”

Pizza works

To get them to attend, Krug recommends bribing them with high priced snacks. “Get snacks that they’re not going to get at any other meeting,” he laughs. “If you have good snacks, you’ll develop a reputation for good snacks and people will come. The same thing is true for lunch: get the good pizza! It actually works. And make sure it’s on the same day every month, so they’ll know when it’s coming up. Try and choose a day when people aren’t in their monthly crunch, and send them reminders.”

At the end of each session Krug recommends you get people to write down the three most
significant usability issues they’ve observed. Then, over lunch, ask them to decide on their overall top three and make a list of the most serious problems in descending order. “You need to focus ruthlessly on the most serious problems,” Krug explains. “You can’t get distracted by low hanging fruit and things that you’d like to fix.”

Next, work your way down the list and write down what resources are required to fix each problem. You do this until you reach the point where you have no more resources available. “You might stop with the first problem, as you’ll only have the resources to fix this one in that month. It’s very important that you stick to that.”

Quick fixes

Don’t be tempted to leave issues for the next redesign, Krug advises. Fix the most serious problems immediately. “I’m trying to get people to think in terms of tweaking rather than redesigning,” he explains. “Do as little as possible. You don’t have to really fix it.

“If you can come up with a tweak that makes it not be a serious problem, you’ve done just as
well and tweaks take much less work. It will leave you with a lot more resources and go much further down the list. Tweaking can get the job done right away.”

Remote testing

Krug believes remote user testing can be equally beneficial. “Now that you’ve got people with highspeed connections, you can use screensharing and voice-over-IP,” he points out. “For observers it doesn’t make that much difference whether they’re watching it over screensharing or sitting in another room down the hall. It also makes recruiting easier because you don’t have to find people close to your offices. The only recommendation I make in the book is that I don’t think you should try facilitating tests remotely until they’ve done a couple in person. You can’t tell straight away when somebody is stuck if you’re not there sitting with them.”

So, it’s straightforward, doesn’t take much time and money and you get to eat some pretty cool snacks. What are you waiting for?

To watch a demo usability test, download test scripts and checklists and to sign up for Steve Krug’s workshops, head to www.sensible.com.