“Citizen usability testers” debate at the UPA conference
One of the sessions I enjoyed most at the UPA conference was the debate featuring Michael Hawley, Steve Krug, Rich Buttiglieri, Jen McGinn and Bob Thomas. Titled “Dangerous in the Wrong Hands, or Power to the People?”, it tackled the issue of “amateurs” running usability testing sessions.
Being firmly in favour of them doing so, Steve Krug had made a t-shirt especially for the occasion. Paraphrasing
Jared Spool Jakob Nielsen, it said: [See comments]
“Billions of websites. Thousands of usability professionals. You do the math.”
I had thought that the debate might closely resemble the one I’ve heard so frequently about “citizen journalists”, and I wasn’t disappointed. At one point from the floor someone said “There’s a janitor out there who wants to be your dentist”.
For me, the question of context is absolutely vital, and there is a continuum running from things that amateurs can do, to things that you need a professional for. Jen McGinn said that it would be worrying if the Government were testing IRS online applications with amateurs running the sessions. I countered from the floor that if the Government was doing that, then the fact that the moderator of the sessions had no training or experience would be the least broken bit of the process.
The most valuable contribution from the floor was
the person Tomer Sharon from Google who stood up and said that “even a usability test with a sample size of two, run by software developers, who ask leading questions, can have value”. There was some derisory laughter in the room, but he went on to say that we spend our entire careers urging everyone else in a project to have empathy with the end user, and surely that is what the developers gain from that type of session?
I very much agree with that sentiment.
The reason I like to include as many members of the project team as possible in lo-fi testing sessions is not because they make the sessions themselves more valuable, but because they present one of the rare opportunities to directly watch and talk with someone using a product you have specified, designed or coded.
The thought of software developers running user sessions prompted the question “so should we write their code?”. The answer ought to be, yes, if you are so inclined, and provided you observe the same sort of continuum.
There is no way I am going to write code that goes anywhere near The Guardian’s production infrastructure. However, being able to quickly knock-up prototypes in code to illustrate an interaction or highlight a solution to a specific issue with a flow seems like a great way to demo things to me.
Like the panel, the room in the end remained split on whether amateur usability testing was a good thing. It doesn’t take much to discover the biggest flaws with website task flows, and there is an issue over whether it is even right for professionals to be charging companies to uncover the glaringly obvious.
And everybody has to start somewhere. It was notable that the composition of the panel were all very experienced practitioners, and it might have been nicer to have included someone who was more of an amateur to get their point of view directly.