“Citizen usability testers” debate at the UPA conference
One of the sessions I enjoyed most at the UPA conference was the debate featuring Michael Hawley, Steve Krug, Rich Buttiglieri, Jen McGinn and Bob Thomas. Titled “Dangerous in the Wrong Hands, or Power to the People?”, it tackled the issue of “amateurs” running usability testing sessions.
Being firmly in favour of them doing so, Steve Krug had made a t-shirt especially for the occasion. Paraphrasing Jared Spool Jakob Nielsen, it said: [See comments]
“Billions of websites. Thousands of usability professionals. You do the math.”
I had thought that the debate might closely resemble the one I’ve heard so frequently about “citizen journalists”, and I wasn’t disappointed. At one point from the floor someone said “There’s a janitor out there who wants to be your dentist”.
For me, the question of context is absolutely vital, and there is a continuum running from things that amateurs can do, to things that you need a professional for. Jen McGinn said that it would be worrying if the Government were testing IRS online applications with amateurs running the sessions. I countered from the floor that if the Government was doing that, then the fact that the moderator of the sessions had no training or experience would be the least broken bit of the process.
The most valuable contribution from the floor was the person Tomer Sharon from Google who stood up and said that “even a usability test with a sample size of two, run by software developers, who ask leading questions, can have value”. There was some derisory laughter in the room, but he went on to say that we spend our entire careers urging everyone else in a project to have empathy with the end user, and surely that is what the developers gain from that type of session?
I very much agree with that sentiment.
The reason I like to include as many members of the project team as possible in lo-fi testing sessions is not because they make the sessions themselves more valuable, but because they present one of the rare opportunities to directly watch and talk with someone using a product you have specified, designed or coded.
The thought of software developers running user sessions prompted the question “so should we write their code?”. The answer ought to be, yes, if you are so inclined, and provided you observe the same sort of continuum.
There is no way I am going to write code that goes anywhere near The Guardian’s production infrastructure. However, being able to quickly knock-up prototypes in code to illustrate an interaction or highlight a solution to a specific issue with a flow seems like a great way to demo things to me.
Like the panel, the room in the end remained split on whether amateur usability testing was a good thing. It doesn’t take much to discover the biggest flaws with website task flows, and there is an issue over whether it is even right for professionals to be charging companies to uncover the glaringly obvious.
And everybody has to start somewhere. It was notable that the composition of the panel were all very experienced practitioners, and it might have been nicer to have included someone who was more of an amateur to get their point of view directly.
Nice post, Martin. I agree with you completely: getting people on the team to come and watch is incredibly valuable; perhaps the *most* valuable part, and a lot of people don't seem to get that.
BTW, in the credit-where-credit-is-due category, my t-shirt paraphrased Jakob [Nielsen], not Jared. He gave a closing plenary at UPA years ago about the future of the profession which said (again, paraphrasing very loosely) that usability professionals shouldn't be doing routine usability tests. We should be doing expert reviews, creating new methods and practices and UIs, and basically thinking deep thoughts. And getting paid well for it.
The other commenter from the floor who said he'd rather have a badly-run test by an amateur than none at all was Tomer Sharon from Google.
Thanks Steve, I've corrected the misattribution above
I have no issue with 'amateurs' conducting usability testing. If this didn't happen then none of us would be doing it.
The debate seemed fairly academic in terms of an agency/ client scenario as amateurs won't ever be doing the testing - that just won't happen.
The proliferation of remote testing tools that have flooded the market have given the means to run testing to the masses. This is definitely a good thing. However what i'm seeing now is people who have taken this on giving it back to 'professionals' to make sense of the findings as they have no idea where to start.
Steve - The thought of ux professionals not testing and doing 'expert' reviews instead makes me want to hang up my boots ; )
>> Steve - The thought of ux professionals not testing and doing 'expert' reviews instead makes me want to hang up my boots ; )
Really, James? I enjoy doing them, myself. Or did you mean "the thought of *other* ux professionals doing 'expert' reviews...." [with the emphasis on the air quotes around 'expert']?
Hi Steve - No it was the thought of not doing any more research with real people and only a set of heuristics and preconceptions for company ; )
Nice write-up, Martin. For the record, I agree that we all had to start somewhere. While we didn't read our position statements, which got the panel accepted to UPA, here was mine -- it really had more to do with the experience necessary to choose the correct method, and if you only know one method, that's the one you'll use:
"Let's face it -- we all started out as usability novices. Clearly, someone had to teach us, to get us started, and for that we are grateful. We wouldn't be here otherwise. However, a little bit of knowledge can be a dangerous thing. There is an old saying that goes, "When you are a hammer, everything looks like a nail." The danger lies in trying to use only one method to fit all research needs, which is more likely to happen, if usability testing is the only method you know. We have a myriad of tools that we can use to conduct user research, each appropriate at different times in the development cycle and for gathering different types of information. Whether the person who is conducting a usability test does it for a living, is not as important as that person being advised by someone with a complete education in the field of human factors or user research. Is a screwdriver all you need to perform rocket surgery? Maybe not, if what you really need to fix that rocket is a soldering iron."