I’ve pulled out some of the key points in a discussion about user research, between Jared Spool and Erika Hall on the UI20 podcast. A transcript is available on that page.
What makes for good user research?
Years ago, the approach to reporting on user testing was different. Jared Spool: “You’d come back and you’d assemble the notes into a giant report. You would write the report in passive voice, and then distribute it. Of course, the thicker you made the report, the more of a thump up it made when you dropped it on the table, which was, of course, the most impressive way to do reporting, and then nobody read it. Then you wondered why it was there.”
Of course, the purpose of user research is to inform high-quality decisions. So research must be read and acted upon. This is why user researchers now produce short, easy-to-digest reports.
Erika Hall: “The value in research is not the answer… It’s creating that culture where you’re constantly asking the same question, and you’re in a position to keep asking the question, finding what today’s answer is, and finding a way to respond to that in your work.”
What is the aim and output of research in a team?
The documentation becomes secondary. Erika Hall: “The goal is not to produce a report. The goal is to create this shared understanding so that everybody in the team knows here’s what our goal is, and we’re very clear about our goal. Here’s what our constraints and requirements are. To really think about the assumptions together and develop the shared vocabulary about here’s what we’re betting on, and here’s our evidence that those are good bets.”
Selling research
Erika Hall: ‘[One thing that] sociologists are studying right now is the fact that data doesn’t change minds… People’s minds are very good at shutting out data that they don’t want to hear…That’s a sales moment for research, when you bring a stakeholder in and you’re like, “Watch this, and see the power, and feel the change in your own mind when you see all of your assumptions blown away.” Those are really, really powerful moments.’
Erika Hall: “Research is challenging given ideas, so it’s naturally anti-authoritarian. If you’re in an authoritarian business culture, you have to work very carefully to change that.”
Asking people what they want leads to unhelpful speculation
Erika Hall: “one of the criticisms of research is that you’re asking people what they want. People will speculate, and this is something you have to be really careful of when you do research about people and their actual behaviors and habits. If you ask the question the wrong way, what you’ll hear is what people are speculating about, which might have no connection to how they actually behave.”
If you talk to people about, “Would you use this feature? What do you like? What do you want?” they’ll imagine these scenarios that may have very little relationship to what they actually do, and what they actually need, and the choices that they make if they’re using something in a real-world scenario.
When building digital products, you should test them early to make sure that what you’re building will work
Jared Spool: ‘if we developed bridges the way we develop online products, the way people seem to want to do it, we would build the bridge and then we would send a car across it. We would watch the car inevitably plummet into the depths below, and then we’d go, “Huh. Maybe there’s something wrong with the bridge.’
Erika Hall: ‘you need to know what your overall goal is, that the research is supposed to be helping you with. you say, “OK, what are our major assumptions that we’re betting on, that carry a lot of risk?”…Then you say, “OK, what questions do we need to ask? What are we trying to find out before we get down to work, or as we continuously work, to help us validate our assumptions?”‘
When you work with data, be wary of projecting assumptions and biases onto the data
Jared Spool:
In a study, a panel was interviewing for the position of police chief. There were two flavours of resume: academic-oriented, and street-oriented.
The male name was attached the to academic resume, the female one the street experience resume.
They picked the male candidate. They said “This position requires a real academic approach, so that’s why we chose the male candidate over the female. It wasn’t because they were male. It was because it was academic.”
But then the names were switched, so that the female name was on the academic resume, and the male name on the street resume.
Then the evaluators would say “This job requires someone with real-street smarts, so we chose…but we’re not biased.”‘
But it’s possible to take measures against bias: “they found that if they had the folks rate up front which is more important, street-smarts or academics, and they were to write that down before they looked at the resumes, then they were more likely to choose the person based on the actual criteria they had decided, not based on gender bias.”