Tips for Getting More Nuanced Answers from User Interviews

“As soon as we decide on a technology over user needs, we fail to be user-centered.” – Caio Braga in “AI and the big why”

Showing Respect for People’s Context and Emotions

While studying for my Master’s degree in Human Centered Design & Engineering, one of the first lessons I learned was to understand people’s challenges before designing. Instead of leading with a certain technology or solution in mind, it’s important to be clear on what problem we are solving for first.

One of the biggest UX trends in the last year has been the conversational UI. Every tech company seems to be embarking on the digital Oregon Trail for the gold rush of chatbots, machine learning, and AI-powered whatevers. I wonder if they have stopped and asked if their audience really need another channel to “engage with the brand” or if the company is just trying to shoehorn this new interaction into their lives.

The great implicit myth of interactive design is that machines are harder and more important to understand than people. Still.

We might *say* we’re designing in a human-centered way, but that’s usually not true. We’re designing tech first and then adapting it to people.

— Erika Hall 🌎 (@mulegirl) March 1, 2018

When I was a UX researcher on an AI team at Capital One, Steph Hay, the VP of Design, would often ask, “What unique value does Eno provide for our customers?” (Eno is a chatbot from Capital One.) Every team working on conversational UIs should strive to answer that question for themselves. Preferably in St. Louis, if we’re continuing with the Oregon Trail metaphor.

That’s why I advocate doing user research to understand how products like chatbots and voice assistants actually fit into people’s real world, day-to-day lives. How do we make sure they are emotionally intelligent and respect people’s contexts, emotions, and social conventions?

Human-Centered Conversation Design

With conversational agents, they can be either helpful or frustrating depending on the situation people use them in. “‘Nailing it’ means different things to different people,” says Steph Hay. And she’s right — getting a conversation right means different things to different people in different contexts. So it pays to do user research to uncover how conversational UIs actually becomes relevant, helpful, and delightful.

To be human-centered is to lead with the human perspective first.

While I was at Capital One, I mostly conducted in-lab, moderated user interviews, and in this post, I want to share what I learned doing research for AI-driven products. Specifically, in defining what “nailing it” meant to customers, I had to probe deeper into people’s answers, which admittedly, I wasn’t great at first. (Everything is a WIP, right?) So this article will focus on 3 strategies I’ve found helpful in eliciting more nuanced answers from my participants.

Three Strategies to Try in Your Next User Interview

1) Phrase your questions in the past tense

Usually, interview questions start out general, but it’s easy to get stuck there if you don’t probe people for more specific responses. This happens when you hear phrases like, “I usually…” or “generally, I…” more than specific stories or details. It’s hard to decide if something actually happened or if a person is describing their idealized scenario.

For example, participants might give their answer in the present tense:

“I usually go to the grocery store after work on my way home. I hate wasting a trip or spending extra time in traffic.”

They might also answer in the past tense. In this case, they begin their response with, “I would…” or “I used to…” For example:

“I would first open my app to check my balance and then if it’s low, I wait until I get home to make the transfer.”

These generalized responses are what the book, Learning from Strangers, calls “generalized present” and “generalized past.” And surprise–as an interviewer, how you phrase your question will influence the kind of responses you’ll get. When I notice participants speaking in generalities, the first thing I do is to check my own questions to see if they are in generalized present or generalized past.

Let’s compare the two types, and then see how we might rephrase them.

Generalized present questions might take the form:

“What happens when you…”

“When do you…”

Generalized past questions might be phrased:

“When would you…”

“When have you…”

Instead, you might try tuning your questions:

“What happened when you…”

“When did you…”

You’ll know you’re on the right track when people start telling stories without you prompting or describe a specific event rather than a summary of what they’ve done. When you develop a certain level of trust with your participants, they might even reveal moments of embarrassment, shame, or inner conflict–the private details you don’t get with generalized answers.

But if you’re getting only surface-level details, you can still follow up by asking about a specific incident. Here’s a sampling of questions I found useful.

• Can you tell me a story of a time that happened, with as much details as you can remember?
• Could you walk me through the last time you did that?
• Could you tell me what happened, starting from the beginning?
• Is there a specific incident you can think of that captured what you just told me?
• What happened then?

If you need a gut check for whether you’re getting enough details, I found it helpful to categorize my questions–and the responses I’m getting–into either internal conditions (thoughts and feelings) or the external forces (people, events, and tools surrounding the incident). Depending on your research goals, you might need more of one or the other, but the framework is a quick way to take inventory of things you’re learning.

2) Develop the story when there’s a possibility that it could influence behavior

A good story has details, and sometimes, good stories could turn into great stories with help from the listener. Most of the time, participants need your help to uncover more concrete and specific details.

I used to think that user interviews were about trying to pull out relevant information from participants, but I found that they are not going to “show you the good stuff” if they feel like they’re being interrogated — or worse, if they don’t care at all. So treat the interview like a partnership instead of an extraction. Make it enjoyable!

If you feel like you will bias the participant — you won’t. You’re guiding them towards revealing a richer story. Once the participant has alluded to something that might be important to the study or your learning goals, it’s up to you to develop that incident in fuller detail. You can do this by…

Clarify what they said

Sometimes you need to check your understanding or unpack a more complex topic.

• What you mean by (a term or phrase)?
• When you say, (a term or phrase), what are you actually doing?
• How would you describe this to someone who had never experienced this before?

Fill in the details

Prompt them to paint you a fuller picture or fill in the gaps to their story. The ethnographic framework, AEIOU, is a great tool to keep in mind if you want to categorize the kinds of details you might ask about.

• Could you walk me through it with as much detail as possible?
• So you were (doing this activity, talking to this person, looking up info about _____, etc.) and (this occurred). What happened after?
• Tell me more about that.
• Can you give me an example?
• What did that look like?
• How do you do that?
• What were other people doing then?
• How did others (i.e. relevant people they mentioned already) respond to that?
• If I were watching you do this, what would I see?

Identify actors, tools, or resources used

Related to the last point, people seldom act in a vacuum. Other people — and the relationship among them — can play a part in what happened. You might ask about the social context and resources available, and how it constrained their perception of the problem and influenced their decisions.

• Was anyone there while that was happening? What were they doing?
• Did you talk to anyone about what was going on?

Probe for inner events

Inner events are what the participant heard or saw (perceptions); what they thought, believed, or decided (cognitions); and how they felt and what impulses they experienced (emotions). Inner events are also great starting points for diving into their stories since they are easier to remember. They include values, goals, hopes, and fears.

• When that was happening, what thoughts did you have?
• What were your feelings when he/she said that?
• Can you remember how you reacted, emotionally?
• Why was that important to you?
• Why does that stand out in your memory?
• Why do you think you noticed that?
• Why does that matter?
• What motivated your response?
• How did you feel about that?
• What was significant about this to you?

Make their body language more explicit

A participant’s body language and facial expressions also tell a story. A gesture, a grimace, or a shrug could indicate that there’s something more, especially if it seems to contradict what they’re saying.

• Can you say something about why this issue generated so much emotion?
• What aspects of this issue do you think prompted such strong emotions?

At the end of the day, you want to be able to visualize the scene and the relationships between people and objects. You want to understand how events developed and unfolded, and what people felt and thought while it was happening. Filling in the details will help you identify factors and needs that were especially important to the participant.

3) Unblock participants if they get stuck

It’s not uncommon to hit a lull the conversation, and you sit uncomfortably in silence in hopes that they start elaborating more on what they just told you. But sometimes a participant is really having a hard time articulating his or her thoughts. They might even run out of things to say.

If you find yourself at a dead end, a strategy I’ve used is called laddering. You can climb up and down the ladder to explore a particular part of the product, story, or thread of the conversation.

• Attribute
• Consequences
• Values

At the top, you explore the attributes of a product, environment, or any surface-level detail that the interviewee has repeated mentioned. Then a layer below are consequences arising from that attribute being present. The most abstract layer is values, which informs how the interviewee perceives the consequences. Values also tell you why a particular attribute is important to the person and the role it plays in their life.

So if you feel like the participant is struggling, ask more concrete “what”-types of questions and then step down the ladder again with specific, “why”-types of questions. Below are specific types of questions you can start asking.

Expand the context

You might bring in more context to the incident to see how it developed. You could find out factors that led up to the event or learn about the consequences of an action taken during the incident. It could also reveal other decisions they considered but didn’t follow through with it.

• How did it start? What led up to that?
• Could you go on with that? What happened next?

Ask about variations

Your product or feature might need to support different scenarios. Because you don’t know what you don’t know, it’s helpful to understand other factors and the potential impact on your designs. You could also determine if a particular scenario is recurring or an isolated event. Either way, you’ll want to follow up a general question by asking about a specific time.

• Did you always respond (or do it) this way? Was there a time when it didn’t turn out the way you thought?
• Have you always felt this way? (If no: What changed?)
• Has your approach changed over time? (If yes: How? What motivated this change?)
• Was there a difference in this case compared to last time it happened?
• Was there a difference compared to (an older time)? (If yes: Why is that? Why now?)

Review all possible influences

Hopefully, the questions you asked at the beginning of the interview had allowed the participants to bring up issues they were having. At some point, you might want to zoom in on specific variables or factors that are of special interest to you. This could help you gather user requirements (list of things this product/feature must have and do) and think through the trade offs of each item.

On specific variables:

• What about _____? Was that important to you?
• Had that influenced your thinking about this?
• What about _____? Did that matter?

On usage/consumption differences:

• Was one more desirable than the other? Why is that?
• What was the benefit of _____? Why was it important to _____?
• How did you feel when you had to _____?

On imagining the absence of something critical to the experience:

• What would happen if (some attribute or outcome) wasn’t delivered? Has that ever happened to you? (If yes: Tell me about that time.)

Conclusion

You’ll learn things during user interviews that will open your eyes but it can feel like pulling teeth sometimes. People are unexpected and surprising, and you need to be ready for whatever comes your way.

I’ve read many articles on conducting user interviews but haven’t found any that really speaks to the specific strategies you can use while you’re talking to the participant. While I’ve shared the 3 strategies that helped me, they might not all apply to every research project.

I think Steve Portigal said it best when he talked about reaching the Tipping Point (not in reference to Malcolm Gladwell) during an interview. He said that in an interview, there often comes a point when the conversation shifts from question-answer to question-story. This is a great marker to reach for in general because this is when you’ll get the richest insights.

If you don’t remember anything else: Focus on making the interview enjoyable and ask about the things they care about. The more engaged people are, the more they’ll share.

[Photo Credit: Andy Kelly on Unsplash]