Note: To protect Covered By Sage, Inc., all information specific to their business model and product strategy have been omitted from this article.

There are already so many articles and books on the topic of user research, but none are specific to internationally-distributed insurance teams. Rather than expect others to go through the process I did for applying user research principles to Sage implementation, I created this documentation.

Introduction

Knowledge management for us will always be a challenge, because the P&C insurance industry is so incredibly multi-faceted. With Sage's design and engineering team in Nepal building an extremely business logic-heavy platform, the need to get up to speed with industry logic is massive. Aside from reading up on the industry, one of the best ways to cover knowledge gaps is through direct interaction with agents.

User interviews is the best way to discover unknown unknowns on requirements and personas. The golden state for a user researcher in a B2C context is to come to an understanding of your user’s needs and wants that is greater than their own. For Sage’s purposes, none of us individually will ever come to understand all of our agent users’ needs more than they do for themselves. There is too much to learn in insurance – and it’s always evolving. Well-executed research with well-documented insights will always be a key priority for us.

I have spent much of my time over the last few months trail blazing a path for the team to be involved in our user research. I put together a pool of a couple dozen P&C insurance agents from a variety of niche markets - mostly focused on commercial lines - from whom we could buy consulting time.

Why do user interviews?

There are three specific areas where integrating user interviews routinely in our product development process will help us. Conducting user interviews will:

1. Surface product requirements. We simply do not know all the requirements for our products that we need to put in our backlog and prioritize.
2. Increase empathy for our users. Time spent with agents sharpens your intuition on their needs and desires more than anything else. Hopefully everyone in the company will at one time or another be involved in our user research.
3. Increase team product IQ. Creating the best-designed products in the industry will require a steady flow of at least a few user interviews and usability testing every month, involving engineers, designers, and non-product teammates. Everyone's role plays a part in delivering the end user's experience.
   

Notes on Insurance Agents

• Agents have a very low bar with design - insurance software design sucks.
• Agents have a very strong ability to praise - they're paid people pleasers.
• To achieve a high design standard with our software, you have to be very perceptive to catch important cues. You have to be cognizant about this in what questions you ask.
• Agents with a stake in Sage (or agents from NYC) give the best feedback. They're more straightforward and willing to tell it like it is.
  

Stages of User Interviews

The stages of conducting user interviews for a team are:

1. Planning
2. Recruitment
3. Facilitation
4. Transcription
5. Analysis
6. Synthesis (with other documentation)

Below are guiding principles for Sage in each of these stages:

1. Planning

• Give yourself planning time and division of responsibility. They're needed to get the most out of user interviews.
• Choose a companion teammate(s) for the research. Without a second researcher involved, all insights will reside in the brain of the lone researcher which will not be conducive to design collaboration or for fair analysis of the work that was complete.
• Set up your companion teammate for success. Assistants need clear expectations entirely spelled out for them. Have them review the principles outlined in this document as often as necessary.
• Prepare good quality questions. Some questions are better asked in surveys to people that may not know about Sage.
• Prepare an agenda. Here are the ingredients and guiding principles of a solid agenda:
  1. Include an explanation of the purpose of interview.
  2. Give user research subject an opportunity to ask questions they may have so you can clear them up before they potentially turn into an unhelpful tangent during the interview.
  3. Prepare a logical sequence of interview items.
• Maintain a list of topics you want to conduct user research on. Usually there is a lot more you and your team want to learn than you would want to plan for any single interview. Spread the items across multiple interviews and cross the topics off the list once you've gained the insights you wanted to receive. This way you don't waste valuable time in interviews repeating items you've already gained insights on and you continually focus on uncovering answers and new unknowns.
• Keep scheduling more interviews until you start hearing the same things. For user interviews, the number you should conduct is however many is necessary to start hearing the same responses. Once you have determined what people in your target segment are saying, you can walk away with the insights you've gathered. If you want statistically significant data, administer a survey taking the insights gained from the user interviews into account.
• Once you start hearing the same things, administer surveys. Once you know enough about a topic and its nuances, it's time to break it down into specific questions for a survey to get statistical data.

Usability Test-Specific Planning

• Prepare an agenda:
  • Include an explanation of the purpose of interview.
  • Give user research subject an opportunity to ask questions they may have so you can clear them up before they potentially turn into an unhelpful tangent during the interview.
  • Prepare a logical sequence of interview items.
  • Include a product link with whatever credentials may be necessary.
  • If it’s a low-fi design, set the expectation to focus on functional aspects in interview so they don't focus on visual-oriented feedback and offer unqualified design advice.
  • Explain how the prototyping tool works or else they will be confused why it's not functioning software.
• Prepare specific user scenarios. The purpose of these is to more objectively measure how usable the product is for various actions. Collaborate with UI designers to compile this list.
• Be fully aware of the latest design iteration you will be testing. If you’re not clear on all the updates before the interview, you won’t be able to address them effectively.
• Make sure all the prototype links are set and point to the right pages. If they’re not, it will derail the interview for at least a bit, but possibly completely disrupt the flow you need for the user to experience to offer the intended insights you seek.
• Address the business logic issues wherever you can during the design process. Ideally you can see progress updates during the design process so that important tweaks to details can be addressed before the interviews. Even if it has little visual design consequence, wrong wording or unrealistic data could cause an unnecessary distraction on business logic from product design. These often trigger 5-minute tangents during your usability tests and that may be 20% of the interview time. Effectively communicate this to the product designers who often do not concern themselves with the wording of messages or how realistic the numbers are that they put in dummy data fields, for example.
• Do not give the interviewee a maze to navigate. In prototypes, do not force a user flow on them through selective links. You take away their flexibility and autonomy to testing out the use cases they want to explore.

2. Recruitment

• Create a basic recruitment strategy. Here are three key questions:
  1. Where will you find your user segment?
  2. How will you catch their attention?
  3. How will you incentivize them sufficiently to offer you their time and attention?
  4. How will you screen your prospective interviewees?
• A/B test variations of your message. Increase response rates through optimizations based on feedback. Sage tip: The most common reason highly-competent agents agreed to offer time for free for was to be more aware of cutting edge insurtech innovation.
• Schedule a 10 minute intro call. Accomplish the following with your intro calls:
  1. Clarify the purpose of the research in the beginning. For example, "Your input will help us in delivering the best experiences to our users that we can. Also, interviews like this help us a lot in mitigating significant risk. It's an extremely expensive mistake to take bad design and crystalize it into code."
  2. Filter out people that don’t fit your target user criteria. For agent interviews, for example, you can waste a lot of time lining up someone who hasn't sold insurance in a long time, is primarily an account manager, etc.
  3. Qualify the individual for a productive interview by establishing what they need to know about Sage. Agents can give much better feedback if they understand what Sage's business strategy is.
• Clarify in the intro call that we are building an agency, not another subscription-based insurtech product. Sometimes agents (principal agents mostly) want to spend time with you because they're looking for a solution for their agency.
• Understand who the individual is so that you can put their feedback into their unique context. Listening to everything interviewees say is lazy and dangerous. You need to factor in where they're coming from when you hear what they're saying. Understanding why certain individuals say certain things is critical to further user segmentation – which is critical to business and product strategy.
• Maximize recruitment effort by building a user pool. Some agents give twice as much good feedback and insights as others. Some agents are not worth setting up another usability test or user interview with. For those that are worth further user research, document information about them in a well-organized spreadsheet that is usable for other teammates.
• For individuals that you leave out of your user pool, keep the door open. Leave these agents on a positive note and keep the door open for any future survey possibilities, etc.

3. Facilitation

• Minimize no-shows by always sending reminders. The first thing you can do to increase the effectiveness of your interviews is to increase they probability they even happen. You must send reminder emails or text messages to people within 24 hours of their scheduled interviews. Otherwise almost half of them won't show up. Even with reminders, about 25% of them don't show up.
• Ensure basic effectiveness of the interview. List of the basics:
  1. Call in from a non-distracting place. Avoid public places. If you must tune into an interview from the Kathmandu office or anywhere with occasional distracting noises, mute the mic.
  2. Use the simplest tools possible in interviews. Some agents find very simple technology steps very prohibitive. Some interviews start 20 minutes late or have to be rescheduled when something as simple as forgetting a password happens. Skype on Mac and Windows are different. Google hangouts was the tool we found to work with the highest percentage of people.
  3. Have your agenda in front of you for reference. Don’t treat it like a script, though.
  4. Train (or remind) the interviewee on the think-aloud protocol. Because you can’t always detect what they’re thinking getting them to think aloud is important. Because you don't see what part exactly of the UI the user is looking at during remote usability tests, getting the interviewee to talk through what they’re doing in the UI is a necessity. Engage the user if necessary to speak about specifics that you can't visually track.
  5. Don't be afraid of pauses. If the user is processing something, give them uninterrupted opportunities to think through it.
  6. Help when necessary. Lightly respond when you don't exactly know what to take away from their pause or if you can tell that they are lost.
  7. Ask what certain words mean to agents. Information design is very important for Sage. Poor wording will get some agents hung up on them for a while and create long tangents in the interview, so note the examples of user confusion due to our wording choices.
• Qualify the interviewee to offer good feedback. Here are several ways to ensure this:
  • Start at a logical point. If the user needs to understand one component of the system in order to have the necessary context for others, you must start at component 1, even if there's no user research needed for it.
  • Don't context switch too much. As an outsider, the interviewee does not have the extensively-developed mental models you do. You may be able to talk about one piece and then ten minutes later about another. This context switch will affect them more than it will affect you. Explore one or two modules of of the system.
  • Don’t ask specific questions about an imaginary version of the product. Asking interviewees during usability tests to visualize a different design than what is in front of them and to offer feedback on the alternative design that they're visualizing is not effective or reliable. Dig deep into the product right in front of you.
• Produce pure, unbiased insights in all ways feasible. Here are several ways to do this:
  • Don't excessively tour-guide them. Point them in directions only when necessary. Too much information corrals user testing subjects into less insightful feedback. Give them just enough information (in a very clear way) to allow them to give you context-appropriate insights that either validate or invalidate what we think we know.
  • Focus on the Interviewee:
  • Re-orient conversation to them. When agent points a comment or question to you about you, address it politely and briefly but refocus the dialog back on them.
  • Do not speak unnecessarily. The more they learn about you the more their thinking and dialog will orient itself around you. Listen 70-90% of the time.
  • Reduce your identity. Don't wear anything or have anything in view of the camera that clearly shows what tribe you're from. It may invoke distracting feelings or biases in the user subject.
  • Maximize unplanned takeaways from every interview. Here are a few pivot strategies to make the most of unplanned opportunities:
    • Shift to strategy questions for industry VIPs. Sometimes you can score the attention of highly-important and -experienced insurance industry experts. In these scenarios, nitty gritty user testing is not the best use of your or their time. Go into product strategy questions. In these scenarios you will have to go into tour-guide mode to frame product strategy-specific questions. Insurance veterans and executives can add insight to competitive research and offer valuable requirements our product roadmap. If there are business logic questions Rashik, Beth, or Aaron also have they can be brought up here.
    • Shift to market research for specialists. If you're in the middle of a usability test and learn that the agent you're interviewing generates a significant portion of their business from a micromarket or industry niche, dig into that. We are strategically interested in navigating these in real time (not from dated, published material). If you’re hearing what we already know, you can shift gears back to the usability testing.
• Optimize insights through expertise. Reduce subjectivity in our design decisions. The more you can research and learn about design principles and UI design the better you'll be able to surface specific, actionable insights for our UI designers.

4. Transcription

• Understand the reasoning behind transcription in order to produce the most useful notes. The whole purpose of transcription is so that the research conducted is usable by the broader team as well as the interviewer themselves after the details fade in memory.
• Video or audio record everything. In addition to text transcription, record the user interview always so that transcriptions can be filled in with details that fell through the cracks and so that key moments can be captured where text transcription won't give it justice.
• Maintain clear and consistent documentation standards. We have room to improve on our interview notes documentation format to increase understanding of teammates and highlight actionable key insights.

5. Analysis & Synthesis

Work with me on this.

Must Reads for Our UX Researchers

The book UX Research is a must read for teammates primarily working in a UX research role. Interviewing Users is also a very good skill-building book for user interviewers.

UX Research
by Brad Nunnally and David Farkas

Interviewing Users
by Steve Portigal