The pitfalls of continuous discovery research (and how to avoid them)
Go with the flow.
Continuous discovery is all the rage—but what is it, why is it popular, and where does it fall down?
Though the concept of continuous discovery research isn’t new in the world of product development, it’s safe to say it’s having a moment.
Anecdotally, I’ve been inundated with requests for “talking to a user every week” or “just getting feedback on an ongoing basis” from product team members. I’ve also heard similar sentiments from other researchers at different companies.
But why? And is it a good thing?
What is continuous discovery research, and why is it so popular right now?
Continuous discovery research is the consistent, regular seeking of user insights and feedback using frequent, short research methodologies that are analyzed over time.
In organizations with less UX maturity or less experience working with proper user research methodology, the requests are often articulated as though they’re casual. For example, a product manager might ask to interview (or the dreaded “talk to”) a power user once a week, or perhaps a designer will want to triangulate user behavior data with support tickets with reports created weekly.
In general, continuous discovery is attractive to stakeholders. It’s a consistent flow of feedback on what they’re building, and, from their perspective, it involves getting insights within shorter timelines than with project-based research.
Based on personal experience and anecdotes from colleagues, I’ve also felt that it’s appealing because it’s perceived as casual. Stakeholders don’t always feel they need specific research skills to do it themselves or help speed things up.
As for the current uptick in requests for continuous discovery research, it’s impossible to know why it’s happening, but the current economic climate offers a reasonable hypothesis.
Let’s take the tech industry as an example. A company aggressively reducing expenses may see user research as a nice-to-have more than a mission-critical activity (though many of us would probably disagree).
As a result, many companies reduce their research teams and budgets during the downturn. This creates a situation where either stakeholders are tasked with doing research without dedicated researchers or a much smaller team with a limited budget is trying to answer everyone’s research needs.
In both cases, continuous discovery can be attractive because its rhythm is predictable, expenses can easily be limited, and insights are constantly coming in.
Fair enough—continuous discovery research is a potent piece of a comprehensive research operation that offers real, timely value. That being said, there are potential pitfalls when it takes precedence over or completely replaces other forms of user research.
Potential pitfalls to avoid when prioritizing continuous discovery research
There is a misconception that continuous discovery research doesn’t require actual analysis methodology and that every user touch point offers actionable insights; this can lead to misleading results and poor decision-making.
Though continuous discovery research doesn’t necessarily have a specific endpoint, it’s important to ensure stakeholders understand the need for scrupulous analysis. Some stakeholders may have experience with this methodology, but for those who don’t, you want to ensure they understand the proper way to analyze continuous discovery data.
If you, as a researcher, will be conducting the research, you can ensure you’re on the same page about the frequency of insights output and how you’ll prioritize different insights and action items over time.
If stakeholders themselves will be doing the research, you should offer some education about the basic best practices when it comes to continuous discovery. Even if your stakeholders have experience reviewing survey data, conducting interviews, and so on, they could use a quick lesson about how and when to analyze this type of research.
Products change over time, and research without a specific timeline may generate data from users experiencing your product differently.
When conducting continuous discovery, you or your stakeholders will uncover a lot of sentiments and pain points. In most organizations, product iterations happen over time. So a crucial part of analyzing the data is looking at it in the context of product updates and learning how they have (or have not) affected what you’re learning from users.
You may have a product pain point disappear, become less prominent, or have a new one emerge when introducing a new feature. Someone must carefully analyze continuous discovery data needs in tandem with product updates. If it is, the ability to look at how user feedback and insights evolve is one of the main perks of continuous discovery!
Continuous discovery research usually prioritizes current users. If it’s the only type of research being done, key strategic research opportunities may be missed.
On my team at Lightricks, whether a stakeholder asks for it explicitly or not, we generally take every opportunity possible to include research participants both from our current and outside our user base. For example, you can gather highly actionable insights from people who use your competitors’ products and try yours for the first time or people who are members of your target audience but haven’t used your product yet.
Continuous discovery research in many organizations needs to address these types of insights. When whoever is doing research is only looking at current users, there are likely some important, strategic perspectives left on the table. Make sure you advocate for time and resources outside of continuous discovery within your user base so that your product team doesn’t miss out on potentially impactful perspectives.
When you fall into a continuous research rut—asking the same general questions repeatedly—you may leave nitty gritty, detailed user-based insights unexplored.
Since continuous discovery tends to happen at a typical cadence without a defined endpoint, sometimes the upkeep of things like an interview guide or the metrics you’re triangulating stays the same for too long.
Goals, perspectives, and internal product initiatives change pretty consistently within most organizations, and continuous discovery requires a certain level of upkeep to ensure that you’re not failing to investigate what’s important now.
So what’s a good way to meet the demand for continuous discovery research?
The demand for continuous discovery shows a genuine desire to learn more about users and to incorporate their perspectives into product development. When done correctly, it yields tremendously powerful product insights. For these reasons, I’m an advocate for making it happen. At Lightricks, I’ve had my own failures and wins when meeting continuous discovery demands, and this is my go-to list of what I’ve found to work well.
- Don’t say no to continuous discovery, but advocate for additional strategic research. The best way to do this is to understand the crux of stakeholder questions and propose corresponding research. For example, suppose a product manager wants to interview power users of a certain feature set on an ongoing basis. In that case, you may suggest a more in-depth parallel project that looks at users using similar feature sets to your competitors.
- Create continuous research sprints so that everyone is aligned around timelines and expectations. I’ve found that using a sprint framework for continuous discovery uses language that stakeholders are familiar with and does a good job of setting natural expectations. For example, my team did monthly usability sprints for some of our products at Lightricks. This meant stakeholders knew to expect monthly deliverables with insights and usability recommendations every month. The month-long framework meant we looked at each data set within the context of the product changes we implemented that particular month. You can use a sprint framework if you’re conducting the research or suggest it to stakeholders executing continuous discovery independently.
- Set periodic reminders to analyze data sets, update interview guides, and review which metrics you’re looking at. Whether you or your stakeholders are conducting continuous discovery research, set periodic reminders to review your materials. For example, if you’re interviewing users, are you asking all the questions relevant to your company’s work right now?
Continuous discovery research generates insights with the added benefit of looking at user perspectives over time, and it’s a great addition to any research operation, big or small. Though economic or other factors may be influencing its current popularity, it’s unlikely to go anywhere anytime soon, and it’s well worth the time of any researcher or product professional to think about how to use it at their organization according to best practices. Over time, your stakeholders might even stop calling user interviews’ talking to users’ and warm up to the rigor required to do it right.