Investing in insights infrastructure for product teams

I started my career in research over a decade ago. In some ways, the profession looks the same, and in a lot of ways, it looks very different.

It’s those differences that I want to talk about specifically—differences in my own practice and the practice of the companies I’ve worked with.

Like many of you, I started my career as an IC researcher, first at Facebook and then at Slack.

At Slack, I quickly switched to leading research operations. I spent about half my time making sure anyone who was in research or data science had all the tools and support they needed to do their job. The other half of my role was making sure that anyone else at Slack who did anything that looked like research or data science was well-supported.

Over the last four years, I’ve functioned a lot more like a personal trainer for product organizations. I left Slack in May 2020 and started Yet Another Studio. The goal was to work with startups from their first research project through to their first research hire and beyond. I also started building courses with ReForge and had the chance to teach over 1,000 people about decision-first research.

I currently partner with lots of founders and leaders in product and growth from dozens of organizations ranging from two-person companies who don’t know how to talk to customers at all all the way up to Figma, Dropbox, Replit, Capital One, and Asurion.

The pitfalls of focussing on methods over outcomes

So the following sums up almost every research project I’ve ever done: I used [methods] to gather [evidence] so that we were more confident that [product/intervention] would drive [outcome].

Researchers overwhelmingly focus on the first six words in this sentence: I used [methods] to gather [evidence]. We love fighting about methods, how many people to interview, the right way to create personas, and whether we should do segmentation.

All of those things are important. The craft matters. But, in my experience, it skews away from the why in our work. The why is the thing our partners in product, growth, engineering, sales, and leadership care more about.

This sentence also helps us grapple with an often uncomfortable truth: there are too many outcomes, products, and interventions for us to touch. We, as researchers, research teams, and research organizations, cannot touch or evaluate everything that we are building.

And this has led me to something that I’ve shared with every company that I work with, and really, a truth that underlies a lot of my work: we need to build organizations that learn, not just teams that do.

I think, as researchers, we need to become much more comfortable giving away our legos and stop trying to be oracles. We need to make it safe for everyone to meaningfully participate in learning.

Lessons learned from our data science partners

I think we have a lot to learn from our partners in data science who have gone through a similar transformation. For many of us, limited data literacy meant we relied on them closely to do the heavy lifting of analytics. They were the ones who went over to the well and pulled up that data water, bringing it back to us so we could drink.

And then we got tools like Tableau, Amplitude, Grafana, and Looker, and the data scientist role changed. They no longer had to go to the well, gather data, and bring it to us. Instead, they were responsible for piping and plumbing in the taps, making sure that data flowed everywhere throughout the company and that it was safe to drink.

In my opinion, this change was necessary. Data scientists shifted their role so they could continue to make the highest leverage contributions to the company. Their job wasn’t to go and do the analysis that other people could do—it was to make data safe to play with.

Insights infrastructure: the highest leverage way to contribute

We very often ask this question: what is the highest leverage thing to work on? But the question that I think more of us need to be asking—especially as things are changing—is this: what is the highest leverage way to contribute?

What special skills and strengths do I have to move my organization forward, regardless of what my job description was when I started at this company? The world is changing and it’s going to keep changing. We need to keep asking ourselves, are we helping move things forward and delivering value?

When you ask this question of research, you see a very similar pattern. Fortunately or unfortunately, our partners in product marketing, product management, design, customer success, and so on are all talking to customers and gathering feedback—and many are doing it reasonably well. And so I’m less interested in fighting over who gets to do this in every situation.

I think the higher leverage thing we need to be doing is building insights infrastructure. I’m talking about playbooks, processes, tools, training, templates—the social infrastructure around how we consume the information that’s important for decision-making.

I think this goes beyond the scope of research operations because research is often only one of many inputs into decision-making. Our partners in data science, customer success, customer experience, and so on, all have roles to play in this world.

As researchers and insights professionals, we don’t just have the opportunity but the responsibility to focus on helping every team make decisions. So, the highest leverage thing we can be doing is embedding those best practices—in other words, building infrastructure.

I want to talk about three things that I think are changing for us:

1. Tools

2. Processes

3. People

Tools

A massive amount of tools support us, and they all exist because they find ways to reduce friction in our process. But friction can be meaningful. Without friction, there’s no traction.

And so, the question is, how do we use tools to increase efficiency and effectiveness without sending people sliding around?

Tools can be helpful around what people know. But organizations don’t know things. People do.

So, what about research repositories? Well, many companies don’t have the beautiful library they think they do. Instead most have something that looks like the room of requirement. You know that there’s gold in there but there’s no way you’re going to be able to find it!

We often expect our partners to read like 50 slides to find an answer and hope they came to the same insight. But that’s changing, as we’ve seen from Dovetail’s new magic features like Magic search, where you can not just ask what the company knows, but you can think about how you bind those searches.

One of my big issues with repositories is that I think they unfairly distribute the burden of finding something to the person who’s looking rather than the person who knows where it is. It also means that we see lots of cherry-picking.

Thankfully, we’re seeing tools starting to take those things into account. We’re starting to think about how we make it safer for people to look at past information. How do we bind their searches? How do we help ensure they’re pulling from the right customers, segments, ACV—whatever we need to use to make decisions.

And it’s not just Dovetail. We’re seeing tools like Enterpret and Monterey that help companies harmonize data. They’re thinking about what lives in customer success, customer support, data science, or research, and how to bring these together to synthesize and harmonize truths into bigger patterns.

But these tools also mean a shift. They mean we focus less on reports and more on reporting.

I’ve long believed that we spend way too much time writing reports—making presentations, formatting fonts—but this work doesn’t end up actually driving value and helping companies make decisions. But these new tools offer the opportunity to think about how we make the source material safer to consume. How do we write things down, tag things, and contextualize them so that when people go into these tools, they are more able to draw the right conclusions rather than cherry-pick a quote that seems like it fits whatever narrative they need?

Processes

What are some of the processes and perspectives that need to change as we start to move forward?

When I see people outside of research excited to talk to customers, I get really thrilled. Instead of telling them to stop, I want to harness that energy and think about how I can better calibrate their eyes and ears so that when they go out into the world and see things, that adds value to how we think as a company.

To do that, we all have to be clear on what research can and can’t do and what data can and can’t say. And, more importantly, how do we harmonize around different kinds of data so that we can come to robust conclusions?

Reporting and synthesis are now being done at run-time. Instead of consuming a premade report, people will spin up reports as they need them.

But this means we have to think about how people work and work together when it comes to making decisions. If they’re starting to pull their own data and get their own reports, we have to somehow correlate those and collate them to make sense of things.

I think decision-making in organizations is a lot like Wheel of Fortune. Here are some of the similarities:

• You have a puzzle you need to solve.

• Different people can contribute different letters.

• You need to agree on what letters are missing and

  at what point

  you feel comfortable trying to solve the puzzle. Are you 100% certain? Or are you guessing? What are the risks and costs?

• You can make a plan to get the missing letters; for example, running an experiment, a customer conversation, playing with a prototype, or doing a field study.

We need to recognize that we’re not the only ones who put letters on the board. We’re all contributing to how these decisions are made. When we recognize where research is one of many inputs, we can expand our perspective. And it means that we can be better partners to our peers who also contribute to decision-making (i.e., voice of customer, customer success, customer support). This results in better data literacy and better decision-making for our organizations—a win-win-win.

People

The last shift I want to talk about is how people need to change.

The root of any problem, in my opinion, is that so often, as researchers, we think we’re only valuable when we’re doing research. We think we need to be the ones in the room writing the survey and doing the analysis.

But we’re not just data-gatherers; we are also data gardeners. We tend to the data in our organization so that people can use it more effectively and come to better conclusions.

We’re also teachers. We share best practices with those engaging in our craft. Anytime we’re bringing someone into the room, whether they’re listening to a session, engaging, or they just want to ask a question, we have to help them understand why we do what we do.

We’re also coaches. Many of us are supporting the people who are doing the work, like our friends in customer support who are on calls with customers. It’s about giving people feedback and helping them improve.

We’re also architects. For those of us who work in research operations, many of us are building systems for participant recruitment, insights management, data collection, and surveys. We’re already building these structures, and we need to build more as we move forward.

We’re also prompt engineers. Those of you who have been playing with AI or starting to use LLMs for qualitative data analysis are probably also doing a bit of prompt engineering—or you will very soon. You know the kinds of questions to ask of this data the same way that you’re writing scripts for your surveys, and you can share those with others in your company so they, too, can come to the right conclusions rather than paying attention to the wrong signals.

Doing research vs. being a researcher

I think that many people are fighting to stay on the left side of the table and that they feel we need to be the only people in the practice doing research. I disagree. I think if you step back and think about what you bring to companies—the strengths and skills you have regardless of your job title—you probably have a much more open and exploratory perspective on what you can do to help move things forward.

This comes back to this idea of leverage. What is the highest leverage way that you can contribute? Just like nature, many of us have seasons. There are times where we need to be the ones doing the work and times when we need to be helping other people do the work. Sometimes we just need to be tending to the garden.

We need to move past the idea that we always have to be operating in one way to be helpful to our company, because that’s just not true.

Closing provocations: If this many things have changed, shouldn’t we?

Something doesn’t quite add up. Unfortunately, while many organizations have smaller research teams, more people doing research, and more teams using more tools and gathering more data, oftentimes, there’s less rigor in how they ingest or metabolize the data. They feel more customer-aware or more customer-centric, but they’re still mistaking the map for the territory.

We researchers are fighting about who gets to talk to customers, but we ignore the new opportunities in front of us. We have dedicated research ops teams, voice of customer teams, customer advisory boards, and a whole host of other things that help gather and shape important decisions, but we’re not always willing to play with them.

And when I look at all of this together, I see an opportunity and a need for us to rethink our practice and what our role is. So I want to invite you to think about those new ways of being—the new modalities—whether it’s less reports, more mixed methods, different partnerships, different titles, and so on.

So, here are some of the things I’m grappling with that I want to leave with you:

1. When was the last time we evaluated our own practices? When did we sit down and think about what it is we’re doing here and who we’re doing this for? Are we really here to help our products and partners? Have we actually talked to them about what they think research is or how they want to work? Have we spent time researching the organization itself?

2. Where are we prioritizing helping one team make one decision better rather than our entire organization making every decision better?

3. How are we getting in the way of building organizations that learn? Where am I gatekeeping unfairly? Where do I feel like I (or my team) need to be the one doing something when I could actually partner with or deputize someone and help them move forward? I think more of us would benefit from having allies and partners.

4. What new roles, modalities, and opportunities exist for us today? We are in a wildly different world. Dovetail can now do things that took me hours when I started my career

   and, in some ways, that’s kind of scary. In other ways, that’s really exciting! My analysis and synthesis are probably much faster.

These sorts of tools will be used in good and bad ways, but we’re in a moment of rethinking what is possible.

If this many things have changed, shouldn’t we?

Editor’s note: This article is a condensed overview of Behzod Sirjani’s talk on the main stage at Insight Out 2024.