Did you know that 87.5 percent of all UX research-generated statistics are false?

87.5 percent!

Wow!

Actually, that’s not true. I made it up. 

But did you believe me, even for a second?

Is it because I used numbers? Or because you’re reading an article in a publication you’ve grown to trust, written by a research expert?

This might happen to you in daily life, too. Think of all the “research” thrust upon us—in commercials, billboards, and news articles. I grew up with television commercials telling me all sorts of convincing things, like “four of five dentists surveyed recommend sugarless gum for their patients who chew gum!” Something about those numbers made me believe—Trident gum wasn’t only delicious but somehow healthy because dentists liked it.

What is it about the process of creating and sharing these statistics that makes them so compelling to humans? Joel Best, the author of the book Damned Lies and Statistics, says that all statistics are products of social activity, a process sociologists call social construction. He says we “assign statistics ‘facticity,’ that is, they are presumed to be facts.”

When we want to understand the extent of a problem, intervention, or other issues, we tend to treat statistics as objective and indisputable. In the process, we ignore how these numbers were created or how they are being shared—or by whom. 

We also deal with these social processes in professional contexts: at work and in our broader network, like on LinkedIn. Let’s start with an example from the latter.

I recently spotted a perfect storm after reading a recent post by Steve Blank. In it, he references an article titled, Startups That Have Employees in Offices Grow 3 1/2 Times Faster.  

His post reads: “Data shows that pre-seed and seed startups with employees showing up in a physical office have 3½ times higher revenue growth than those that are solely remote. 

Let the discussion begin.”

The discussion did indeed begin. 

After I dove into the data on which the article is based, it turns out the headline is quite misleading. My comment on his post, reproduced below, looks more closely at why.

When you look at the comments on Steve Blank’s post, you see lots of questions criticizing the data. Hello, critical readers! Nice to have you here.

But you also see a lot of comments from company leaders running with the headline, posting things like:

“Humans are social creatures by nature.” 

“...in my bones, I believe people can co-create more fluidly if they do it in the same room.” 

“I’m not even surprised.” 

A well-known figure looks at a poorly executed study, writes an article about it, and shares it with a misleading headline.

At work, we are surrounded by the result of poorly-informed decisions every day: This is one example of how it begins.

This example isn’t about debating remote versus in-office culture or how this differs based on company size, industry, or other factors. It’s about considering how people in leadership positions—people, in this case, who may hold the power to force employees back to the office—are possibly making a decision based on a dubious post. 

A dubious post that used a small study, which drew on questionable definitions of “in office,” then shared by a VIP in the business world.

It’s easy—especially for those more qualitatively inclined—to sit back and judge those who take numbers at face value. We know it’s more complicated than that. The world can’t be whittled down to multiple choice or percentages.

Not so fast. This happens with qualitative data too, and the result can be even more disturbing in some ways. Certainly, people do not always understand the statistical methods used to generate certain results. Additionally, they also may not know how we analyze qualitative data. 

Truthfully, analyzing qualitative data can be (even) less transparent. The danger, then, is that people can pull out a finding and say they don’t “need” more people to validate its veracity.  

Here’s an example: Stakeholders who run with a single, qualitative data point—encouraged by an advocacy-driven researcher.

It’s popular in the UX research world to keep stakeholders abreast of our ongoing research projects. I’ve heard of many researchers who post updates immediately following user interviews, for example, by providing an interesting quote from the interview and posting it in a larger internal Slack channel.

Imagine this: A non-research stakeholder sees the quote and reposts it across the company as “proof” that “improving our information architecture is priority number one!”

When that study was completed, it turned out that this was an isolated comment from one interviewee, which stemmed from a misunderstanding.

The quote, in and of itself, was highly compelling and drove interest in the research project. But the interest was misplaced.

The ball had already started rolling. That quote made it to a VIP decision-making table. You know, the one where roadmaps and OKRs are planned. 

Later, of course, those people also decide whether research provided an accurate path forward—or not.

What can we, as researchers, do to encourage others to think more critically about data? This can be tricky, especially when the data being held up as “truth” comes from the mouths of leadership. Here are some suggestions:

Did bad data drive your last decision? If you’re a researcher, I hope not. But we all slip up sometimes. The good news is that, with our training, we have the tools to look more critically at our own—and our stakeholders’—choices.