Did bad data drive your last decision?

Janelle Ward

Did you know that 87.5 percent of all UX research-generated statistics are false?

87.5 percent!

Wow!

Actually, that’s not true. I made it up.

But did you believe me, even for a second?

Is it because I used numbers? Or because you’re reading an article in a publication you’ve grown to trust, written by a research expert?

This might happen to you in daily life, too. Think of all the “research” that is thrust upon us—in commercials, on billboards, and in news articles. I grew up with television commercials telling me all sorts of convincing things, like “four of five dentists surveyed recommend sugarless gum for their patients who chew gum!” There was something about those numbers that made me believe—Trident gum wasn’t only delicious but it was somehow healthy, because dentists liked it.

What is it about the process of creating and sharing these statistics that makes them so compelling to humans? Joel Best, the author of the book Damned Lies and Statistics, says that all statistics are products of social activity, a process sociologists call social construction. He says we “assign statistics ‘facticity,’ that is they are presumed to be facts.”

When we want to understand the extent of a problem, or intervention, or other issue, we tend to treat statistics as objective and indisputable. In the process, we ignore how these numbers were created or even how they are being shared—or by whom.

These social processes are also something we deal with in professional contexts: at work, and also in our broader network, like on LinkedIn. Let’s start with an example from the latter.

I recently spotted a perfect storm after reading a recent post by Steve Blank. In it, he references an article titled, Startups That Have Employees in Offices Grow 3 1/2 Times Faster.

His post reads: “Data shows that pre-seed and seed startups with employees showing up in a physical office have 3½ times higher revenue growth than those that are solely remote.

Let the discussion begin.”

The discussion did indeed begin.

And it turns out—after I dived into the data on which the article is based—that the headline is quite misleading. My comment on his post, reproduced below, looks more closely at why.

When you look at the comments on Steve Blank’s post, you see lots of questions criticizing the data. Hello, critical readers! Nice to have you here.

But you also see a lot of comments from company leaders running with the headline, posting things like:

“Humans are social creatures by nature.”

“...in my bones I believe people can co-create more fluidly if they do it in the same room.”

“I’m not even surprised.”

A well-known figure takes a look at a poorly executed study, writes an article about it, and shares it with a misleading headline.

At work we are surrounded by the result of poorly-informed decisions every day: This is one example of how it begins.

This example isn’t about debating remote versus in-office culture, or diving into how this differs based on company size, industry, or other factors. It’s about considering how people in leadership positions—people in this case, who may hold the power to force employees back to the office—are possibly making a decision based on a dubious post.

A dubious post that used a small study, which drew on dubious definitions of “in office,” then shared by a VIP in the business world.

It’s easy—especially for those of us more qualitatively inclined—to sit back and judge those who take numbers at face value. We know it’s more complicated than that. The world can’t be whittled down to multiple choice or percentages.

Not so fast. This happens with qualitative data too, and in some ways, the result can be even more disturbing. Certainly, people do not always understand the statistical methods used to generate certain results. Additionally, they also may not understand how we analyze qualitative data.

Truthfully, the process of analyzing qualitative data can be (even) less transparent. The danger, then, is that people can pull out a finding and say they don’t “need” more people to validate its veracity.

Here’s an example: Stakeholders who run with a single, qualitative data point—encouraged by an advocacy-driven researcher.

It’s popular in the UX research world to keep our stakeholders abreast of our ongoing research projects. I’ve heard of many researchers who post updates immediately following user interviews, for example by providing an interesting quote from the interview and posting it in a larger, internal Slack channel.

Imagine this: A non-research stakeholder sees the quote and reposts it across the company as “proof” that “improving our information architecture is priority number one!”

When that study was completed, it turned out that this was an isolated comment from one interviewee, which stemmed from a misunderstanding.

The quote, in and of itself, was highly compelling and drove interest in the research project. But the interest was misplaced.

The ball had already started rolling. That quote made it to a VIP decision-making table. You know, the one where roadmaps and OKRs are planned.

Later, of course, those people also decide whether research provided an accurate path forward—or not.

What can we, as researchers, do to encourage others to think more critically about data? This can be tricky, especially when the data being held up as “truth” is coming from the mouths of leadership. Here are some suggestions:

Did bad data drive your last decision? If you’re a researcher, I hope not. But we all slip up sometimes. The good news is that, with our training, we have the tools to look more critically at our own—and our stakeholders’—choices.