GuidesUser experience (UX)Ensuring data quality in UX research: best practices and pitfalls to avoid

Ensuring data quality in UX research: best practices and pitfalls to avoid

Last updated

12 June 2023

Author

Claire Bonneau

Reviewed by

Eliz Ayaydin

UX research is a critical part of creating truly user-centered products. Data gathered through the research process can inform decision-making, lead to more satisfying and relevant projects, and boost business success.

But what happens if you notice red flags while you’re looking through a data set? Perhaps the participant responses are suspiciously similar, the insights simply don’t match the data, or your entire cohort lacks diversity.

Those are just a few of the pitfalls UX teams might find themselves up against. This guide will help your team avoid some of the common challenges in gathering data that’s valid, reliable, and beneficial to your customers.

Don't let UX problems go unnoticed

Use Dovetail to analyze all your customer interviews and uncover hidden UX issues that are costing your business every day.

Analyze with Dovetail

The importance of data quality in UX research

Data quality is of critical importance in UX research. Whether or not you gain accurate data impacts every aspect of the research process, as well as the reliability, usefulness, and accuracy of insights.

Here are some of the key areas that data quality impacts:

  • Insight accuracy: gaining correct and reliable insights is impossible without accurate data. Data that is skewed, biased, or gathered incorrectly will lead to false or even exaggerated takeaways. This can impact the entire project.

  • Solving the right problem: solving the wrong problem can lead to issues in organizations. Without reliable data, your products might address challenges your customers don’t have instead of those they do have.

  • Making the right decisions: the decisions your team makes (e.g., to release a new feature, focus on a certain market segment, or produce a new product) must be based on high-quality data. If not, decisions are more likely to end in failure.

  • Long-term success: relying on low-quality data not only impacts the project at hand, but it can also have a broader long-term impact on the organization as a whole.

Identifying potential sources of bias and errors in data collection

It’s not uncommon to spot things that don’t add up when looking through a data set. Imagine you’re looking through notes from a focus group and it appears that every participant had the exact same feedback and sentiments. Or perhaps you’ve deployed a survey and a user has checked option A for every single answer. In another scenario, you might notice that your data set has no diversity.

Detecting these errors, spotting bias, and reacting accordingly is essential for maintaining data quality.

Here are some red flags to look out for:

Unreliable responses

These occur when a participant hasn’t answered honestly, or an honest participant responds inaccurately.

It’s impossible to weed out every case, but looking for faulty response patterns that stand out is a good step.

For example, a participant answering the same response choice for every question could be a red flag. Another red flag is someone putting one-word answers for open-ended questions or copying and pasting answers.

Identify and remove these responses from the overall data set whenever this occurs. Consider reviewing the research study’s design if you notice a trend of unreliable responses. It might be too confusing or long.

Confirmation bias

Your team should come into a research project with an open mind instead of fixed ideas. If there’s a desired outcome, this can heavily skew the results and lead to confirmation bias—the tendency to accept ideas that align with the desired outcome even if they don’t represent the data.

Sampling issues

Diverse views ensure your insights are reliable and applicable to a broader user base. Information from a narrow group will likely skew the data and not accurately represent the market.

Recording errors

Creating data entries accurately is of the utmost importance. Manually entering data or collecting data in multiple formats can make this process tricky and unreliable. Various tools can increase accuracy and speed up the data entry process.

Measurement errors

Data quality doesn’t just matter during the collection process; it’s also critical during measurement and analysis. Manual processes won’t usually cut it when it comes to large data sets. As with data collection, advanced tools can help prevent mistakes.

Tips for designing effective survey and interview questions

Asking the right questions can help you avoid unhelpful or inaccurate responses. This is true whether you’re deploying a survey, conducting an interview, or running a focus group.

Here are some ways to design effective questions:

Start with clear goals

All useful projects start with clear, measurable goals. When writing your survey or interview questions, link them back to the overall goals to keep the project aligned and on track.

Avoid leading questions

The goal of asking your participants questions is to get into their heads, not confirm what you want or expect to be true. Accurate and reliable data occurs when you ask questions neutrally without suggesting the answer.

Ask one question at a time

When participants feel overwhelmed, they might answer in ways that don’t align with their thinking. That’s why it’s essential to ask one question at a time, allowing the person to consider their answer carefully.

Try including a red herring question with a strange answer in your survey. This will help you weed out poor data from distracted participants who are trying to get through the survey quickly.

Use simple language

Keep your questions concise, clear, and free of technical language to avoid confusion and misunderstandings. The more conversational, the better.

Use a mixture of open and closed questions

Closed questions are easier to calculate and quantify, but open-ended questions give the data nuance and color. A mixture of both can ensure you gain both logical and unexpected insights and context from participants.

Checking the quality of the response in an open-ended question can help identify poor data. Look out for one-word answers or responses that don’t make sense (in comparison to thoughtful, logical responses).

Ensuring reliable and valid measurements

In addition to asking the right questions, there are other measures you can take to ensure your data is reliable and valid.

All researchers should keep the following in mind:

Consistent ranking

While you might gain responses from multiple sources, using consistent ranking scales is essential. This is especially true for closed questions. Using ranking scales will ensure your data is easier to analyze and draw insights from.

Consider bias

Thinking about bias at the outset will help you eliminate it as much as possible from your project. You can take steps to remove bias by ensuring there isn’t an expected outcome, not asking leading questions, and avoiding biased samples by working with a diverse participant cohort.

Use advanced tools

Collect data through a standardized tool. This will ensure your measurement process is accurate and efficient, making the analysis a breeze.

Addressing data quality issues as they arise

Addressing any data issues at the outset is important. This means highlighting them before any conclusions are drawn or business decisions are made. 

This can be performed by: 

Identifying the issues

As you complete the analysis process, flag any data that looks out of place or potentially flawed. 

Dig deeper

Don’t make assumptions too quickly. Instead, take a good look at any flagged data to understand whether it’s flawed and why. Properly understanding this can help you identify what went wrong and come up with potential solutions.

Deal with missing data and outliers

Consider how missing data and outliers impact the overall project. It’s helpful to determine whether the missing data is integral to the project or impacts the validity of the findings. You might have to re-do the research or fix those entries if this is the case.

Fix the data

You can perform fixes where an incorrect data entry has been made or a response is clearly incorrect. Usually, this means removing the low-quality entries to prevent them from skewing the results. In some cases, you might need to validate the responses and recheck any potential errors.

Document any changes

Avoid performing fixes or making changes without detailed notes to ensure all parties are aware of the edits and the reasoning behind them. This avoids work happening in silos without explanation.

Best practices for data analysis and interpretation

Gaining insights that can lead to positive actions relies on the data analysis process. Once accurate data has been collected, it should be analyzed correctly to draw helpful conclusions that will lead to better products and happier customers.

Accurately analyze your data with the following steps:

1. Clean and organize the data

Things like missing values, data that raises red flags, inconsistencies, and outliers ought to be excluded from the larger data set before an analysis is performed (if they haven’t been addressed already).

2. Select the right analysis technique

Once the data is cleaned, choose an analysis technique that aligns with your project and overall goals. It should also align with the data you’ve collected. Techniques such as descriptive statistics, user journey mapping, and thematic analysis are just a few of the techniques you might choose to organize the data.

3. Use the right tools

The right tools can speed up the process and ensure you gain accurate and reliable insights. With Dovetail, for example, teams can have all their customer data in one place. The tool allows them to tag themes, segment data, uncover patterns, and share insights to action quickly across the business.

4. Stay unbiased

You should be unbiased from the beginning to the end of the project in order to stay objective and avoid coming to conclusions too quickly. Stay curious to see what story the data tells and what you can learn from the research process. Don’t just wait for what you hope it will reveal. The results may be surprising—that’s why this process is essential.

Evaluating the trustworthiness and credibility of research findings

Evaluating the trustworthiness and credibility of all research findings can help you maintain high-quality data and get reliable insights. It doesn’t matter whether or not your team performed the research. This keeps data integrity high and avoids reliance on research that doesn’t tell the whole story.

The following questions can help you evaluate the trustworthiness of research:

  • Were the research methods used relevant to the project?

  • Is the data accurate?

  • Has the data been cleaned? Have outliers and missing data been removed?

  • If you performed this research again, would these results likely be replicated?

  • Did the results come from a diverse cohort of participants?

  • Did the process involve any coercion or bias?

  • Is the data set large enough to be statistically significant?

  • Are there missing aspects that require additional research?

  • Are you relying on secondary research that has been validated and peer-reviewed?

  • Were there any conflicts of interest that need to be addressed?

  • Have you documented your findings, processes, and data edits?

Boosting data quality with best practices

A critical aspect of UX research is ensuring data quality is high. This ensures that any research has reliable and meaningful results and is free from bias and inaccuracies.

Following best practices can help you avoid common pitfalls and keep your research’s integrity and validity high. Practices like recording data accurately, working with diverse participants, following data cleaning practices, and using advanced tools for analysis can help.

Gaining high-quality data can help foster trust in research outcomes. It can also lead to more accurate insights, which you can leverage to create better products—all with the goal of benefiting the end user.

FAQs

What are the best practices for ensuring data quality in UX research?

Some of the core best practices for keeping data quality high include accurate data collection, consistent ranking systems, cohesive survey questions, data cleaning practices, and detailed analysis with advanced tools.

What are the common pitfalls to avoid when collecting data in UX research?

The common pitfalls include asking leading questions, having a lack of diversity in the sample group, entering data inaccurately, letting bias influence results, and relying on unclean or low-quality data.

How can I ensure my data is reliable and valid in UX research?

Data that is reliable and valid is collected from a diverse group of participants without bias, accurately recorded, and cleaned. Outliers and missing entries are removed, and the data is analyzed using advanced and accurate tools. These, among other practices, can help ensure data is high-quality and reliable.

Get started today

Go from raw data to valuable insights with a flexible research platform

Start freeContact sales

Editor’s picks

6 tips on choosing a research topic

Last updated: 18 January 2023

User persona templates

Last updated: 24 June 2023

User persona templates

Last updated: 29 May 2023

What is customer experience design?

Last updated: 14 November 2023

How to conduct a successful UX audit

Last updated: 3 April 2024

7 usability metrics you should be tracking

Last updated: 19 November 2023

Usability testing templates

Last updated: 6 April 2023

A guide to using AI in UX research

Last updated: 22 June 2023

Latest articles

How to conduct a successful UX audit

Last updated: 3 April 2024

7 usability metrics you should be tracking

Last updated: 19 November 2023

What is customer experience design?

Last updated: 14 November 2023

User persona templates

Last updated: 24 June 2023

A guide to using AI in UX research

Last updated: 22 June 2023

User persona templates

Last updated: 29 May 2023

Usability testing templates

Last updated: 6 April 2023

6 tips on choosing a research topic

Last updated: 18 January 2023

Related topics

Patient experienceResearch methodsSurveysMarket researchCustomer researchUser experience (UX)Product developmentEmployee experience

Product

ChannelsMagicIntegrationsEnterpriseInsightsAnalysisPricingLog in

Company

About us
Careers14
Legal
© Dovetail Research Pty. Ltd.
TermsPrivacy Policy

Log in or sign up

Get started for free


or


By clicking “Continue with Google / Email” you agree to our User Terms of Service and Privacy Policy