What is internal validity?

Research studies are only as good as the quality of their takeaways. Through internal validity, you can measure whether your research is conclusive enough to have confidence in its results.

Getting to that point is complex, especially considering how many factors influence the internal validity of your study. There's also the inherent conflict with external validity, which considers how broadly applicable your research takeaway is to other topics.

This guide dives into the nuances of this critical concept, from a basic definition to typical threats and potential steps to measure the validity of your research.

Internal validity measures if your research is sound. It’s the degree of confidence with which you can say the results from a study are conclusive and cannot be influenced or changed by other factors. If you’re confident that the variable you changed led to the study's outcomes, your study has high internal validity.

No study is ever fully internally valid or invalid. Instead, it's a spectrum. The study can be more or less internally valid based on the existence of threats to its accuracy and causal relationship. Understanding these threats can help us build a methodologically sound study with reliable results. 

What is internal validity in research in simple terms?

Put simply, proving internal validity means demonstrating that the thing you're changing in a study is responsible for the outcomes you're measuring. In other words, it's the degree to which you can prove that your research results are not just coincidental.

It's about establishing a cause-and-effect relationship that ensures you can draw valid conclusions from your research. 

Why internal validity matters

Whenever you conduct research, you need to be able to trust the results. Establishing internal validity accomplishes that, allowing you to treat those outcomes as actionable takeaways and discoveries about your audience. 

Establishing this validity helps you eliminate other potential explanations for research outcomes. You can draw confident conclusions from it rather than having reports full of uncertainties and what-ifs. Internal validity is one of the most effective research tools if you use it properly.

What is an example of internal validity?

Consider a website test that tries to establish whether a hero image or video performs better for first-time visitors. If you establish internal validity, you can confidently use the results of your study to determine which option performs better for future use. 

In this scenario, several factors affect internal validity. For example, users who see the hero image may be different demographically from those seeing the video. They might visit the website at a different time or already have pre-existing brand awareness compared to the other group. 

Any of these factors could reduce the validity of the research, clouding the cause-and-effect relationship between the choice of image or video and the resulting performance.

Internal validity increases in importance as research complexity rises. For example, researchers in France sought to determine whether early ventilation increased mortality rates among patients with severe ARDS. But factors like participant selection bias and measurement errors threatened internal validity and rendered the study inconclusive.

Disadvantages and challenges of internal validity

Internal validity is never black and white because you’ll never generate a 100% confident research result. Some factors and threats will always exist to potentially weaken the research findings.

Researchers try to fight this challenge through a few tactics:

• Running a control group to provide a baseline of results to compare the findings to

• Eliminating uncontrollable variables as much as possible, like ensuring that all groups in the study have the same demographic makeup and external environment

• Repeating the same study multiple times to see if the results remain consistent over time, even as uncontrollable variables change

The biggest disadvantage is that the more internal validity increases, the more external validity decreases. External validity is the ability to generalize research findings. Making the study more specific naturally makes it less applicable to less-controlled situations.

While internal validity refers to the degree to which we can be confident about research results, external validity is the extent to which these results apply to different situations.

Both internal and external validity are different concepts. But they're closely linked and equally important as you look to build a research experiment.

Internal validity matters because you need to ensure the variable you're changing is actually responsible for the result you're measuring. 

External validity matters because the change you're measuring should apply more universally. If you can’t extrapolate conclusions to the "real world," the study is essentially useless for business insights or a wider audience.

Both types of validity occur in degrees, meaning a study can't be completely internally or externally valid. At the same time, they tend to oppose each other, with internal validity increasing as external validity decreases.

The trade-off between internal and external validity

The natural step to increasing internal validity is to control the study’s variables as much as possible. But the more that happens, the less you can generalize those findings to other contexts.

Consider the same example of studying a video or image hero preference for your website. Ensuring internal validity requires controlling the study as much as possible:

• Demographics: Users who see the video and image versions should come from the same demographic backgrounds to remove bias based on gender, age, and other variables.

• Timing: They should visit the website at the same time to remove the possibility that users simply like watching videos more in the morning than in the afternoon.

• Same page: They should all visit the same page on the web page, like your home page, to reduce the chances that non-hero-related design elements influence their actions.

• Same subject: They should all see an image and video from the same topic to eliminate the possibility that they respond differently based on the content rather than the format.

However, these steps will make the research findings less applicable to other contexts. You might be unable to conclude whether your users prefer video or images on social media or other website pages. 

Such a narrow and specific pool of participants may limit the demographics of your users. That demographic may not even be a target or ideal pool of users. You may also need to reconstruct the entire study to identify which demographics prefer videos. As internal validity increases, external validity decreases.

The key is establishing a balance between internal and external validity. For every research experiment you're planning, consider how universally you want to apply the findings and how to eliminate potential threats to that all-important cause-and-effect relationship.

Every research experiment inherently has some threats that could affect whether its results are accurate or trustworthy. These nine threats are especially common when considering internal validity:

1. Attrition

If your study measures actions over time, some participants will likely drop out. As a result, your data might become biased even if you picked an initially representative or random sample. However, we sometimes want non-targeted users to drop off as they don’t represent our typical target audience.

2. Historical events

An external event that you have no control over could influence results. For example, an election occurring in the middle of a study about political preferences could bias participants in one direction. 

3. Maturation

The passage of time has a natural effect on study participants. This applies more to technology since it moves so fast. Participants may discover new solutions, competitors, or devices. This can greatly influence their behaviors in ease of use and degree of product novelty.

4. Diffusion

If you're conducting a study with multiple groups, how isolated are they from each other? If they can interact with or observe each other, their actions may influence each other in ways that lead to less confidence in the results.

5. Confounding

A confounding variable in the study is one you're not accounting for that might unintentionally influence your result. In mathematic terms, it's an independent variable that could influence the effect you're looking to measure, like the effect of connection problems on the mood of your participants.

6. Experimenter bias

It's human nature to see what we want to see. The problems begin when that bias influences the person conducting the research. That might happen through the types of questions we ask, favoritism towards one group in a multi-group experiment, or how the researcher interprets the results.

7. Instrumentation

This threat happens when you use different instruments to measure results for different groups or over time. For example, one group of participants may have 15 minutes to complete a task, while the other group has 30 minutes.

8. Testing

If you give someone the same test three times, their results may naturally improve. Yet, it's tempting to attribute that improvement to the variable you're introducing. That's the testing threat in a nutshell, and it can significantly affect internal validity.

9. Statistical regression

Regression to the mean is the statistical tendency for results on the extreme end of the spectrum to score closer to average next time. For example, participants who score extremely low or high the first time they take a test may get closer to average the next time, even without involving another variable.

Awareness of these threats allows you to build a better and more accurate study, especially if you know how to counter them.

In a world full of threats, how can you ensure internal validity? While it's impossible to eliminate these threats completely, a few factors can help your research head in the right direction:

1. Blinding

This process involves using a control or placebo group to ensure participants are unaware of whether they're part of the research. This prevents them from intentionally or unintentionally influencing the results through their behaviors or responses.

2. Random selection

While participants must be representative of the audience you're researching, selecting them randomly within that larger group prevents accidental bias. This allows you to be more confident in your takeaways.

3. Randomization

Randomness also matters within the study. Avoid consciously selecting participants of the control or placebo group for any reason. Instead, randomly assign them to each group to avoid any systematic bias.

4. Study protocol

A standardized way to conduct your study prevents experimenter, instrumentation, or testing biases. When everyone follows the specific procedures, you can be more confident that the results will be valid.

5. Experimental manipulation

Sometimes, you might have to manipulate independent variables to remove the confounding threat. For example, you can minimize the effects of different groups participating at different times of day if every group gets an equal period of rest before the experiment.

Beyond these more structural factors, a few other steps can also help you improve the internal validity of your research. For example, increasing your sample size to be statistically relevant fights the testing threat, as does hiding your actional test or questionnaire in a series of irrelevant and unrelated questions or tasks. 

Measuring internal validity requires some level of qualitative judgment. In other words, you'll need to estimate your degree of confidence based on your assessment of the threats described above.

That said, a few steps can help you measure the internal validity of your research.

How to check whether your study has internal validity

Three basic questions can help you check whether your research outcome is a result of the variable you're changing or outside factors:

1. Did the outcome happen after you changed a variable?

If it didn't happen in this exact order (you changed something, then the outcome changed), other factors are likely at work.

2. Did your change and the outcome change together?

In other words, did the outcome happen every time you changed something? Repeating the same study multiple times can help give this answer.

3. Could any external factors cause the change?

This is the most difficult question to answer. It requires closely examining your testing environment and the participant groups you're working with, keeping the nine common threats we listed above in mind. 

Depending on your type of study, a few other steps also help to measure internal validity. For example, you can measure the accuracy of survey results by cross-checking them against other surveys on similar topics to see if the same or similar results hold. 

You'll never be able to get a definite answer on internal validity. But with the correct understanding, you can design your research to establish a high confidence level. Measuring validity can confirm that you've designed everything to get accurate, actionable research results.