Examples of sampling errors in research

Leveraging market research insight effectively can give your business the edge ahead of your rivals. But you must play your cards right to get the most out of your research.

Unreliable research results translate to a waste of resources and time. And sometimes, you may have to repeat the expensive process all over again.

Sampling and non-sampling errors are the issues that can rear their ugly heads in every survey. Fortunately, you can minimize or steer clear of them.

A sampling error is the difference between a sample's mean value and the entire population.

By definition, sampling means you aren't measuring the entire population's data. So this "error" is generally unavoidable whenever you sample from a population, even if you construct a representative sample. Quantifying it is also impossible (although you can calculate a margin of error, assuming your sampling is random).

What's sampling error vs. sampling bias?

While sampling error is a deviation of the sample's mean value from the population's true value, sampling bias is an expectation that a sample will not accurately represent the population. For example, if the sample has proportionally more men than the entire population.

What is sampling error vs. standard error?

We've seen that sampling errors are the deviations of sample values from the whole population's true values. But how can you measure these statistical errors?

That's where the standard error comes to your aid. Standard error expresses the degree of sampling error, essentially between the calculated mean of the population and one which is considered known or accepted as accurate. This allows you to understand and communicate errors in a meaningful way.

And what about non-sampling errors?

Non-sampling errors include other discrepancies caused by an external factor when collecting data. They can happen whether you're sampling (for example, in a national survey) or not (for example, surveying your entire workforce about employee experience).

In other words, they can be present in the sample and the entire population. These errors fall under two categories:

• Random errors —They often offset each other, so they aren't significant

• Systemic errors — They can affect the whole sample and render the collected data useless

Non-sampling sources of error include:

• Non-responses

• False information from participants

• Data collection errors

• Measurement errors

• Data entry discrepancies

• Coding errors

• Poor questionnaire design leading to inaccurate participant responses

• Biased analysis conclusions

• Biased processing

• Improper researcher training

• Inadequate data

While you can reduce sampling errors by increasing sample size and employing proper sampling procedures, these methods cannot remove non-sampling errors. Instead, you need to employ quality control measures for proper design, implementation, and monitoring of all elements of the survey process.

Sampling errors come in different shapes and sizes. Here are the common ones.

1. Population specification errors

Population-specific errors happen when you don't know who to survey.

For instance, suppose you're researching bread consumption in US homes. Should you interview the whole family, the children, or the mother?

While the mother might be the decision-maker who usually buys bread, children might influence the type of bread to eat.

2. Non-response error

A non-response error occurs when the survey fails to capture a useful response because the potential respondents were unreachable or refused to cooperate.

Suppose you need to research your target market before rolling out a new product or service. Most of your participants might be your current buyers who already know you—only a few might be new people who've never heard about you or used your product.

3. Sample frame error

Sample frame error happens when you select a sample from the wrong sub-population. In such cases, the sample doesn't accurately represent the entire population under study.

Suppose a researcher uses a phone directory to pick a sample frame. The result will be erroneous inclusions (people regularly move house and change addresses), exclusions (people unlist their contact details, and some adults don't own a telephone), and multiple inclusions (households can list several numbers).

4. Selection error

Selection error happens when people self-select themselves to participate, so only participants interested in your research respond. You can eliminate this error by encouraging participation.

There are various examples of sample errors. Some of them include the following:

• Sampling bias

• Sample selection bias

• Sample coverage error

• Sample contamination

• The sample size is too small

• Sample unrepresentativeness

Example 1: YouTube Channel with different viewers

Suppose your favorite YouTuber films a certain video once a week, and you want to survey its viewership. Most of your viewers consist of youths aged 12 years to 35 years. You must draw a sample proportionately representing demographic factors such as gender, education, and age.

For instance, people aged 12 to 18 have plenty of time to spare. That means most of them can watch the videos more consistently. On the other hand, those aged 18 to 35 may not have time for it due to increasing commitments.

Since you don't know the demographic data of all your viewers, unknown sampling errors will inevitably occur. But it's possible to employ analytical methods to determine the amount of variation the error causes.

Example 2: real-life presidential election prediction

In the 1936 presidential election, incumbent president Franklin D. Roosevelt was battling it out with Alfred Landon, a Republican Kansas governor. Before voting, Literary Digest's poll had predicted that Landon would emerge the winner with 57% of the votes and the incumbent president would lose with 43%.

With a sample size of 2.4 million respondents, this was among the magazine's most expensive and largest poll. But the official result came out different: Roosevelt scooped 62% of the votes while Landon got 38%.

According to analysts, the sampling error was a whopping 19%. Its nature was sampling frame error. Literary Digest sampled respondents from car registrations and telephone directories. However, most citizens didn't have cars and phones during those years, and those who owned them were mostly Republicans. That's why the poll erroneously crowned the Republican candidate, Landon.

It's often hard to study or interview every member of a population, especially if you're dealing with thousands or millions of members and you don't have enough resources. Therefore, researchers usually employ random sampling.

But still, it's rare to obtain a random sample that's 100% identical to the target population.

So the calculation of sampling error helps analysts estimate a sample size's uncertainty amount. You need to include the figure in your final report.

Is the sampling error large? It shows that the sample size isn't representative of the true population. In this case, you may have to discard the results and repeat your survey.

The sample size isn't the only thing to watch out for when designing a representative sample. Calculation of sampling error allows you to know whether you need to consider other factors, such as confidence level.

Calculating sampling error: How do you find the sampling error?

It's impossible to know the sampling error. But if you've done random sampling, you can estimate it using the margin of error.

Let's switch to mathematics mode, shall we? Here are easy steps that will help you calculate sample error:

• Take your population's standard deviation and divide it by the sample size's square root.

• Multiply the result you got by the Z-score value, which corresponds to your confidence level (CL).

• The result (margin of error) is your approximate sample error.

Here's the formula:

Sampling error = Z x (σ/ √n)

Where:

• Z is the Z score value corresponding to the CL

• σ is the standard deviation of the population

• n is the sample size

While it's impossible to eliminate sample errors, you can significantly reduce them. Here are our true-and-tried tactics.

1. Study the actual population

Before sampling your target group, research its demographic mix. That way, you can understand the population parameters and ensure the sample resembles the entire population.

2. Increase sample size

Having a larger sample is a common trick that helps slash sampling errors. The more sample members, the closer the sample gets to the true population, reducing the potential for deviations.

For instance, you're studying the employee experience of a manufacturing company with 100 employees. A sample of 20 workers is more representative than that of 10 workers. Generally, more samples equal additional expense and time to conduct the research.

Calculate your ideal sample size

3. Random sampling

Always conduct random sampling to help reduce sampling errors. Your best bet is to use a systematic sampling approach instead of picking participants haphazardly.

Let's go back to the employee experience example. First, compile a list of the names of all the employees in that department. Next, pick workers whose names appear as 5th, 10th, 15th, 20th, etc.

4. Improve sample design and methodology

Remember to improve sample design and consider the different sub-population within the larger population. Again, staying with the employee experience example, does 40% of the workforce come from the production department? If so, ensure 40% of those you interview come from that department.

Your best bet is to do stratified random sampling. Split the population into homogeneous sub-populations called strata. Then pick sample members from the group using random sampling. That way, the sample population's composition will resemble that of the entire population.

Other tactics that can reduce sampling error are here:

• Reduce the confidence level (with the implication, however, you will be less confident the result isn’t a result of randomness within the sample).

• Improve survey questions, e.g., via cognitive testing, to ensure respondents understand the meaning behind the questions.

• Use accepted sampling techniques as described above.

Sampling errors are the always-present villains whenever researchers take a random sample from the target population. But you can reduce them and make your results credible enough to be used in your crucial decisions.