Margin of error: What it is and how to find it (with examples)

Survey results are not always accurate because the sample responding to your survey may not perfectly represent the population you’re targeting. Calculating the margin of error will help you to understand the degree of uncertainty in your survey results.

When conducting surveys, the margin of error is one of the most important metrics to understand. This guide will show you how to find it and what it means for your data.

A margin of error shows that you acknowledge survey results will vary from sample to sample, and also from the actual population mean. It allows you to express how much the results could vary based on the rules of probability, and how much confidence you have in this variance. 

The margin of error can be presented as high and low values on either side of a mean value reported in a survey. For example, “The average respondent had 1.7 drinks per bar visit, ± 0.5 drinks (95% confidence).”

A term often used interchangeably with “confidence interval”, the margin of error indicates how much confidence you and others can have in your survey results. The smaller the margin of error, the closer the survey findings are to the correct value; the larger the margin of error, the further they are.

Always expressed as a plus or minus, the margin of error is vital in surveys because:

• It helps to determine the uncertainty of statistics, enabling surveyors to make better decisions regarding their findings

• It accounts for disparities between findings

• It tells the surveyor how effective the results are

• It helps the surveyor identify how accurately the sample represents the chosen population

The margin of error is used when responses cannot be recorded from every member of the population being studied. It is a cheaper and more practical way to capture a representation of the whole population. Since surveys cannot accommodate the entire population, the margin of error helps account for incomplete results.

It can only be used, however, when the sample is randomly selected. If some kind of bias occurs in sampling (e.g. survivorship bias, volunteer bias, healthy user bias), the margin of error is not an accurate measure of how well the survey represents your population of interest.

When there is bias in sampling, increasing the sample size only makes us more confident in the wrong result. Consider how many polls misjudged several states in the northern midwest of the US, such that people were surprised by Donald Trump's victory in those states, and thereby the US presidential election.

Margin of error is not an indication of an error with the survey itself. On the contrary, it is used in the case of good random sampling survey design.

However, there are kinds of survey errors that influence results. These include:

Coverage error

A coverage error happens when the target audience members are excluded from the sample. It can also be defined as surveying the wrong sample population by not targeting the right people.

Sampling error

A sampling error occurs when a subset of the population of interest is surveyed, rather than a truly representative sample.

Non-response error

This error occurs when all or a portion of the data from the sample is missing. For example, it may happen as a result of unavailable respondents, resulting in an incomplete collection of information.

Measurement error

Measurement errors occur when the collected response differs from the actual value. They can be caused by respondents, survey personnel, or interviewers. A measurement error results from:

• Inadequate personnel training

• Poor questionnaire design

• Insufficient quality control

A confidence interval is related to the margin of error. It is essentially the margin of error expressed as a range of possible values, together with the confidence level expressed as a percentage.

For example, 100 people were surveyed about their television-watching habits. The mean average number of hours watched per week was 35 hours. The margin of error was calculated to be plus or minus 2 hours, and the level of confidence in the margin of error was calculated to be 95%, so the confidence interval can be expressed as: 95% CI = 33,37.

To determine the margin of error for your survey, follow these six steps:

1. Determine your sample size

The larger the sample, the smaller the margin of error. Since the margin of error is proportional to the square root of the sample size, you’ll need to increase your sample four times to halve the margin of error, and nine times to get three times more accuracy.

2. Gather data from the population

Once you’ve determined your sample of the population you’re interested in measuring, you need to gather data. Sampling must be random. Failure to sample randomly will introduce bias into your results that the margin of error calculation cannot correct for.

Non-random survey designs are still useful, but do not calculate a margin of error for a non-random sample.

3. Choose a confidence level

You’ll include a z-score in your formula. This is the count of the number of standard deviations between the value and the mean. It is based on how often it’s acceptable for the actual result to fall outside the margin of error you calculate.

The most common confidence level is 95%, where the real value of a statistic will be outside the margin of error only 1 out of 20 times. Higher confidence levels will generate larger margins of error and vice versa.

Select your confidence level and the corresponding z-score using the table below.

4. Find the standard deviation of your sample

The most accurate formula for margin of error uses the standard deviation of the population for whatever statistic you’re calculating. However, population characteristics for many statistics, especially for many survey questions, are unknown, so we use the standard deviation of our sample instead.

Calculating the standard deviation of the sample is as easy as downloading your survey data and using the Excel formula =STDEV(). To calculate the standard deviation of one of your survey statistics, just add the column that contains the data between the brackets in the formula. For example, if your data is in Column B, enter the formula =STDDEV(B:B) in an empty column. 

5. Apply the standard deviation formula

The formula for standard deviation is:

MOE = z σ/√n

In other words, multiply the z-score by the standard deviation divided by the square root of the number of cases.

We know n (number of cases) from Step 1, z (z-score) from Step 3, and σ (the standard deviation) from Step 4. We just need to solve the calculation to find the margin of error. Here’s an example of how to do that in a spreadsheet formula:

The last step is to multiply the z-score by the standard deviation.

Let’s look at a hypothetical example.

If you have a target audience of 5,000 customers, this is the population size. Due to the cost of conducting a complete survey, you may want to choose a sample size of 500 participants (n). To calculate the margin of error, you will use the above method and substitute the values. We’ll use a confidence level of 95%, so a z-score of 1.96.

(σ) is the standard deviation = 0.05

1.96 x 0.05 = 0.098

Square root of 500 = 22.36067

0.098 / 22.36067 = 0.00438

Margin of error = 0.00438

The margin of error is affected by sample size and confidence level.

How sample size affects the margin of error

A sample size is a subset of the population under study who are the direct participants in the research. The larger the sample size, the closer the results are to the exact representation of the population, as long as the sample is random.

The margin of error is calculated from the sample size because it is a measure of how accurately a statistic represents the full population.

Calculate your ideal sample size

Understanding the confidence level of a survey

The confidence level is a measure of how likely it is that the collected sample accurately represents the population of interest. For instance, a 95% confidence level shows that 5% of the surveys will not reflect reality. A confidence level of 95% has a corresponding z-score of 1.96.

Smaller z-scores from lower confidence levels create smaller margins of error, but a lower confidence level means you can’t be as sure that your margin of error is meaningful.

The margin of error is often mistakenly thought to be either completely accurate or not accurate at all.

One critical assumption in the use of the margin of error is that the sample is representative of the population. This is only the case if the sample is random.

As already discussed, bias can occur in several ways. Let’s look at some of the more common types in more detail:

• Non-response bias: if some people are more likely to respond than others, you may miss the data from those who chose not to respond so your data does not represent the full population.

• Survivorship bias: if you’re launching multiple surveys over time, those who complete the later surveys may be different in important ways from those who completed only the first few surveys.

• Response bias: poorly worded questions may influence respondents to answer in ways that don’t reflect the actual preferences of the population, for example, overstating their willingness to pay for a product or service.

Margin of error should not be calculated or reported when data is collected in a non-random way, as the sample does not represent the population it is drawn from. Imagine using the heights of players in a basketball game to estimate the average height of everyone at the game! The true population height would fall well outside the margin of error.

There are several ways to reduce the margin of error, including:

Increasing the sample size

Using a larger sample size will allow the surveyor to make more observations, thus enabling a more exact estimate of the population. The two factors have an inverse relationship. A larger sample size will create a smaller interval around the data sets and vice versa.

Using a lower confidence level

A lower confidence level will create a smaller margin of error, but you can be less sure of the results.

The margin of error gives us an idea of how close a mean value from a sample is to the mean of the population from which the sample is taken. A smaller margin of error from a random sample tells us the actual population mean is close to the value we found in our survey.

The margin of error is the value added or subtracted in a confidence interval, while the standard deviation is a measure of how far the data is dispersed from, or clustered around, the mean. The standard deviation is used in the calculation of the margin of error along with a z-score and the sample size.

The margin of error is written as a number with the plus-or-minus sign after the mean value, for example: “The mean is 12 ±5.” 

The margin of error can be used to create a confidence interval, written as a range of values above or below the actual results from a survey, for example, “The mean is 12, with a confidence interval of 7 to 17.”

The margin of error represents how close the sample mean is likely to be to the mean of the full population.