What is the true population value? Hypothesis testing and margins of error are here to help
We all know that some value for the percentage of the US public that supports President Trump’s ban on immigration from seven Muslim countries exists - but this number has not been revealed to us. Without the luxury of conducting a census where this question is asked, the best we can do is estimate this number from a survey or poll. And probability theory provides us with the tools for making a statement about the precision of this estimate.
For example, a CNN/ORC poll of 1002 Americans taken between 31 Jan 31 and 2Feb estimated that 47% of respondents supported President Trump’s ban while 53% opposed it. But does this poll prove that a majority of Americans oppose the ban? One way to test this estimate is to determine if the difference is significantly different from 50% or 0.5. This can be done by computing a z score for the observed value of support p, which in this case is 53% or 0.53, and the null hypothesis value of po which is 50% or 0.5. The z-score is found with the formula:
So what does this value of 1.90 mean? This z score is from the standard normal probability distribution shown in the picture below. The 0.95 is the area for acceptance of the null hypothesis that the po is 0.50. The blue areas of the chart show the rejection region for a two tailed hypothesis that the po is not 0.50. In order to reject the null hypothesis, the z score would have to be less than -1.96 or greater than 1.96. Because our observed z score of 1.90 is in between these two z scores, we cannot conclude that the US population proportion of those opposing Trump’s travel ban is different from 0.50 or 50%.
If we cannot conclude that the true population proportion is different from 50%, can we still make a statement about what the true population proportion is? Yes, we can do so using a confidence interval. The formula for the confidence interval is given below.
The second half of the equation is:
This half gives the margin of error of the estimate which is often quoted in media reports of polls and surveys. The critical z value of 1.96 is multiplied by the standard error of the estimate for the margin of error for a 95% confidence interval. The margin of error for the CNN poll is plus or minus 0.031 or 3.1%. This gives an upper confidence bound of 56.1% and a lower confidence bound of 49.9%. Because the upper and lower bounds contain 50% we also cannot conclude with 95% confidence that a majority of the US public opposes Trump’s travel ban using this poll - although it comes close.
The 95% confidence interval is the universally agreed upon standard for estimating the precision of estimates from surveys. This standard comes to us from academia, as most scientific journals will not accept results from inside the 95% acceptance region. Ways to improve the precision of the estimates include increasing the sample size of the survey or changing the type of hypothesis being tested. For example, if we were to use a one tailed hypothesis that po is greater than 50% while the null hypothesis is that it is less than or equal to 50%, the criteria for significance changes. In this case, the critical z value is 1.645, as is illustrated in the image below. We reject our null hypothesis if the z value is greater than +1.645. Since our observed z-score is +1.90 we would reject our null hypothesis and conclude that po is greater than 50%.
There are many things that can influence the precision of survey and poll data: Sample size, the method of selecting the sample, the way in which the questions were administered (phone, a person interviewer, computer, et cetera), the way the questions are worded, and more. The methods discussed here are mostly dependent on sample size. The larger the sample size, the smaller the margin of error will be. In the poll we discussed, a sample size of 582 was not enough to conclude that a majority of the US population opposed Trump’s travel ban in a two-tailed hypothesis. This sample size was sufficient to make that conclusion in a one tailed test. The one tailed hypothesis test is recommended when there is prior reason to believe that that the actual population proportion is either greater than or less than 50%. Otherwise the two tailed approach is recommended.
Image: Thomas Hawk.