Polling: “Margin of Error”

Posted by jns on June 24, 2005

This is not a particularly recent poll, although the assertion is still true. But that’s not the point.

The New York Times > Washington > New Poll Finds Bush Priorities Are Out of Step With Americans

The poll was conducted by telephone with 1,111 adults from Thursday through Monday. It has a margin of sampling error of plus or minus three percentage points.

Have you ever wondered where that “margin of error of + or – 3 % points” comes from, or why the weird number of people polled (which most people react to by thinking it’s much too small)?
The simple answer is simple. Follow along:

1. The sample size is 1,111
2. The square root of 1,111 is 33.33
3. 33.33 / 1111 = 0.03, or 3%

The answer is that simple. In random sampling from a uniform population, the best estimate of how good the average result is will be

+/- [sqrt(N) / N] = [1/sqrt(N)],

where N is the number of [statistically independent] samples.
The inverse works too. If you are told that the error is E%, then

N = 1 / (E/100)²

is the original sample size.
There is no mystery about this relationship between error and sample size in polls, and it is not what determines careful or “scientific” polling. It is simply an unvarying, mathematical result giving the best guess you can make about the error in an average calculated from random (i.e., statistically independent) samples taken from the larger population that one is trying to characterize.
The trick, of course, is in that bit about taking “random samples”. That’s the part that polling organizations work very hard at: to convince their customers that they (and they alone among their competitors) know how to take very good, very nearly “random samples” from any given population — all Americans, all likely Republican voters, all women under 18 who watch MTV, all men over 50 who eat chocolate ice cream at least twice a week, whatever group the poll’s sponsor is interested in.
All the work, or artistry (some would like to say “science”) goes into selecting the samples so that they will be randomly drawn from the population of interest; none of it goes into calculating the margin of error.
So now, when you hear a margin of error quoted, you can amaze all your friends by revealing the exact number of people who were asked the question, and sound amazingly clever.