Further Reading Account for unequal response rates. The response rate for a
survey is the percentage of eligible people in the sample who completed the survey. Response rates vary because some kinds of people participate in surveys at higher rates than others. When survey researchers know some charac- teristics of nonrespondents, it is appropriate to weight to adjust for these differences in response rates. Typically, very little is known about people who fail to complete a survey, but one thing that is routinely known is where they live. To adjust for unequal response rates, weight by the inverse of the response rate in each area covered by the survey—and do the same for each other group for which response rates can be separately calculated.
The bottom line is that most real-world surveys need properly constructed weights in order to be accurate.
Compare to benchmarks. Poststratification is weighting to
make survey estimates match known population bench- marks, such as the percentage of men and women in the earlier example. The first step in poststratification is choos- ing which “benchmark” characteristics to match. The best way to do this is compare the composition of your survey sample to that of the greater population using every statis- tic you can find for which you know the population’s char- acteristics with reasonable certainty. Typically, these will be statistics for which there are official population data, such as demographic characteristics from the decennial census or from major Census Bureau surveys such as the Cur- rent Population Survey or American Community Survey. If your weighted survey sample differs from known popula- tion characteristics by more than a few percentage points in terms of age, sex, race/ethnicity, education, or other key variables, then it is a good idea to adjust the weights to cor- rect the largest of those errors. Rake. Raking is a method to perform this adjustment.
To rake sand smooth, you repeatedly draw a rake across it in different directions until the lumps are all smoothed out. To rake data, you do something analogous: Apply the cor- rection factor for one variable, then another, then another, and then repeat the whole process until the errors are gone. Typically, fixing one variable (such as the sex distribution) will introduce new small errors in another (such as the age distribution). But if we fix sex, then age, then sex again, then age again, repeating five or ten or fifty times, the dif- ferences from benchmarks will get smaller and smaller until they fade away, like ripples raked away in the sand. One controversial topic in weighting is the occasional practice of raking to match a target distribution of party
“How to Analyze ANES Survey Data”
http://www.electionstudies.org/resources/ papers/nes012492.pdf
identification. Matching a known population characteristic is appropriate, but raking to match party identification is unwise unless you are absolutely certain of the distribution of party identification in the population being surveyed. Party identification changes over time, is subject to mea- surement error, and is far more volatile than population de- mographics. For these reasons, academic survey researchers are skeptical about weighting to party identification. Review and adjust. After raking, it is important to scruti-
nize your weights carefully. Repeat the benchmark com- parisons and make sure that the survey gives more accu- rate results than before. Check for extreme values in the weights, which might cause statistical outliers, and cap ex- tremely high weight values, such as those more than five or six times the average. Also, importantly, a statistician should examine the design effect—the increase in variance (or mar- gin of error) caused by weighting. If weights have produced an unreasonably large margin of error, then the weights should be re-computed using fewer variables in raking, us- ing fewer categories in those variables, or imposing a lower cap on the weights.
Use design-consistent statistics. Finally, it is important to analyze weighted data with appropriate statistical methods. Because real-world surveys do not use simple random sam- ples, it is usually not optimal to use statistical methods that were invented for them when analyzing survey data. Con- sumers of poll data should ask analysts if they have used design-consistent estimation methods that account for weights and design effects in producing analytical results, from topline statistics to sophisticated models. An outline of appropriate methods for design-consistent analysis can be found in “How to Analyze ANES Survey Data,” which is available online. (See above for Web address.) The bottom line is that most real-world surveys need
properly constructed weights in order to be accurate. Sur- vey researchers, data analysts, and clients alike are increas- ingly insisting on these methods to improve the accuracy, and thus the value, of survey data.
Matthew DeBell, Ph.D., is director of Stanford operations for American National Election Studies.
June 2011 | Campaigns & Elections 31
Page 1 |
Page 2 |
Page 3 |
Page 4 |
Page 5 |
Page 6 |
Page 7 |
Page 8 |
Page 9 |
Page 10 |
Page 11 |
Page 12 |
Page 13 |
Page 14 |
Page 15 |
Page 16 |
Page 17 |
Page 18 |
Page 19 |
Page 20 |
Page 21 |
Page 22 |
Page 23 |
Page 24 |
Page 25 |
Page 26 |
Page 27 |
Page 28 |
Page 29 |
Page 30 |
Page 31 |
Page 32 |
Page 33 |
Page 34 |
Page 35 |
Page 36 |
Page 37 |
Page 38 |
Page 39 |
Page 40 |
Page 41 |
Page 42 |
Page 43 |
Page 44 |
Page 45 |
Page 46 |
Page 47 |
Page 48 |
Page 49 |
Page 50 |
Page 51 |
Page 52 |
Page 53 |
Page 54 |
Page 55 |
Page 56 |
Page 57 |
Page 58 |
Page 59 |
Page 60 |
Page 61 |
Page 62