Fox News Voter Analysis Full Methodology Statement

FOX NEWS VOTER ANALYSIS FULL METHODOLOGY STATEMENT

The Fox News Voter Analysis (FNVA), conducted in partnership with the Associated Press, provides a comprehensive look at voting behavior, opinions and preferences as America votes.  It is based on surveys conducted in all 50 states by NORC at the University of Chicago, as well as actual voting results by county collected by AP.

The FNVA survey encompasses interviews with an estimated 140,000 registered voters and is conducted Oct. 29 to Nov. 6, and continues through the end of voting on Election Day.  Both voters and nonvoters are interviewed to provide a full picture of the election, including why some Americans voted while others stayed at home.

FNVA combines respondent interviews from three data sources: (1) a random sample of registered voters drawn from state voter files; (2) a sample of self-identified registered voters conducted using NORC's probability-based panel, which is designed to be representative of the U.S. population; and (3) a sample of self-identified registered voters selected from nonprobability online panels.  Interviews are conducted in English and Spanish.  Respondents receive a small monetary incentive for completing the survey.  Participants selected from state voter files are contacted by phone and mail, and can take the survey by phone or online.

The FNVA survey of voters and nonvoters nationwide is compiled from results of the 50 state-based surveys and a nationally representative survey of about 4,500 registered voters conducted on NORC’s probability-based panel.  It includes approximately 35,000 probability interviews completed online and via telephone, and 95,000 nonprobability interviews completed online.

In 25 states, FNVA is based on roughly 1,000 probability-based interviews conducted online and via phone, and roughly 3,000 nonprobability interviews conducted online.  In 25 additional states, FNVA is based on between 750 and 1,000 nonprobability interviews conducted online.  Consistent with Fox News standards that recommend against relying on nonprobability samples, the FNVA only reports nonprobability samples after they have been calibrated either with actual vote in a state or probability samples.

Sampling Details

Probability-based Registered Voter Sample

In each of the 25 states in which the FNVA includes a probability-based sample, NORC obtained a sample of registered voters from Catalist LLC’s registered voter database.  This database includes demographic information, as well as addresses and phone numbers for registered voters, allowing potential respondents to be contacted via mail and telephone.  The sample was stratified by state, partisanship, age and race.  In addition, NORC attempted to match sampled records to a registered voter database maintained by L2, which provided additional phone numbers and demographic information.  After the matching, NORC had phone numbers for 86 percent of sampled records, including cell phone numbers for 60 percent of records with a phone number.  Prior to dialing, all probability sample records are mailed a postcard inviting them to complete the survey either online using a unique PIN or via telephone by calling a toll-free number.  Postcards are addressed by name to the sampled registered voter if that individual is under age 35; postcards are addressed to “registered voter” in all other cases.  Telephone interviews are conducted with the adult that answers the phone following confirmation of registered voter status in the state.

NORC’s probability panel

During the initial recruitment phase of the panel, randomly selected U.S. households were sampled with a known, non-zero probability of selection from the NORC National Sample Frame and then contacted by U.S. mail, email, telephone and field interviewers (face-to-face).  The panel provides sample coverage of approximately 97 percent of the U.S. household population.  Those excluded from the sample include people with P.O. Box-only addresses, some addresses not listed in the USPS Delivery Sequence File and some newly constructed dwellings.

Nonprobability Sample

Nonprobability participants are provided via an online panel.  Digital fingerprint software and panel-level ID validation is used to prevent respondents from completing the survey multiple times.

Weighting Details

The FNVA employs a four-step weighting approach that combines the probability sample with the nonprobability sample, and refines estimates at a sub-regional level within each state.  The 50 state surveys and the probability panel survey are weighted separately and then combined into a survey representative of voters in all 50 states.

National Survey

The national survey is weighted to combine the 50 state surveys with the nationwide probability panel survey.  Each of the state surveys is weighted as described.  The probability panel survey receives a nonresponse-adjusted weight that is then adjusted to national totals for registered voters derived from the U.S. Census Bureau’s November 2016 Current Population Survey Voting and Registration Supplement, the Catalist voter file and the Census Bureau’s 2017 American Community Survey.  The state surveys are further adjusted to represent their appropriate proportion of the registered voter population for the country and combined with the national probability panel survey.  After all votes are counted, the national data file is adjusted to match the national vote for members of the U.S. House of Representatives within each state.

State Surveys

First, weights are constructed separately for the probability sample (when available) and the nonprobability sample for each state survey.  These weights are adjusted to population totals to correct for demographic imbalances of the responding sample compared to the population of registered voters in each state.  The adjustment targets are derived from a combination of data from the U.S. Census Bureau’s November 2016 Current Population Survey Voting and Registration Supplement, Catalist’s voter file and the Census Bureau’s 2017 American Community Survey.  Prior to adjusting to population totals, the probability-based registered voter list sample weights are adjusted for differential non-response related to factors such as availability of phone numbers, age, race and partisanship.

Second, all nonprobability sample respondents receive a calibration weight.  The calibration weight is designed to ensure the nonprobability sample is similar to the probability sample in regard to variables that are predictive of vote choice, such as partisanship or presidential approval, that cannot be fully captured through the prior demographic adjustments.  For states that include probability samples, the calibration benchmarks are based on estimates from the probability-based registered voter list sample in that particular state.  For states without probability samples, the calibration benchmarks are based on county level estimates from a multilevel regression and post-stratification model that incorporates all probability and nonprobability cases nationwide.

Third, all respondents in each state are weighted to improve estimates for sub-state geographic regions.  This weight combines the weighted probability sample (if available) and the calibrated nonprobability sample, and then uses a small area model to improve the estimate within sub-regions of a state.

Fourth, the survey results are weighted to the actual vote count following the completion of the election.  This weighting is done in 10-30 sub-regions within each state.