Last week the group investigating the failure of polls to accurately predict the 2015 British election released its final report. You can read about it on ResearchLive and download the report here. As expected, the report points to unrepresentative samples as the main culprit. And, as I wrote in a previous blog post, there are implications for MR if we choose to take notice and act on them.
In the final report the investigating team makes two recommendations (out of a total of 12) for obtaining more representative samples.
Longer field periods
For telephone this mostly comes down to “more callbacks to initially non-responding numbers (both non-contacts and refusals), and ensuring a better mix of landline and mobile phone numbers.” They also see potential advantages in longer field periods for online polls that include “more reminders, as well as differential incentives for under-represented groups, and changes to the framing of survey requests.”
The recommendation here is primarily for online polls that rely on non-probability sampling. The report calls out the need to “investigate new quota and weighting variables which are correlated with propensity to be observed in the poll sample and vote intention.” Put another way, they call for a better understanding of the behaviors and attitudes that drive self-selection and voting behavior, then use them in sample selection. They also correctly point out that we have not yet figured out how to do this as a matter of course, so longer field periods are “likely to prove more fruitful.”
This call for longer field periods is a stark contrast to proliferating business models in MR that promise results not just overnight but sometimes in a matter of hours. Technologies and processes that reduce cycle time in getting a survey to the field are one thing, but if the British election farce (and others like it around the globe) has a lesson to teach us it may be about the dangers of speed in data collection. Fast turnaround has an undeniable appeal to clients, but it’s also a roll of the dice.
The report is even more stunning in its assertion that almost 20 years into online data collection we still have not figured out how to draw an online sample capable or producing accurate and reliable data, or at least there is no generally recognized best practice for doing so. I think we understand the problem more clearly today than just a few years ago and have a general view of how to go about it, but doing it remains a major challenge. I’m not sure how much longer we can ignore confronting it.