Last week I did a short presentation for a NewMR session called “Looking Forward.” This post is a summary of that presentation (with an “improvement” or two). You can view the presentation at the NewMR website.

I was trained as a historian and so I tend to view contemporary and even future events from a historical perspective. In this case, the title references history in two ways. First, the title is taken from a 1965 book with the same name by Peter Laslett in which he shows that some of what we believe to be historical “facts” don’t hold up when we go out and collect real data. In his case, that real data were records of baptisms, deaths, marriages, etc. collected in country churches in pre-industrial England. In doing so he demonstrated that, despite Shakespeare’s Juliet, women were not typically married off in the teens; most households were not comprised of large extended families; and the peasants did not live perpetually on the edge of starvation.

But I also chose the title to remind myself how simple and elegant research once was compared to how complicated and sometimes ugly it has become. A hark back to the days when virtually every client’s problem could be solved with a well-designed and executed survey.

But alas, the world is different now and the proposition here is that the practice of contemporary market, opinion, and social research is the result of four interrelated trends that have played out over about the last 50 years.

1. The relentless march of technology

Below is a rough timeline of the major deployment of technology in the search for better data and insights (at least that was the rationale):

• From paper and pencil administered in person or via mail to
• representing the questionnaire as a computer program administered via various electronic channels to
• harvesting data that already exists.

• 

The timeline also shows our expected next wave—automation and applications of AI.

But has this ongoing deployment of new technologies led us to better data or to richer insights? It has shortened cycle times and generally lowered costs, but the question of data quality is an open one and leads us to the second trend.

2. The unmooring of research from its scientific foundations

Historically, survey research rested on two main scientific pillars: representivity (based on probability theory) and measurement (based in mostly in cognitive psychology). Both have been abandoned in contemporary market research.

The term “sample quality” has been redefined to mean hygiene rather than representivity.  It is more about identifying fraudulent respondents, ferreting out speeders, and verifying identity than it is about accurately representing the target population. To quote Andrew Grenville in The Insights Revolution:

Sample is research’s Achilles’ heel. What started as a slow, downward slide in sample quality is now a race to the bottom, at blistering speed. Unless we quickly course-correct. it is sure to end in a terrible, and potentially fatal, crash.

Questionnaire design has fared no better. It, too, has been redefined and now emphasizes usability and engagement rather than measurement accuracy with a clear focus on what we need to know. The debate now centers on issues such as mobile first, use of color and graphics, game-like features, and asking simple straightforward questions with vague wording and unspecified recall periods. “Which brands have you seen ads for recently?”  We no longer focus on question wording, framing, and context, and questionnaires are seldom tested before they are put in the field.

3. The Gospel of Innovation

Then there is the constant search for the next big thing spurred on by a new generation of professional evangelists preaching disruption as an end in itself. Innovate or die! We are seeing new entrants that differentiate primarily on technology rather than data quality or insights. And there always seems to be at least one new technology on the horizon best characterized as a solution looking for a problem to solve. (The obvious current candidate is blockchain.) Yet despite all the sturm und drang few of these new methods achieve significant penetration. Even GRIT, the ongoing survey of those paying close attention to all these new innovations, finds relatively little adoption and, in some cases, declining adoption over time.

4. The loss of public confidence in market, opinion, and social research

In parallel with all of this we have lost the confidence of the public whom we depend on for what we do. This loss of confidence has been showing up for decades in declining cooperation and in widespread suspicion of the validity of results in published polls. Work by GRBN is documenting the erosion of trust in our capacity (or willingness?) to meet our obligations to protect the privacy and confidentiality of those few people who are still willing to complete our surveys.  

How do we stay relevant and sustain our industry over the long haul?

There is little sign that these four forces are weakening.  In fact, they seem to be getting stronger and I don’t think that’s a good thing for what we do.  Being fast, cheap, high-tech, and wrong is not the answer. Rather, we need to find ways to reconnect to science. If the classic survey paradigm no longer works, invent a new one. But “some data is better than no data” is not a paradigm. It’s a surrender to market forces that may well end badly for us.

To say it more plainly, we need to stop pretending that our work is better than it is. All the data we work with, whether collected via a survey, harvested from social media, or generated by a massive machine learning exercise on a client’s data warehouse is flawed. It’s our job to understand those flaws while still making sense out of what we see. To quote the statisticians Stephan and McCarthy in their book, Sampling Opinions:

Samples are like medicine. They can be harmful when they are taken carelessly or without adequate knowledge of their effects. We may use their results with confidence if the applications are made with due restraint…Every good sample should have a proper label with instructions about its use.

We should do the same for every bit of research we do, and not just about the sample.

Of course, acquiring the capacity to evaluate data, document its biases, and factor those into the conclusions that we draw is a challenge in itself. It is a skill to be learned and the core of it is an understanding of the basic principles that are the foundation of good quality research. Researchers take this perverse pride in having “fallen into” research with little or no background or training. Yet they are surprisingly resistant to learn what they don’t know, to go back and study the basics that are as relevant today as ever before.

Finally, there is the challenge of restoring the public’s loss of confidence in our work. That’s a heavy lift and probably begins with earning back some trust.  We will not earn back that trust through association public relations campaigns alone, but rather in how we behave, how we treat those whose data we collect and analyze. It is now more important than ever that we separate what we do from direct marketers and a tech industry that essentially uses our personal data to harass us, to fill our screens with evidence that they know way too much about each of us. Crossing that bright red line between research and marketing is more tempting than ever, and not crossing it has never been more important.  

Our future depends on it.