Future of Feedback: Rethinking How We Ask Questions

At the Quirks Event in Chicago, Andrew Grenville of Maru / Matchbox shared research on research from his book The Insights Revolution: Questioning Everything.

How we interact with each other keeps evolving. Andrew started in the industry doing face-to-face interviews with paper questionnaires, before adapting the questionnaires to phone research, then to the web, and now to mobile.

Failing to keep up with changes in how people communicate can be a matter of life and death. When researching the Zika virus, public health officials in Brazil wanted to figure out the breeding grounds for mosquitos, where there were puddles and temporary bodies of water, so they could dry them up and eliminate the mosquitos. In order to do that, they did a survey asking for geotagged photos, including how many people lived nearby, so that they could prioritize bodies of water to treat. They sent out email invites and got 300,000 responses. They then texted people, and got an additional 400,000 responses, and the people who responded via text differed socioeconomically and by age. But in Brazil many people use WhatsApp instead of texting, so they sent out invites there and got 750,000 more responses. Had they used only one mode, they would have collected many fewer responses. Thanks to their broad reach, the effort to eradicate Zika was a success.

In most countries, this is a generational story – technology adoption is very different by age. For recent Maru / Matchbox studies, survey completion by device type is 55% by computer for 18- to 24-year olds, rising to 80% of those 65 and up. In contrast, smartphone usage drops from 38% for the youngest cohort to 11% of the oldest. Interestingly, tablet usage rises with age, perhaps as eyesight worsens, from 3% for 25- to 34-year olds to 9% for 65+.

People are receptive to other ways of doing surveys, but this varies demographically:

  • 62% have virtual assistants, 48% have used it, and 44% would try surveys on it, with Millennials making greater use of virtual assistants and being more willing to take surveys with them. Three quarters of Americans will have a virtual assistant by the end of next year.
  • 39% of Americans would prefer virtual voice surveys to text surveys.
  • And 50% of 18 to 34-year-old Americans are open to receiving text message invitations to surveys on their mobile phone, dropping to 24% of 55+.
  • While 22% are willing to provide video responses, only 64% of those were able to do video, and of those half the videos were off topic, had poor or no lighting, but the remainder had many great responses – but those will not be completely representative but can drive home some issues raised through other modes.
  • For an app-centered community of 350 respondents, only 21% would join a community to discuss investing, 14% would join if an app was required, and 5% would join and use the app. The requirement of downloading the app and using it diminishes the percentage of people willing to participate and would skew who was willing to participate to those who were younger with less money to invest.

The future is device agnostic design, which will require changing questionnaire design. “Simple questions will rule the future. How can we simplify? When Alexa asks the questions, how will it be?” Andrew then read out a grid, slowly, to laughter at the miserableness of the experience. You can’t do the big grids on a mobile screen: “It is ugly, it is hateful, no one wants to do it, you get terrible data.”

“The Yes/No is the only question that survives in a device-agnostic environment,” he said.

In his book, Andrew shares research out of Australia into the validity of binary formats to Likert scales. Comparing a binary brand rating to a scale, it takes the respondent three times longer to answer the scale question. Binary produces the same insight. He shared research where the responses to a select all that apply question, a binary choice, and a top-2 box from a scale produced different numbers but the same results for most and least popular and the relative popularity of the choices. For correlation with spend, a binary correlation is actually higher for most items, producing the same conclusion, with similar sensitivity.

In conclusion, “we must move forward, we can’t continue with the old ways, we have to shake things up. Binary simple questions can get you the same answer, still do what you do, provide reliable information that reflects reality even though you need to change.”

“The future will be fragmented – the way we communicate – and we can’t ask everyone to sit at a desktop and take a survey. Imagine what the situation in Brazil might have been if they didn’t use WhatsApp as an option: they would have missed half the responses and put lives at risk.” The need is to reach people where they are at, without putting restrictions on them – seamless, simple, easy.

2 thoughts on “Future of Feedback: Rethinking How We Ask Questions

  1. Completely agree with you that the future of questionnaire design is device agnostic design, especially as more and more types of devices, like Alexa, become popular to consumers.

    Although, I disagree that “The Yes/No is the only question that survives in a device-agnostic environment”. If anything, I think Alexa encourages respondents to talk more when answering questionnaires!

    When you fill out a text survey, I can see how you would fill out “Yes” or “No” answers. Writing takes more effort. If Alexa asks you a question, it’s easy to just talk back on and on. In fact, I think you’d have problems getting the respondent to stop talking!

  2. That’s a good point. There certainly is lots of opportunity for gathering more qualitative information using Alexa and her ilk.

    In the context of the talk, I was focusing on quantitative measures, particularly scales and rank order questions–and they certainly won’t translate well in a device agnostic world.

Leave a Reply

Your email address will not be published. Required fields are marked *