How Gen AI is Revolutionizing Market Research with Stefano Puntoni

[00:00:00] Stefano Puntoni: When I, we look at the, you know, GI applications in business, everybody’s focusing on how we can do or we do better. I call this like, you know, doing things better, but I think it would be better to focus on. What else can we do? Basically doing better things, so the, the task really shouldn’t be just automating the things we’re doing today.

[00:00:20] MRII Announcer: Welcome to MRII’s Insights and Innovators podcast, where we talk to top market research professionals to get their inside stories about innovative and enduring best practices. Now here’s your host for today’s episode. 

[00:00:34] Nick Graham: Hi, and welcome to today’s episode, how Gen AI is Revolutionizing Market Research. I’m your host Nick Graham and our guest today

[00:00:41] Nick Graham: is Stefano Puntoni. He’s the professor of marketing at the Wharton School and one of the leading behavioral scientists studying the business applications of artificial intelligence. Stefano is a co-author of two widely discussed Harvard Business Review articles. I’m sure they’ve made their their way onto your LinkedIn feed, exploring how generative AI is [00:01:00] transforming market research, including the rise of synthetic data and digital twins.

[00:01:04] Nick Graham: Stefano, thanks so much for joining us today. 

[00:01:07] Stefano Puntoni: Thank you for having me, Nick. Pleasure to be here. 

[00:01:10] Nick Graham: Great. Well, this is obviously a very hot topic right now, and having read your articles, I know this is gonna be a very thought provoking, uh, uh, discussion for our audience. I think it’ll excite some people. It might scare other people, but I think fundamentally, I think what we’ll talk about today will help us rethink a lot of our assumptions about what market research is, how it’s done, and even some of our assumptions around AI as well.

[00:01:32] Nick Graham: So with that preamble, uh, love to get into the conversation today. So. I think starting out, you know, Stefano, a lot of people, I think, including myself and my peers, as we talk about a AI and market research, I think a lot of people immediately go to efficiency, right? Efficiency and making the way we currently do things faster.

[00:01:53] Nick Graham: Faster surveys, faster, cheaper analysis, more automated reporting. But the thing, something that really strikes me [00:02:00] as I look at how you’ve written about gen AI within the context of market research is really that this opportunity. For it to change what market research is, how we think about it, not just how efficiently it is done today.

[00:02:13] Nick Graham: So maybe you can share a little bit about your thoughts on, uh, this more, this bigger reframe of gen AI and market research. 

[00:02:21] Stefano Puntoni: Yeah, I think, uh, the conversation around generative ai, uh, is often focused on efficiency and focused on productivity. This is not true only for market research. I think it’s true basically for almost all, uh, enterprise deployment of gen ai.

[00:02:35] Stefano Puntoni: And, um, I get why that is the case and, uh, I think, you know, obviously if we can pursue efficiencies and create a productivity, you know. Profit driven companies ought to be doing that, so there’s nothing wrong with it. But, uh, it is important to realize that the full benefits of gen AI will have to involve more than shaving off some time here and there.

[00:02:58] Stefano Puntoni: It’ll have to involve doing [00:03:00] things differently. To achieve things that were difficult to achieve or impossible to achieve before. So in the context of marketing research, like I said, this is a general point. This is true for any kind of gen AI deployment. In the context of marketing research, you can think of a few.

[00:03:14] Stefano Puntoni: One is to think about to not, um. Uh, time saving in terms of saving money by, in terms of, uh, uh, just in time insights. So there are things that we would, uh, uh, like to act upon had we known this in time and often we don’t because maybe then the social media storm has already broken out or whatever it might be.

[00:03:35] Stefano Puntoni: And the GAI might give us a way of detecting things, uh, and, you know, acting upon things. Sooner and faster, which may be, you know, very important, uh, uh, for a number of reasons. That’s one. The other one is there are data, uh, types or, or samples or whatever it might be that are hard to get. And so, uh, in many situations, the alternative to, [00:04:00] um, uh, you know, synthetic data, so to speak, is not human provided data, but it’s no data.

[00:04:06] Stefano Puntoni: And so, uh, in many situations you might now base your decision or insight in situations where. Largely we’re basing them on anecdotes and gut feelings. This may be because the decision wasn’t important enough to warrant market research. You know, clearly a well done market research study is expensive, and companies even the most data driven one, so to speak, taking most decisions based off on insight that come from anecdotes or whatever we did yesterday, not data because they haven’t got the budget or, or even the.

[00:04:40] Stefano Puntoni: You know, the bandwidth it could be. So now potentially you could support with some evidence, um, many more types of decisions. Um, third, you have samples like, uh, you know. In B2B in healthcare, in many industries that are really hard to get, imagine that you’re a B2B [00:05:00] company, um, and your target audience is procurement directors.

[00:05:04] Stefano Puntoni: What you’re gonna do, you’re gonna, you know, fill the survey with a thousand respondents. Yeah, forget it. Not gonna happen. The best you can hope for in many situations that you gather a panel of five people through some, you know. You know, harassing people on LinkedIn and eventually you spend half an hour talking with them.

[00:05:20] Stefano Puntoni: You know, that’s for, you know, in many practical purposes, that’s the best you can hope for. And now you could do something much better than that, basically. Or in some industries, you know, companies are not even legally allowed to talk to certain parties matter. You know, if you are a European healthcare provider, you or a pharma company, you can’t talk to patients basically.

[00:05:39] Stefano Puntoni: And so, uh, that would get around, uh. Uh, a lot of this, uh, legal and maybe even, uh, you know, ethical things. There are things that you would like to ask people about when maybe you don’t think it’s appropriate to do so, uh, because it could be traumatic or for whatever reason, and you might be able to get some insight through that.

[00:05:55] Stefano Puntoni: So this is all, and then finally, the, maybe the most interesting category [00:06:00] is, uh, you might be able to achieve entirely new types of data. Data can never get before. And so I make a couple of examples. One is that idea of digital twins, so building a digital replica, a real individual, and so now. You know this old idea in sale that you can make a first impression only once might actually not be valid anymore to the extent that you are able to calibrate your pitch, calibrate your messaging, using the digital twins before you actually, so you can iterate, and then when you think you got something that should work well, you go into it.

[00:06:33] Stefano Puntoni: For real, I haven’t seen. We, we don’t have as of yet, good, rigorous evidence supporting how well that works. But, uh, you know, it could work maybe. And so that’s interesting. And another one is that we can basically build the social systems in, uh, um, you know, vitro, so to speak, meaning we can. Assess the evolution of social systems in a basically [00:07:00] sandbox environment that you can create and populate with, uh, um, you know, basically synthetic, um, people, you know, constructed off the back of an LLM, and you can observe how they interact.

[00:07:11] Stefano Puntoni: You can observe how market share, how communities form, how competition is shaped by incentives, by policy, by innovation. And, uh, this is a big frontier in say, for example, sociological and psychological research. And we could never do anything like that. You cannot randomly assign people to a high versus low tax regime.

[00:07:30] Stefano Puntoni: The best you can observe is, you know, historical data will happen, but you cannot do experiments like that. Uh, maybe you can, uh, try to leverage kind of like field or cause experiment. I mean, economists have done a lot to be able to study things in the wild, but I think this is a brand new opportunity. So there’s a lot of stuff like that, you know.

[00:07:47] Nick Graham: No, no, I agree. And I think what, I mean, think what’s, what’s interesting. And what you’re saying is, again, I think we’re very much focused on the world as it exists today and the data sets that we have today and the the research methodologies and approaches we have today. But I think what [00:08:00] you’re saying is this affords us an opportunity to actually.

[00:08:04] Nick Graham: Do things in a very different way in the first place. So maybe there’s a speed to insight and a speed to action, which, you know, from most corporates or, or clients or agency partners is, is still a critical barrier, right? Which is, I need to be able to act quicker on these, on these insights. And then I think there’s, you are raising an ability to do things, to be able to.

[00:08:23] Nick Graham: Access audiences to access data, to iterate on learning. Um, a lot of things we would want to do, but just we’re not able to do, or so to your point, simulate scenarios in a way that we can’t easily do today. So to me, it almost opens, opens the aperture to expand the way we think about market research. 

[00:08:43] Stefano Puntoni: Yeah, I, I think about it, um, in a broad sense when I, we look at the, you know, JI applications in business.

[00:08:50] Stefano Puntoni: Everybody’s focusing on how we can do or we do better. I call this like, you know, doing things better, but I think it would be better to focus on. [00:09:00] What else can we do? Basically doing better things. So the, the task really shouldn’t be just automating the things we’re doing today. It should be automating the things that we are not doing at all.

[00:09:09] Stefano Puntoni: I think that’s 

[00:09:10] Nick Graham: right. Exactly. And I think to your point, it’s a great opportunity. I feel like there’s so much, and, and to all the reasons you said, I think there’s a big focus on efficiency because it’s, it’s a. It is compelling. It’s an easy story, I guess, to tell, particularly for, you know, companies that are looking for productivity.

[00:09:27] Nick Graham: But at the same time, I I, I think this only works if it, and I believe it can if it draws or drives the effectiveness, right? We can make better decisions, better quality decisions, more holistic decisions as a result of, um, of, of gen ai. 

[00:09:41] Stefano Puntoni: Efficiency is easy because we, we know what we’re doing today. So basically it’s just asking the question, can we do it a bit differently?

[00:09:48] Stefano Puntoni: Effectiveness is hard because you need. You need imagination. 

[00:09:53] Nick Graham: Right? Exactly. Totally. So, um, maybe let’s talk about synthetic data now because I think, you know, that’s [00:10:00] one topic that gets people, um, either very excited or very skeptical or a combination of the two. Um, I think you mean the articles and, and papers that you’ve written.

[00:10:09] Nick Graham: You’ve talked very clearly about synthetic data, so this idea of. Of data that’s not collected from surveys and interviews, but it’s more that’s generated by our AI to mimic, mimic survey data, mimic interview data to mimic human preferences and behavior. What I still think. There’s a lot of misunderstanding and misconception about synthetic data, what it is, what it isn’t, how it should be used, how it shouldn’t be used.

[00:10:35] Nick Graham: So I’d just love your thoughts on, as a, as an insights professional, how should we really be understanding synthetic data? And its, and its, um, contribution, I guess. 

[00:10:45] Stefano Puntoni: Yeah, I think there are a lot of misconceptions, um, around the synthetic data. I think one we already just discussed, which is, is a focus on efficiency instead of, uh, thinking about what it could do about effectiveness.

[00:10:55] Stefano Puntoni: That’s one. But, uh, um, there are a bunch of others. I think one of them [00:11:00] is to. I think that what we do today or have today, it’s, uh, the end of the road. And, uh, I’ve seen a lot of people say posting on, uh, LinkedIn where they go and chat a pt. They ask Chacha PT to make a, you know, a prediction about what the person would think, and then it gives an answer.

[00:11:16] Stefano Puntoni: It even an nonsensical or very vague or very bland or whatever it might be. And then the conclusion of the post is dismissing. L lms, you know, outta hand saying, you know, this is uh, BS and never gonna work. That’s not wise in that, in that this technology is still improving very rapidly. That’s one thing.

[00:11:33] Stefano Puntoni: Second, leveraging gene AI for effective market research is going to require the development of many complementary innovations, and it’s gonna take us time to learn how to use it. Innovations in terms of. The use cases and how we should you go about them, but also seems statistically, I mean, what kind of, uh, hybrid sort of bi Asian like statistics do we need to, uh, deploy when we have both human provided data and synthetic data?[00:12:00] 

[00:12:00] Stefano Puntoni: Um, what kind of, uh, uh, models and fine tuning or whatever. And there’s a lot of people working on this, and I think we’re making, you know, every month there’s some interesting insights that, uh, um, are improving the way we understand how one can use it. But it’s gonna be a long road. It’s gonna take a, you know, a decade.

[00:12:18] Stefano Puntoni: So people should, uh. Not see where we are today as the end point is, we are just scratching the surface. So that’s one. Um, the other one is that I think a lot of inside professionals feel almost uniquely threatened by this technology because almost like taking, threatening the professional identity in some fundamental ways.

[00:12:37] Stefano Puntoni: I think that’s one reason why you often see this almost angry kind of, uh, uh, reactions by, uh, some people on, on against this technology. And I can totally understand where they’re coming from. I’ve written extensively about, uh, the experience of identity, threat with, uh, artificial intelligence is, was actually my first line of work into this domain of behavioral science of ai.[00:13:00] 

[00:13:00] Stefano Puntoni: And so that’s totally natural at the same time. You know, it’s a bit like Pandora’s box. You can’t wish it away. Um, that’s one. So like the ostrich approach is not gonna lead you anywhere. But I think a bit more positive than that. I don’t think this is gonna be a human replacement story in many ways.

[00:13:18] Stefano Puntoni: There’s gonna be some of that. And I think obviously if companies can improve productivity, that will have an impact on, you know, the budget. It’s gonna have an impact on, you know, um. Some of these professional services, uh, um, industries that support market research. But at the same time, to me, the value of human expertise is not gonna be diminished.

[00:13:38] Stefano Puntoni: If, if anything, if an inside professional, uh, who is asking the right questions, can draw the right insight, understand basically what the priorities should be. And I would go about finding out, uh, you know, uh, intelligence that can support decision making that. Expertise should become more productive with ai.

[00:13:56] Stefano Puntoni: And, but from basic economics, we know that when an asset becomes more [00:14:00] productive, you want to use more of it. So if anything, I think there should be more opportunities for skilled professionals. Um, obviously, you know, um, market research is an industry, it’s very easy to do bad market research. And uh, and of course if that’s a space you’re working on, then there is a threat where synthetic data might end up being taking over a lot of that.

[00:14:20] Stefano Puntoni: But, uh, if you are, um, you know, a skilled. Professional doing, uh, advanced work in a way that, uh, few other people can do. I do think that, uh, you have a great future in, uh, with the generative ai. You need to just start learning about it, experimenting and uh, and um, you know, figuring out how that, uh, expertise that you have can be almost multiplied and amplified by this new 

[00:14:46] Nick Graham: Exactly.

[00:14:46] Nick Graham: And to your point, and what does, um. With the automation of some of this, what does that free up that human capital to act or that human expertise to actually focus on? Because you know, if I talk to any of my colleagues in the insights industry today, it’s still a huge amount of [00:15:00] people’s time is taken up with the doing part of research and analysis, right?

[00:15:04] Nick Graham: Where, um, I think everybody’s. Yearning to build, to spend more time on the, what’s the upfront problem we’re trying to solve for? What does it mean? How do we land it into, into action? And actually, to me this shouldn’t be feel as a threat. I understand why it does feel as a threat, but I think it’s, I think we need to pivot our focus to what does this enable me to now do and spend my time doing?

[00:15:24] Stefano Puntoni: You know, I think ultimately it goes back to this, um, overemphasis on efficiency. Because I think if, if all you see it is as productive and efficiency tool, you know, that’s code word basically for headcount reduction. And then of course people threaten. But I think if we were focusing more on what it enables, um, in terms of capabilities that we then the d conversation.

[00:15:46] Stefano Puntoni: Yeah. 

[00:15:47] Nick Graham: One of the things, and you touched on it, uh, as you were talking about synthetic data. I think one of the things I’ve seen is, is exactly those two reactions I’ve, I’ve heard. Um, seen on LinkedIn and I’ve heard some, uh, people in the [00:16:00] industry almost revel in rejecting it. Revel in showing how wrong it is.

[00:16:04] Nick Graham: Right? As you say, there’s a, there’s a, there’s a truth because the, the technology is necessary where it will be in future, but there’s also a sort of a defensive mechanism of like, let me just, you know, to your point, in a worldwide feel threatened by it, lemme just prove how wrong this is and how it can’t possibly replace, um, uh, custom research.

[00:16:21] Nick Graham: On the other hand, you know, ’cause I’ve got colleagues, uh, and peers who are working with data and analytics partners, or IT partners where there’s almost the opposite, which is there’s a push from those teams to say, well, why isn’t everything synthetic data? Guys, you have a lot of data already. Why do you need to go and do any more research anymore?

[00:16:38] Nick Graham: Why can’t you just use what you have and, uh, create synthetic data from that? That answers all of your questions. I, I think my sense is. Neither of those extremes are particularly helpful. The answer is somewhere in the middle, right about and more context dependent. There’ll be some context and scenarios, in which case synthetic data might be much better than our current [00:17:00] approach.

[00:17:00] Nick Graham: And then there’ll be others where synthetic data won’t ever in the near future, really replace the need to go and do fresh research. But I’m just kind of interested, you know, where your thoughts are in terms of, uh, in terms of that debate. 

[00:17:14] Stefano Puntoni: Yeah, no. So I totally agree. I think just like it’s not gonna be human expertise versus, uh, you know, artificial intelligence.

[00:17:20] Stefano Puntoni: It’s not gonna be human provided data with synthetic data. It’s gonna be and not, or, and so there will be context where, you know, synthetic data can be trusted to be the main. Basis for insight. Um, and if you want, we can start going through the list of what that might look like, although we don’t know yet.

[00:17:39] Stefano Puntoni: And that list is gonna change. But there is also obviously reasons to believe that there will be, um, you know, lots of room and need for human provided data. Um, now what’s interesting there, 

[00:17:53] Nick Graham: sorry, I was gonna say, could you give an example of a, a context in which you think synthetic data might be end up being better [00:18:00] versus, you know, a scenario where you might still need.

[00:18:03] Nick Graham: Fresh human research. 

[00:18:05] Stefano Puntoni: I mean, I’ve made already a bunch of examples. I think when I was talking about the effectiveness piece. I think all of those apply. But imagine, I dunno, I’m, I’m making up something I, you know, just on the spot. But imagine you’re a private equity company and you’re thinking of investing in a mature company.

[00:18:21] Stefano Puntoni: In a mature industry, okay? I dunno. Automotive soft drinks, I mean, you name it. And now you wanna do, as part of your due diligence process, you wanna do a perception map. To try to understand how these brands, uh, you know, compete with each other. I would think that probably you can do it just as well using, uh, LLMs.

[00:18:40] Stefano Puntoni: And then it would be doing, uh, you know, um, using a sample of people at the fraction of the cost and the speed and potentially with much more flexibility. So I think that’s will be one case where it’s really about, you know, the, um. The industry is fairly stable. There’s a lot of evidence about these brands in the training data because they’re B2C, so there’s a lot of [00:19:00] stuff on, you know, the open internet and in magazines and whatever that have been used to train the models so they understand, you know, the difference between, I don’t know, Ford and Chrysler or whatever it might be.

[00:19:10] Stefano Puntoni: Then, um, you know, just tapping into that knowledge, people don’t really often think about it in this way. But, uh, when you use a language model, you’re basically leveraging the training data. And the training data is people, you know, so it’s people talking to each other on Reddit or X or in, uh, you know, being interviewed or people writing books, and it is all kind of stuff in there.

[00:19:30] Stefano Puntoni: And so, um, the human insights provided by LMS are not coming out of a machine. They’re ultimately coming outta humans. It’s just some kind of aggregation. Um, done by the statistics of what fin size might look like given the prompt, um, with regard to the other extreme. Actually, you were mentioning a domain of innovation with something really new, you might, uh, want to talk to people and clearly that is true.

[00:19:56] Stefano Puntoni: For example, you want to, uh, explore unmet needs. [00:20:00] LMS might be good at that because people might talk about problems, uh, that might be a blind spot for industry. So I don’t dismiss synthetic data for unmet need analysis. But, uh, uh, you could also imagine that, uh, certain niche populations that are not very well represented in training data or anything like that, they might be stuff.

[00:20:17] Stefano Puntoni: That are not being met could also be that, you know, the world is changing. And so because of whatever social or technological change, the training data doesn’t quite reflect the needs of this customer group very well. So if that is the case, then you do want to talk to, to real people, obviously. Uh, but that doesn’t mean that, for example, gen I can’t, cannot be used for innovation.

[00:20:39] Stefano Puntoni: People say, oh, it doesn’t represent the human, uh, um, real humans very well. In the case of innovation, you don’t care. About approximating humans very well. You, you care about good ideas and, and if you can find them talking to an LLM, even if the LM is a poor presentation of a consumer is, if its idea is good, you should use it.

[00:20:59] Stefano Puntoni: So I, I think [00:21:00] in that sense, I think the purpose of using synthetic data free innovations is a little bit different from the context of using it for uh, uh, let’s say more additional marketing research where you might want to. Have, um, data that, um, you know, um, statistically representative for example of what the population of, of interest thinking.

[00:21:19] Stefano Puntoni: Yeah, 

[00:21:21] Nick Graham: yeah. No, no, it makes sense. Um, I guess let’s move and talk about digital twins. ’cause I think that’s the other big area that, uh. Uh, is on everybody’s, uh, minds right now. Now, as I understand the way you describe. So, so for, and for those who don’t understand, I guess, or don’t, haven’t, don’t know a lot about it.

[00:21:41] Nick Graham: So, digital twins are effectively AI generated replicas, right? Of, of customers, um, that firms can, that you can interact with, that you can test ideas against, that you can learn from, as you said before in the sort of like iterative. Way before, before it may be ever going into research or certainly ever going into market.[00:22:00] 

[00:22:00] Nick Graham: I think it’s a really interesting space and it’s one that comes up a lot in discussion. I’m interested in your thoughts on how digital twins, a bit like synthetic data, how they could really change the way we think about market research. 

[00:22:14] Stefano Puntoni: Yeah, so this is another area where there’s a lot of misconceptions and, um, people use terminologies like synthetic personas and digital twins often quite interchangeably.

[00:22:22] Stefano Puntoni: The way that I think about it is actually quite different. It’s just a definitional issue, but, uh, you know, I can call it whatever you want, but I think it’s good to be clear exactly about what we’re talking about. And synthetic persona, the way I define them, is basically a, um, a. Basically a, a digitally invented customer.

[00:22:41] Stefano Puntoni: Basically, it’s a profile you create based on information about your target market. So you, you have, you know, demographic, psychographic, behavioral information about your target segment, for example, and you enter all that information. Into an LLM to generate an instantiation of what a customer might be.

[00:22:58] Stefano Puntoni: But that, uh, you know, [00:23:00] they say Nick Graham that I’m creating out of in air doesn’t actually exist. It is not real person. But you could imagine a digital twin, which is, I actually have a lot of data about, uh, customers, individual customers. Maybe I’m a company with a large database of existing customers for whom I know lots of information about their life and their interaction with the firm, or maybe.

[00:23:20] Stefano Puntoni: It’s a panel, uh, for women. There’s been long track record of measuring information, whatever it might be. And then now what I try to do, I try to construct a digital approximation of that real person. So now the, the digital twin of Nick Graham is actually your digital twin. It’s not some kind of like, you know, uh, sort of, uh, um, invention, but uh, something that is trying to get at what that particular person is thinking.

[00:23:46] Stefano Puntoni: And that is a quite a different. Way of thinking about it. I mean, obviously there are parallels in the, in the, in that you’re using the same foundation models to create both. So they, they’re originating from the same language model, but, um, uh, the way that you should think about that data is quite [00:24:00] different.

[00:24:00] Stefano Puntoni: And right now I think most of the work has been done around. Synthetic personas and, um, relatively little, uh, is still understood about digital twins. It’s not even clear that there is oftentimes much of an advantage in built in digital twins rather than relying on those synthetic generic personas. Um, so at this point I think we need just need a lot more research.

[00:24:23] Nick Graham: And one thing, I mean, I think one thing that I often hear from folks in the industry is. Human beings incredibly complicated and multifaceted. How, how can an LLM ever possibly replicate the, you know, magical, crazy complexity of individual human beings? So, you know, what are your thoughts on can it, should it, does it even matter that it can’t replicate that that perfectly?

[00:24:49] Stefano Puntoni: Yeah. So, so last, whether it matters, it depends very much on what you’re trying to achieve. I think. Um. This is actually a very interesting topic and one that in [00:25:00] my teaching I, I explore in quite some depth is basically, uh, almost boarding on philosophical. I think. Um, now just one, uh, one premise. I think it’s something that people, this is another misconception actually.

[00:25:12] Stefano Puntoni: So we are going through a long list of these misconceptions. I hope people find it useful. Um, one of them is that, um, I call it the anthropomorphic trap. And so you observe a, uh, machine producing an output that is incredibly human-like and we cannot help basically. But, uh, assuming that the process that produced that output is also human-like, and that doesn’t have to be the case, lms don’t think the way we think.

[00:25:42] Stefano Puntoni: I think they think it’s intelligence of a sort, but it’s a very. In a way, narrow form of intelligence and, uh, it’s a subset of the type of, uh, cognitive capability that we have. And so, um, in a way the, from a practical standpoint, the question, you know, the proof is in the pudding. Does [00:26:00] it do a good job?

[00:26:01] Stefano Puntoni: Okay. So that, that’s something that we just have to test and tweak and tune and learn, which use cases and which do not. I mean, this is basically part of the, uh, exploration that academics and startups and, and established firms are doing right now. But there’s a bit of a more fundamental question about whether the mechanics of these language models can support.

[00:26:23] Stefano Puntoni: Um, really understanding psychological mechanisms. Right now, the answer is probably no, insofar as, um, they lack the, um, let’s say cognitive, um. Mechanisms that shape our own cognition, or at least some of them. Um, but again, it’s an open question. I think there’s, um, you know, a graveyard of, uh, uh, statements being made about AI cannot do this only to be proven six from now.

[00:26:52] Stefano Puntoni: But basically, if you think about the way I’m thinking about right now, this is basically, and I haven’t written up, uh, these ideas, it’s already. [00:27:00] Very kind of like, uh, still up in the air, but just to share an initial insight. If you go back to the history of psychology and, uh, and this le equation, that behavior is really a function of the person and the situation.

[00:27:13] Stefano Puntoni: Um, basically when you think about the impact of the situation on behavior, it’s really how psychological mechanisms are engaged by situational cues and by the context. And that may be difficult. For, uh, LMS to capture because they don’t have the same, um, uh, cognitive building blocks operating. Um, but if you think about, you know, how people.

[00:27:39] Stefano Puntoni: Think and behave and translate that into beliefs, into values that might be able to capture it very well. Because I think a lot of those beliefs are very well articulated in language, and so they should be very well captured in the training data. So in a way, I think if you ask them, you know, do you care about this and that, or what do you think about this?

[00:27:58] Stefano Puntoni: It should [00:28:00] be, you know, quite good at approximating it. If you’re thinking about, okay, how does. This belief changes as a function of, uh, showing them these price points versus that by process’ more like an experimental design setting or a conjoin setting. That sounds to me like it’s gonna be much harder to do that.

[00:28:15] Stefano Puntoni: That doesn’t mean that we are not getting there. Uh, I don’t know if we will, but I’m just saying that it’s not gonna be straightforward. Um, 

[00:28:22] Nick Graham: so something you said sort of struck me, which can, and it, and it resonates with what I read in the articles as well. So I mean, I think you have some good examples in the article where there’s a.

[00:28:32] Nick Graham: Ey double blind study where the synthetic personas sort of reproduced with, was it 95% accuracy? Um, the real survey results you had, uh, another example where digital twins were able to replicate people, survey responses, um, the, the ones that they did two weeks later. So there’s, to your point, maybe, I think you’re right, we have to acknowledge the fact that the mechanism of getting there is very different from, from a human.

[00:28:58] Nick Graham: However, to your point. [00:29:00] If your proof is that the actual outcomes are highly predictive, does should we be worrying lesser? Should we be worrying less about how it gets there? As long as it can? Obviously there’s obviously something then that the analyst is able to do to predict the pattern of. Uh, responses and behaviors and actions.

[00:29:17] Nick Graham: Maybe it doesn’t matter as long as it’s still able to do that, and we have to maybe worry a little bit less in the short term about the fact that it’s doing, it just, it’s doing it in a very way that’s very different from the way that human being would do it. 

[00:29:28] Stefano Puntoni: Yeah. So that would be the pragmatic standpoint, right?

[00:29:30] Stefano Puntoni: Say in the end, does it work? It doesn’t work. If it works, what do you care how it does it? But I do think that we, we care how it does it first because I’m academic and for me, understanding why things happen is crucial. Um, but um, but also I think it does point potentially to the kind of use cases that are most likely to be good, you know, like, um, um, replicating a survey about beliefs.

[00:29:56] Stefano Puntoni: Like in case of the EY survey or you know, a lot of other [00:30:00] data look very impressive. Those ones are of the kind that I just illustrated, this idea of eliciting beliefs and so I do expect those to work very well. If you’re thinking about something that is more about the situation side of the layer winning equation, more like how do they react to changes in contextual, how do those engage different cognitive processes to produce a differential answer?

[00:30:21] Stefano Puntoni: Those ones are gonna be harder and so then. You know, price optimization to me is gonna be more difficult not to crack than, you know, belief, estimation, and approximation 

[00:30:32] Nick Graham: because you’ve got multiple factors. And, and again, I guess it really depends on what training data you can give it. If you can give it good models to build from, it’s gonna make it easier for it to predict the outcomes.

[00:30:43] Nick Graham: But to your point, um, it’s gonna be harder the more, I guess, layers of complexity that are gonna be in, in the decision of the decision making or the, uh. The response process. Right. 

[00:30:53] Stefano Puntoni: And of course these things might change again, as the technology evolves. There’s an enormous amount of interest right now in what they call neuro symbolic [00:31:00] ai, which is basically trying to add some symbolic logic or, um, you know, more of a top down kind of, uh.

[00:31:08] Stefano Puntoni: Uh, system, you know, there’s a lot of discussion in computer science about the world models and mental models. For example, right now, transformer based language models, like, you know, those powering Chacha and Gemini and all those tend to have no such, um, you know. Mental models is the way we do, or, or symbolic logically.

[00:31:31] Stefano Puntoni: There’s some debate about as to whether this may be emerging in this network. And we are not gonna get into the details in the, into the weeds of that part because it gets quite technical quickly. But essentially, um, we don’t know if that can improve, but there are many efforts. You saw it with young Deun Living Method to form a new startup.

[00:31:48] Stefano Puntoni: Five years from now, these models might be architected quite differently and all of a sudden, very new capabilities may be emerging outta that. At, at this point we don’t know. 

[00:31:57] Nick Graham: So obviously, I mean, there’s a lot of uncertainty still, right? [00:32:00] And uncertainty about, um, what the technology we have today, and as you say, the way it’s gonna evolve.

[00:32:05] Nick Graham: If you are, if you are, if you were running an insights analytics team, uh, agency side or client side today, what do you think these developments requires to start rethinking today? You know, none of us can predict exactly what the future’s gonna hold for this technology, but what should we be starting to rethink?

[00:32:22] Nick Graham: Where should we be experimenting? Where should we be holding back? You know? So what, what advice would you have? 

[00:32:28] Stefano Puntoni: Yeah, I think this is still a phase of exploration, um, what we do in the first HPR article, which I would, uh, you know, maybe recommend everybody to take a look and, uh, think about. We have little diagnostic tool where basically we say, okay, can you think about, um.

[00:32:46] Stefano Puntoni: Um, elements of the research process. We, we have that as a column in the table, and then you’ve got capabilities of language models, which we have as a raw in the table. You can assemble your own framework. We have a four capabilities and three stages, [00:33:00] uh, set up. So it creates a four by three grade, but you can, you know, come up with your own whatever it might be suitable to you.

[00:33:07] Stefano Puntoni: And then for each of those cells, start exploring what use cases it might be. And we found it quite easily to populate every cell of that four by three grid with use cases. And then the question then becomes. You know, start chipping away and see whether, whether this can help doing, I dunno, desk research versus, you know, translating questionnaires or moderating interviews or running statistical analysis, summarizing results or, and then this goes on and on and on.

[00:33:34] Stefano Puntoni: There are many different types of applications. Some of them are gonna be more valuable than others. Some of them might turn out to be more of the, you know, shaving some time of the project kind of innovation. Uh, but some of them might be more transformative than that. 

[00:33:47] Nick Graham: And I think that’s a great ’cause. I feel like there’s a risk with any, like with any new technology that people jump into.

[00:33:54] Nick Graham: I, I saw this recently with the, with the client on, uh, digital twins. It’s like, we should go and do, we should go [00:34:00] and use digital twins. It’s like, but what’s the problem you’re solving for? So, to your point, mapping out the flow of your research analytical process and then thinking about what are the problems that it could solve for, to me, feels more like a more sustainable way to, to start to experiment.

[00:34:15] Nick Graham: Then just randomly going, I need to go and play with technology when I don’t really know what application I’m trying to drive it against. It doesn’t stop us as human beings. It’s very tempting, but yes. 

[00:34:26] Stefano Puntoni: If your, um, let’s say learning and exploration is dictated essentially by formal, I think that’s probably not a very good indication.

[00:34:34] Stefano Puntoni: Actually, we published a book, uh, two years ago called Fusion Driven Analytics 

[00:34:38] Nick Graham: on my bookshelf just here. It’s a great book. I’ll, I’ll add the link along with the HBR articles. It’s very good. Very helpful 

[00:34:44] Stefano Puntoni: we making there is that everybody wants to be datadriven and, you know, datadriven decision making, that’s what they call standard.

[00:34:50] Stefano Puntoni: And we say this is actually getting wrong. It shouldn’t be your decision making that is data driven. It should be your analytics that is decision driven. So in the end, the first question is, what are you trying to do? [00:35:00] Um, now my point about the grid, the four by three or whatever that might look like for a, um, for a given, uh, professional that is about.

[00:35:09] Stefano Puntoni: Trying to learn what you can do today. But I think if you’re a leader of, uh, say a large market research company or, or, or running a big department, uh, in a, in a firm, whatever it is, I think you ought to also start talking to people and explore what the future might look like. And it’s very uncertain with certainly do not know, but to some extent we are shaping the future we want.

[00:35:32] Stefano Puntoni: So I think being, uh, looking ahead of, uh, this year and next year and thinking about if it is true that this capability is gonna emerge in three years, what does that mean for the way that we are thinking about, and it could be that it’s gonna look entirely different whether we produce, uh, insights. I, I just don’t know, to be honest, but I, I think it’s something that, um, at least senior leaders in this organization ought to be, um, thinking and starting talking about.

[00:35:58] Nick Graham: And where should they be? I [00:36:00] mean, apart from your, your good self and uh, your colleagues, where should people be looking for inspiration, insight around where, where gen AI is going and what it might mean, I guess for market research. 

[00:36:12] Stefano Puntoni: It’s a overwhelming, and I think. What I hear a lot of people telling me our own students is saying, you know, I feel overwhelmed with this.

[00:36:20] Stefano Puntoni: There’s so many news, so many things happening. I don’t even know how to keep up. I don’t even know where to start. And, um, I mean, obviously I also experience the same, um, although I’m thinking about this video all day. And I’ve learned to be kind of zen about it. I mean, what can I do? I mean, I can’t read every paper.

[00:36:40] Stefano Puntoni: I basically do my best. I pay attention to the things the seem important. And I, you know, the rest, there’s a lot of background noise. And then, uh, I also admit that, um, I don’t know what I don’t know and, uh, and I live without with that. So I think you need to come to some kind of acceptance of the fact that you’re never gonna be completely on top of things.

[00:36:58] Stefano Puntoni: Uh, at least as long [00:37:00] as he moves this fast. Um, but, uh, I would say there’s a lot of interesting stuff being done, and not all of it is surfacing in, uh, newspapers and magazines and, uh, and um, you know, web, uh, blog and sub stacks and stuff like that. So I think. To pay a bit attention to the academic frontier, I think might be a, this is certainly a space where there’s a lot of interesting stuff happening and, uh, so follow people who are, uh, doing work in this space.

[00:37:29] Stefano Puntoni: And that can show what this technology can do, and there’s lots of them, uh, to choose from. And, um, maybe that could be an interesting way to stay engaged and learn maybe ahead of the curve. But ultimately, like I said, you have to also live with the fact that, um, you know, it is overwhelming and you have to.

[00:37:45] Stefano Puntoni: Yeah, 

[00:37:46] Nick Graham: and I think your advice of anchoring it in that grid of problems that I need to solve for, and then mapping back into that, you know, where the technology is today, where the technology might be in future, at least starts to give you. [00:38:00] A controllable way of starting to think through the next few years without feeling totally overwhelmed by every new, every new development.

[00:38:07] Nick Graham: Closing out on developments. I guess, and again, not asking you to totally predict the future, but what are some of the developments in this space that either you think will surprise people though, that you particularly are following and uh, and looking forward to? 

[00:38:22] Stefano Puntoni: I, I think it’s some we already mentioned like maybe development in, uh, the underlying architecture, architecture, language models.

[00:38:30] Stefano Puntoni: Um, but, um, I think one very interesting move has been how. Um, vibe coding might become more increasingly accessible to people without a lot of, uh, kinda more technical skills. You’ve seen, uh, Claude Cowork being released, uh, a couple of weeks ago, and so I expect that within a matter of months you’re going to have a very advanced, uh, coding agents that are very easy to use for people with zero coding expertise.

[00:38:57] Stefano Puntoni: Now, there are, there are dangers in that, um, [00:39:00] like, you know, the, the monkey, the typewriter kind of, uh, kind of issue. And so we, we have to see a bit how that works out. But I think, um, technology can be incredibly empowering. You know, it can give you the feeling, the opportunity to do things you never thought you could do.

[00:39:17] Stefano Puntoni: And so I’m very excited about that. 

[00:39:19] Nick Graham: Fantastic. Well, I think that’s a perfect way to leave it. Stefan. I thank you. This has been a, an amazing conversation. I’m sure that the audience will have gotten a lot out of it, um, learned a lot about where Gen AI is going, even though, again, we don’t know exactly where it will go, but I think you give some a good indication of what it can do today and where it might go in future.

[00:39:37] Nick Graham: And I think for me, my big takeaway is you’ve really helped us, really helped people rethink about, as you said, beyond just efficiency. Beyond just maybe some of those closing things that Gen AI can do faster slightly, you know, slightly more efficiently than today. How we can challenge it to really think about how we can expand the frame of market research using [00:40:00] Gen ai.

[00:40:00] Nick Graham: Actually, there’s a real role for things like synthetic data, synthetic personas, digital twins, to help us do market research differently, and as you said, in a way that. I feel excited and, um, energized. I think that, you know, as you said, it doesn’t necessarily mean it just replaces actually augments. It augments, it compliments.

[00:40:21] Nick Graham: There’s still clearly a role for fresh research and human expertise, but this, this will allow us to do some things. In the future that we can’t easily do today. And I think that’s a really exciting and energizing place for people to be. So thank you very much for helping us to see where we are, what’s coming, and where we need to rethink.

[00:40:38] Nick Graham: And thanks to everyone for listening to Insights and Innovators. Uh, big thanks to Stefano. Until next time, I’m Nick Graham. Thank you very much. 

[00:40:47] MRII Announcer: Thanks for joining the Insights and Innovators podcast for Market Research Institute International. Click subscribe to never miss an episode and visit us@rii.org for more market research insights.