Is research holding us back?

I think I managed to alienate the entire market research community here today. I was invited to speak at the MRIA‘s annual qualitative conference. The theme of the day was “Exploding Paradigms” and the organizers assembled a great list of people to talk about the forward edges of qual. I was really excited to see the list of topics and speakers, and it was a really well done event.

Despite the forward-looking nature of many of the day’s talks, I took a bit of an extreme position to make a point about the problems with much of the status quo. My talk was called “Is Qualitative Research Holding Us Back?” – a subject that I seem to go on about a lot recently.

The gist of it is that given the communications industry is in a period of massive change, it is a time when more than ever, we need to be grounded in an understanding of people’s evolving behaviour and needs. But research hasn’t been evolving at the same rate as culture, and instead of seizing the moment of opportunity, I feel despite the occasional ethnography or metaphor study, a lot of qualitative research still looks like the same old focus groups we’ve been watching for years. It’s based on faulty assumptions, has not kept up with cultural change or scientific learning about how the brain works, and may actually be hindering success.

I don’t assign blame to any party – I think we’re all guilty. Marketers have cut their research budgets to the point where most client research managers are too understaffed and overstretched to think about innovation; agencies have used research too selectively and selfishly for too long to have much credibility any more; and research practitioners are fighting for a piece of shrinking budgets and facing rushed timelines so they often lack the leverage to effect major changes. But that means there’s no one to really push for change. And this situation runs the risk of damaging qualitative research’s value and credibility at a time when it is most needed; and researchers, clients, and agencies need to work together to win that credibility back.

None of this should come as a shock to most people who’ve practiced or commissioned market research, but it’s still a tough subject to bring up to a room full of researchers. So I have a feeling I ruffled a few feathers. And admittedly I took a bit of an extremist position to make the point. But I also got some nice positive feedback so hopefully I’m still on a few Christmas card lists.

Published by jasonoke

Global Client Leader, WPP Married to @meredithoke. I have some kids. I travel. I eat. I internet.

22 thoughts on “Is research holding us back?

  1. Hi Jason
    We see the same in the digital realm, where personalities like Donald Norman and Jarred Spool have questioned if we are asking the right questions? (or getting the wrong answers).

    When many of our research models where developed 10 to 30 years ago the “user” were an employee experts on a 2-color monitor doing the same tasks (mainly textbased og spreadsheet) all day.

    Today we talk about participants, being exposed to hundreds of colorful graphic user interfaces on the web or desktop. Their attention span has shrinked to a few seconds and their demand and anticipation shot through the roof. Making it even more complex is the additional tools and activities they are engaging themselves in and the individual ways they want to use these.

    If we continue to ask the same questions we will continue to produce the same solutions. solutions we already see are to conservative and to clunky for an apt and demanding pariticipant.

    Therefore I certainly enjoyed your presentation and have taken the liberty of posting it on our APG-site (in norwegian unfortunaley :o)

  2. Hi Jason

    Just so I’m clear, are your issues with qual research based around the drawbacks with claimed behaviour and the dominance of one personality over the group, among other problems intrinsic to the discipline?

    As you point out, these are age old problems but well designed and professionall undertaken research can overcome these. No research methodology is perfect, but it is not just luck when research results are accurate. And while neuro research and the ilk are highly interesting techniques, but at this stage the cost far outweighs the benefit, in my opinion.

    I disagree with your position and feel research still has a fundamental place in a marketing plan (though the precise place will of course depend on what the plan is). It does frustrate me when people use research as a comfort blanket. Instead, it should be used as a reassurance or to tweak the marketing bod’s bright idea. For while it is the marketer’s job to convince the public, it does help to know where the general public stand on the notion of the proposition. And I believe qual exploration, followed by a quant analysis of the primary market segment, is the best way to achieve this.


  3. Thanks for the comments.

    Helge – of course feel free to share. Thanks for the kind words, even in Norwegian.

    Simon – my position isn’t that research isn’t valuable – I firmly believe it has key a place in the marketing plan. Properly designed research is one of the most important things a marketer does. My issue is that for something so important, most research design hasn’t kept up with changes in culture or science.
    The drawbacks of claimed behaviour are one huge part of it – not only do we often misrepresent our own behaviour; but because of the mind’s complexity, the simple act of being asked to think about things can change our answers and perspective.
    The thing is, the tweaks required to make research more valuable are often minor ones, but ones that are counter-intuitive to people stuck in the old frameworks. They require asking people questions that are less literal, less direct, and sometimes require asking people fewer questions altogether. They also require allowing respondents to have more fun and flow with their answers, giving up some control and direction in the research. These things can seem scary to people who don’t understand the reasons and benefits for doing so.
    It’s a lot easier, and easier to explain to your boss, to just ask people directly “which proposition statement do you like best.” Unfortunately it’s also completely the wrong thing to do.

  4. Thanks for the clarification Jason – I’m in total agreement. This is one of the reasons why we are increasingly moving towards a more ethnographic approach – allowing us to capture what a respondent does rather than what we claim to. Of course, this then requires the budget holder to overlook the rather steep cost and the possibility of the hawthorne effect coming into play.

  5. Hey Jason,

    I really feel that this article is right on the spot. It is something I too have spent a lot of time thinking about. One problem, I guess, that comes with more observational methods is the fact that to a client it’s not proof in the same way that clear answers on clear questions are. Those are often not really true/accurate/relevant/useful for many reasons (as you point out; what we say is not always what we mean and do).

    The change that will come, or have to come, Is that a client must trust the agency and its planning department equally much because of historical successes having arrived at strategies through whatever method they feel most suitable. Whether it’s observing only, talking with someone in a café and just thinking hard. And here’s another “problem”; the fact that making decisions on “gut feeling” is not considered qualified enough. I beg to differ as gut feeling is based on years of mistakes, successes, having read about stuff etc etc. We live this day in and day out and get payed to do it. A gut feeling (based on observations, talks, books etc) must be considered a qualified decision. That’s also the way qual research has to change or evolve. Always observing, but maybe less formal (no respondents being ready to answer questions). No team out observing but rather you having a problem to solve – living that problem for weeks. I mean, that’s how most of us function anyway.

    I don’t know, it’s hard to get everything down on paper and make it sound like an OK theory – but your article is MOST relevant and important and if it did shake the room, good.


  6. Market researchers should welcome this kind of constructive critique. And I have to say, Qualitative research in North America probably needs it more than others. Although what you presented could apply to all market research.

    An Admap editorial recently described market research as being “in crisis”. Given that the market research industry is growing steadily, the share prices of global research agencies remain strong under the circumstances and consolidation continues this is obviously an exaggeration. But I guess we can say there is a malaise. Your presentation adds to an increasingly loud criticism of our data collection and the interpretation of the data we collect. What seems to be coming under fire is confidence in the ability to predict behavior based on research – which is obviously the purpose of doing it.

    Plenty of people are talking about moving past the questionnaire – or discussion guide. And we are fortunate that technology is offering us better opportunities than ever before to do that: data integration, neuroscientific analysis and observation using digital technology. Much of this innovation is in Quantitative research. We need to understand the role of qualitative research in these initiatives. It should be significant.

    One of the problems in the market research industry is that we are wedded to our data collection methods because that is how we make much of our money. We sell data collection with a mark up to cover ‘insights’. And as research agencies grew and attempted to make their offering ‘scalable’ they developed models that are hard to change without undermining a revenue source. They also developed norms for those models based on data collected using the methods of the day. This leads to inertia. Ironically, Qualitative should be less subject to this than Quantitative research.

    For both methods the challenges are not insurmountable and innovation will prevail. Thanks for pushing that process along.

  7. I had the pleasure of attending Jason’s session at QRD Day. It was thought-provoking and I don’t see feather ruffling as a bad thing.

    I am a young qualitative marketing researcher thirsty to be operating at the highest standard possible. I see the research discipline as constantly evolving as new knowledge is gained and more advanced techniques are integrated.

    It’s easy to fall into the comfortable, scalable, operationalized routine of doing research – especially if you have a business to run and have been “doing it that way” for many years…

    I do feel it is a two way street between researchers and their clients. Suppliers would do some interesting stuff were they given the opportunity. It takes a bit of a leap of faith, though – on the client’s part as much as the supplier’s.

    Thanks, Jason, for keeping us on our toes and asking us to be accountable for our actions! It will make for better research…

  8. Good for you. I have been frustrated by most research and most researchers for many years. They, like most of traditional advertising, haven’t significantly changed their approach in 20+ years. Some innovation, and some clients brave enough to support the innovation, is sorely needed.

  9. Jason

    Great post. Wish I could have seen/heard your talk. Obviously the slides leave much out. Any chance you’d be willing to do live on an Internet Broadcast or maybe lay in the audio using Slideshare’s audio attachment tool?

  10. Glad you survived and the grumblings of a research renaissance are a brewing.

    I am planned to be in Toronto in a month or so. Should grab a coffee. I usually stay in the Germain and hang out at our office on Yonge and Bloor, so have the city well covered.

  11. Thanks for sharing your deck – some great stuff in there. Even without the voiceover of your presentation it made total sense. Couldn’t help being a little depressed that this is such revolutionary talk though. Surely the people who do research in the same way all the time can’t be finding anything new (both clients and practitioners). For us, it doesn’t take a massive change in approach to make a significant impact – all of our qual approaches these days break some of the conventions, but that hasn’t been revolutionary. We find that our clients expect that we do some things differently as cost of entry.

  12. Thanks for this. I am teaching an advertising research course next semester. I will use this to kick things off. (I recognized slides referencing the Malcolm Gladwell discussions from his Pop Tech talk, correct?) I think many things are having trouble with “keeping up with culture and science” … the problem is too many things start with research and therefore it holds a greater responsibility to keep-up.

    Thanks for getting the discussion going.

  13. I am an admirer of yours, and a researcher. I have a background in psychotherapy. You say that “despite the few ethnography and metaphor studies, a lot of qualitative research still looks the same.”

    While I do not disagree that ethnography and metaphor studies are an evolution of qualitative research, they have pitfalls similar to standard focus groups as well. Ethnography, for example, is great at showing us what brand of soap people really use, or how they actually prepare a dinner meal- no doubt about that, and that is more valuable information than hearing a respondent’s representation of it in a focus group when a client is seeking hard-nosed, actual facts about a client’s habits.

    However, qualitative research is often charged with finding-out about emotions, hidden meanings, feelings and uncovering unconscious motivators. This is where ethnography, as is practiced by most, still runs into the same problems – it is being used to and lauded for its ability to uncover facts, not emotions.

    What is interesting, from a psychotherapeutic point of view, is the lies or misrepresentations that people actually tell us in a focus group. I’m not referring to the fact that people mistake the brand-name of dishwasher detergent they use, but why they would tell us they wash their dishes every night, when they do it every three. What are they protecting when they tell us that? What image do they want me, as a moderator, to have of them? If they actually lived their misrepresentation, how would they feel? Does the ethnographer go in to a house and simply record the three days of dirty dishes, or is the ethnographer actually asking about the feelings those create? And if asking about feelings, how do you know the participant is telling the truth – and that’s the exact same predicament you find yourself in doing standard qualitative research.

    I’m being overly simplistic in the above, but I hope you get the point. When you deal in a medium that seeks to motivate people emotionally, and when you need to understand their innermost feelings to do so, you need to have a method, regardless of situation, that has a way of getting inside what the respondent is protecting. While I support ethnography for factual information, I’d rather have a person sit across from me and misrepresent themselves – that’s where the real learning about a consumer starts. Too many times we’ve been told not to take what the respondent says literally – the question is, what do we do after that…

  14. Jason – this is a great post! Even if people do not agree completely with your position, no one can argue that qualitative research is frequently too literal and rationale. I have been in the qualitative research industry for the last 7+ years and have always struggled with using conventional activities/questions/methods to really dig beyond the surface.

    I think a lot of this has to do with the types of methods we use. As a researcher, we are most familiar with point in time methods like focus groups, bulletin boards, and interviews. We try to use these methods to really understand the irrational – which even if we can achieve, does not address the concept of evolving behavior and needs. Also, I think that given the nature of the researcher-client relationship, it is hard to push for change and experiment when the client is paying the bill.

    The good news is that I think the industry is waking up. At PluggedIN ( we specialize in online market research communities which are a great way to really develop this understanding of people on an iterative basis. Having a captive (and engaged) audience allows us to try out new activities, push the envelop, and make research fun and entertaining. I think that as these ongoing and interactive methods become more prevalent, researchers will have the liberty of experimenting with innovative and entertaining techniques. Again, great post – I think conventions ultimately need to be challenged in order for us all to evolve.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: