Ask someone to describe their favorite movie. The answer has nothing to do with the movie.

A class exercise that reframed how I think about research design.

One morning, Judy Franks, our Media Insights professor, asked us to pair up with someone we didn't know that well. The exercise was simple, pick a favorite movie or book, and describe it in one or two lines max.

I picked a Bollywood movie I love, Dear Zindagi, and I said something along the lines of, "when life feels rough in the moment, knowing something better is coming gives me hope. Tough times prepare you for beautiful beginnings," and that movie reminds me of that.

The classmate I paired with didn't pause, she said "You're an optimist!" She was right, it is the most central thing about who I am. And the movie isn't necessarily about optimism, but my description of it was.

She described her favorite book, and by how she framed it, I guessed she is someone who values quality over quantity when it comes to the people in her life, she prefers ‘Few, real, deep connections’. We're great friends now, years later, and I can confirm it's true! 

I don’t remember the book she chose 😬, and she probably doesn’t remember the movie I picked, but we both learned something about the other that still stands true. The favorite movie or book might change with time, but that insight about what we value, that won’t.  

Because the movie or book itself didn't reveal anything, what we highlighted in those stories did.

But….

Most research isn't designed to catch that insight.

Think about the questions we rely on in consumer insights. "Select up to three words that best describe [brand]." "How appealing is this concept to you on a scale of 1 to 5." "What is most important to you when choosing a [category product]." "How likely are you to purchase."

These aren't bad questions, they give us data. But they all do the same thing. They point inward and ask the respondent to introspect, evaluate, and report back on their own thinking.

The thing is, most people can't do that accurately. Harvard professor Gerald Zaltman found that 95% of our decisions happen subconsciously. When asked directly, people give clean answers. Price. Quality. Convenience. The real drivers sit underneath, emotional and symbolic, and often invisible to the person themselves.

Data shows (ironic lol), that stated-preference research predicts actual purchase behavior with roughly 34% accuracy. Soo.. two out of three times, what someone says they'll do and what they actually do don't line up. Which we call the ‘say-do.’

They're not lying. They themselves don’t have reliable access to their subconscious reasoning. 

The Yellow Walkman is the cleanest example of this.

Sony ran a focus group for a yellow "sports" Walkman. Participants loved it. Cool. Fun. Sporty. Way better than the boring black one. Everyone said they'd buy it. At the end of the session, each person was offered a free Walkman to take home. Yellow or black. Their choice. 

Guess what….

Nearly everyone grabbed the black one. 🤨

They meant what they said in the room. But when the choice was real and unobserved, the truth showed up in what they reached for, not what they'd said literally five minutes earlier.

The researchers asked which color people preferred. They got an answer. It just wasn't a real one.

If instead they'd tried to understand what actually mattered to this group, they might have learned something more useful than a color preference. That these people value fitting in over standing out? That "cool" is something they admire at a distance but don't want to carry. That would have been an insight. The color answer was just data that happened to be right and wrong at the same time. 

One reframed question can change everything.

Researchers proved this at scale. In 2016, traditional election polls asked people who they planned to vote for. They missed five swing states and the election. Then researchers tried one change: instead of asking who you'll vote for, they asked what percentage of your social contacts will vote for each candidate. 

It was the same people, but one reframed question, and? The reframed question was nearly twice as accurate in the states that decided the election. 82% versus 46%.

They've since tested it across eight elections in five countries. The reframed question outperforms the direct one every time. Not because people know their friends' politics better than their own. Because pointing the question outward gave people room to say what they wouldn't say about themselves. 👀

So what actually works? 🤔

You can't replace every direct question :( surveys need them. They give you the quantitative structure a study runs on. That's not the problem.

The problem is when direct questions are the only thing in the survey, and we expect an insight to come out of them. Insight almost never lives in a scale rating or a checked box. 

It lives in the gap between what something is and how a specific person sees it. 

Zaltman knew this. His method doesn't ask people what they think about a brand. It asks them to choose images that represent how the brand makes them feel. One participant chose a sunrise for Starbucks. Not because of coffee. Because the brand meant warmth, ritual, and hope. You don't get there from "describe Starbucks in three words."

Two people can pick the same movie. One calls it a story about breaking free. The other calls it a story about healing. One person sees rebellion, the other sees repair. Same movie. The difference is the person.

Two things worth trying.

Your survey probably has 25 questions that give you data. Ask yourself if each question can be reframed in a way that can help you get closer to the truth. 

Second, consider what one well-designed question could add, one that doesn't ask the respondent to evaluate, rate, or rank, but gives them something to react to or describe in their own words. 

The quantitative data gives you their conscious reasoning and that one reframed question? That can help you get closer to the subconscious reasoning and close the say-do gap.

Next time you're designing a study, before you finalize the questionnaire, stop and ask yourself: is there one question in here that gives the respondent room to reveal something? If the answer is no, well, you might get all your data back and still miss the insight. 

The best insight I've found didn't come from a direct question. It came from giving someone room to reveal what they actually value.