You researched your product, messaging and campaigns, analysed the findings, put the conclusions into practice, and still nothing improved?
You may have asked the wrong people, in the wrong places.
That one mistake can make research do more harm than good. Especially when you look at a small slice of reality, the convenient one close at hand. The places you have not examined may hold the information you need to grow your business.
Before checking the results, check whom you asked
Sampling sounds like something from a university statistics course. In practice, it concerns anyone trying to learn about their customers.
The question is simple: can the people whose opinions you collected actually help you answer the question you set out to investigate?
- If you want to improve service for existing customers, talking to them is a good starting point.
But…
- If you want to understand why other people are not buying, you also need their perspective.
Be careful about who volunteers to speak
One particular trap is self-selection bias: participants volunteer for a study, and those willing to respond may differ from those who remain silent.
A similar distortion arises when you invite only selected people, such as current customers. The selection mechanism is different, but the question remains the same: whom are you leaving out, and how does that affect your conclusions?
Three examples of how we can fool ourselves
1. You ask customers whether your offer is good
You send customers a survey. Most are happy, praise your service and say they would recommend your business. Time to open the champagne? Yes and no. That praise is a reason to be pleased. But you sent the survey to people who are still buying from you.
What about those who left? Received a proposal and chose a competitor? Visited your website, could not understand it and closed the tab? You do not have their opinions.
Where is the mistake?
To become your customer, someone had to:
- find you,
- consider your offer good enough,
- accept the price,
- decide to work with you,
- and, to stay longer, have a reason not to leave.
That is a lot of conditions filtering potential customers out. People could have dropped out at any of those stages.
It is worth finding people who were filtered out along the way. Learn what they compared, what was missing and when they decided against buying. Was price the problem? Or could they not work out what they would get for their money? The answers from people who dropped out may be uncomfortable, but that is precisely the point!
2. You develop your product around the loudest users
A company sells an online tool with free and paid versions. Its user group regularly requests new features: a report, an integration, additional settings.
The team listens, implements changes and collects praise. Somehow, revenue is not growing.
Where is the mistake?
Perhaps the most active users are on the free version and have no intention of paying? Perhaps paying customers need something else, but are busy working and do not participate in the group?
Before rebuilding the product, check who is suggesting changes, how they use the tool and whether the same problem affects the customers you want to serve.
Ten comments may point to an important problem. On their own, however, they do not tell you how widespread it is or whether solving it justifies a month of work.
It is easy to confuse the visibility of an opinion with its importance to the business.
3. You show your website to people who have known you for years
Your employees, friends and long-standing customers look at the new website. Everyone understands what you do.
Great. Except they knew the answer before opening the website.
They can fill in missing information and decode industry jargon. A new visitor has only what you put in front of them.
So show the website to people who fit your target audience but do not yet know your business. Ask them to explain, in their own words, what they can buy, whom it is for and what they should do next. Without prompting.
If you need five minutes to explain the offer, you have just found a clue about what the website may be missing.
Here, the choice of participants overlaps with the curse of knowledge: people familiar with your business may miss the difficulties a newcomer faces. I discuss this further in The curse of knowledge: when knowing too much stops you selling.
More responses can reinforce a mistaken conclusion
A thousand responses look more impressive than twenty. It is easy to assume that a result based on that many answers must be reliable.
Yet a thousand responses from your followers still come from people who chose to follow your brand and then chose to respond.
A larger sample can reduce random variation in the results. It does not, by itself, eliminate the systematic exclusion of part of the market.
Before sending another survey, check whether you need more answers from the same group or answers from someone else.
How can you check this in your own business?
Work through three questions the next time you analyse your results.
1. What decision do you want to make?
“I want to understand my customers” is too vague. “I want to find out why companies drop out after receiving a proposal” helps you decide whom to interview and what to ask.
The more specific the question, the easier it is to check whether your data actually answer it.
2. Who is in your data, and who is missing?
Separate current customers, former customers, people who declined a proposal and contacts you never managed to speak to. You do not always need to study every group. You do need to know whose perspective you require.
Do not assign motives to people who have not responded. Silence may result from poor timing, the wrong contact person or an undelivered message. On its own, it does not tell you that your offer is too expensive or that the market has no money.
3. How will you reach the missing perspective?
Instead of asking the same satisfied customers again, invite people who recently ended their relationship with you or declined a proposal to a short conversation.
Ask about a specific decision: what did they compare, what concerned them, and what did they eventually choose? Compare their answers with behaviour: the stage at which they dropped out, their order history or their use of the product.
Do not assume every former customer is unhappy. They may have closed a project, changed their needs or lost their budget.
Remember that, once again, only some of those invited will respond. A few conversations can reveal a problem and help you form a hypothesis. They do not automatically establish what percentage of the entire market thinks the same way.
Record what you still do not know, too. This helps distinguish verified information from a convenient guess.
To sum up: ask the people you do not hear
The easiest people to ask are those close at hand. Satisfied customers, active followers, friends and your own team.
Their answers matter. But if you want to understand why you are not winning more customers, it is worth reaching out to those you have not yet convinced.
Before changing your price, adding a feature or rebuilding your offer, ask:
Who is missing from these data, but should be included?
You can spend months improving your offer based on feedback from people who already buy it. Meanwhile, the reason other people do not buy will still be outside your survey.
Sources
- Rolf Dobelli, The Art of Thinking Clearly, the chapter on self-selection bias. The inspiration for examining how the selection of people we observe affects our conclusions.
- Pew Research Center, For Weighting Online Opt-In Samples, What Matters Most? (2018). On the limitations of samples whose participants volunteer, and why increasing sample size does not automatically remove selection bias.
