Alternative title: Surveys make entrepreneurs lazy.
Entrepreneurs tend to be skeptical about the notion of doing qualitative research as means to validate their business hypotheses and establish if there’s a real business opportunity. This attitude comes from the notion that, in order for information to be reliable, it needs to come in the form of tables and charts derived from quantitative research. “In ‘hard data’ we trust!” seems to be the mantra. This is particularly true for those working in the consumer space as tabulated data provides a condensed view of thousands of people at once, which is understandably, music to their ears. And what’s the best way to obtain the numbers that feed into those lovely pie charts? You guessed it: surveys.
The allure of surveys
Surveys make an appealing case: they allow us to gather data from thousands of people with relatively little effort. In the case of online surveys the temptation is irresistible as with just a few clicks we can setup a questionnaire, distribute it, tabulate the incoming data and visualize the results. What’s not to like?! Well, surveys are not the right tool for the type of job that needs to be done at early stage startups.
We don’t know what we don’t know
As obvious as it may sound this riddle-esque idea has an implication so subtle it escapes many: we can only inquire directly about known unknowns.
Think about it this way: we can only ask questions about the things we know we don´t know. When it comes to survey questionnaires, the range of themes addressed in those yes/no and multiple-choice questions (and their answer choices) are limited by our current knowledge of what the range of possible themes is. We are, by definition, leaving out key questions that could potentially unearth breakthrough insights simply because we are not aware of them (as if they were out of sight).
But wait! That’s why we have open-ended questions in surveys, so people can speak their minds, no?
In theory: yes. In practice: rarely.
Who hasn’t designed a survey questionnaire that included questions like “Why?” to cast a wide net and added “Other. Please Explain.” at the end of a set of answer choices (of a multiple-choice question) to cover their back? But after going through the rather bland and generic responses from those few kind people who after rolling their eyes made the effort of providing an answer, it becomes rather obvious that we were being naïve. By asking these tell-me-everything-I-don’t-know-about-you questions we assume that people are willing (or capable) of analyzing themselves on a given situation, and then articulate conclusion in a tidy sentence. And all this on a split second. There’s no way than will happen! It’s worth saying that using open-ended questions in survey questionnaires is not the issue, but rather how we use them. Open-ended questions yield better results when they focus on having people describing past events (e.g. “What happened that time when…?”) or behavior (“How do you…?”). But even the responses from questions such as these should be treated with a pinch of salt, especially those inquiring about people’s problems (but that’s the subject of another post).
Bumping into unknown unknowns
This is precisely where surveys fail and qualitative research has the upper hand. Bumping into unknown unknowns is what early stage startups should learn to do in a systematic manner. And there’s no better way to break free from the cage that represents our current knowledge of things (i.e. both known-knowns and known-unknowns) than by talking to people in the form of customer development interviews. It’s through (structured) conversations that we give ourselves the chance to be surprised (and many times inspired) by something someone says. It’s the ‘randomness’ of people’s stories that allows those I-would-have-never-thought-of-that-in-a-million-years! moments to happen, opening in front of our eyes a window of possibility which we didn’t know was there.
The right time for doing surveys
Simply put, surveys are most effective when we want to define the relative importance of things and to establish priorities. This means that we make the most of surveys only when we know (based on the learning from customer development interviews) which are the right questions to ask, and ideally, what are the most likely answers. For example, through surveys we can refine the demographic and psychographic profile of our target segments (from interviews we learned there was more than one), identify how widespread is a behavior (which unexpectedly emerged in the stories of a few people) and establish who are the main competitors (including those makeshift solutions people showed us).
The many excuses for not doing interviews
At the top of the list there is laziness. Plain and simple. It’s undeniable that preparing and conducting customer development interviews require much more legwork than running online surveys from a laptop, and sadly, this is one of the main reasons why entrepreneurs (even if they fail to admit it) prefer to look the other way. Others who consider themselves domain experts (usually due to their previous work experience) say that talking to people is something they’ve already done in the past or they believe there’s nothing new they could learn thus they would rather spend the time doing something else. A few others are simply unaware of how they could ‘do research’ besides the techniques they see being used by big established companies. But the truth is that no matter the excuse, failing to spend time face to face with prospects represents a risk very few startups can afford i.e. missing out on bumping into the nugget of information that will change their fate.