Are You Running the Right Survey for the Wrong Reason?

The goal of a survey is to objectively test a hypothesis. Don’t allow bias to creep into your surveys, either in your design or in your respondent pool.

You’ve decided to run a survey and you’ve brushed up on the best practices for survey design. Now you must answer the single most important question of this entire process: Are you running a survey to test your hypothesis or to prove it?

The difference may seem subtle, but its implications are enormous. To test a hypothesis is to design an unbiased survey that may either validate or invalidate the underlying assumptions.

To prove a hypothesis is to collect data to support a predetermined narrative. This perhaps conjures an image of a villain twirling his pencil-thin mustache as he plots a devilish scheme. In practice, it’s not nearly as nefarious. When you have a hypothesis, you typically believe in it strongly backed by some level of research you have already conducted.

In other words – whether you know it or not – you are biased.

When you are biased, you can inadvertently build that bias into your research surveys. To avoid this you need impartial input. Someone not invested in the project should review the questionnaire for leading questions and language that can influence the respondent and alter their responses. If you are outsourcing your survey, it is incumbent upon the research organization to help you avoid these pitfalls. They have no investment in your research other than to ensure you get good results.

Okay, so you run an unbiased survey with an intent to test rather than prove. What happens if the results come back disproving your hypothesis?

Results that run counter to your expectations are more impactful. Yes, it’s great to get results that validate your assumptions. However, invalidated assumptions mean that something was wrong in your initial research. This outcome is substantially more important; you can avoid making a poor decision that wastes money, time, and resources.

It’s not just the survey design. You must also consider the respondent pool. In the same way, you want to avoid bias in your design, you also must avoid bias in who you invite.

Beyond the Customer

Let’s say that you are interested in knowing what people think of a new service being provided by a gym. Naturally, you survey the gym’s membership. You learn that people are generally pretty excited about the service and usage will increase substantially. Great results, right? Well, maybe…

Devout loyalists will tell you what you want to hear. They are members for a reason – they have already shown to have an affinity for your type of services.

Who’s missing?

Surveying casual gym-goers provides you with a sense of interest from less serious users. Members from competing gyms can tell you how competitive the new service will be to capture new members. Lapsed members can give you an estimate of how many member “win backs” you can achieve. Non-gym users can help you understand how much of a lure the new service will be.

The universe of respondents is much broader than just members, and to ignore their perspectives is to ignore critical data points that could shape the success of the new service.

The Takeaway

Surveys are a tool to aid in the decision-making processes. If you design a survey that proves a hypothesis rather than objectively tests it, your decisions will be biased and your previously confident decisions will lead to frustratingly failed outcomes.

---