The most expensive survey mistake is one you do not realise you have made until the data comes back and drives you toward the wrong conclusion.
Poorly written survey questions produce systematically skewed answers that look like real data. Respondents answer the question you asked, not the question you meant to ask. That gap between intended and actual is the source of most survey bias.
This guide covers the most common sources of bias in survey questions, how to recognise them, and how to fix them before your survey goes into the field.
Why Question Design Matters More Than Sample Size
Most researchers focus on sample size as the primary indicator of survey quality. Sample size matters, but it does not protect you from question bias. A sample of 10,000 people answering a leading question will give you a precise measurement of the wrong thing.
Question bias is structural. It is baked into the way the question is written, and it affects all respondents in the same direction. Unlike sampling error, which reduces as sample size grows, question bias does not reduce at all with more respondents. It just becomes more precisely wrong.
Leading Questions: The Most Common Problem
A leading question signals to the respondent what the “correct” answer is. This happens more often than researchers realise, often without any intent to manipulate.
Examples of Leading Questions
Biased: “How satisfied were you with our excellent customer service team?”
Fixed: “How would you rate the customer service you received today?”
Biased: “Don’t you agree that WhatsApp surveys are more convenient than phone surveys?”
Fixed: “Compared to phone surveys, how would you rate WhatsApp surveys for convenience?”
Biased: “How concerned are you about the risks of AI in research?”
Fixed: “What is your overall view of AI in research?” followed by a balanced scale
The fix is usually simple: remove the evaluative language from the question stem and move it into the response options if it belongs there at all.
Double-Barreled Questions
A double-barreled question asks about two things in one question, forcing a single answer to cover both. Respondents have no way to indicate they agree with one part and disagree with the other.
Example: “How satisfied are you with the speed and accuracy of our service?”
A respondent might find the service fast but inaccurate. Their answer to this question is meaningless because it is averaging across two distinct evaluations. Split every double-barreled question into two separate questions.
Loaded Questions and Assumed Premises
A loaded question contains an embedded assumption that the respondent may not accept. Answering the question at all forces acceptance of the assumption.
Example: “Since switching to digital research, how much time have you saved?”
This assumes the respondent has switched to digital research and has saved time. A respondent who has not switched, or who does not feel they have saved time, has no valid answer option.
Fix: Establish the premise as a filter question first. “Have you switched any of your research to digital formats in the last 12 months? Yes / No.” Then ask the follow-up only to those who said yes.
Scale Design: Getting It Right
Response scales introduce bias when they are unbalanced, when the neutral point is unclear, or when the scale length is inappropriate for what is being measured.
Balanced Scales
A scale should have an equal number of positive and negative options. An unbalanced scale like “Very satisfied / Satisfied / Somewhat satisfied / Dissatisfied” has three positive options and one negative. It will reliably produce inflated satisfaction scores.
Balanced version: “Very satisfied / Somewhat satisfied / Neither satisfied nor dissatisfied / Somewhat dissatisfied / Very dissatisfied”
Scale Length
Five-point and seven-point scales are the most widely used for attitudinal questions. Five-point scales are easier for respondents to use. Seven-point scales give you more variance in the data, which is useful when you need to detect small differences between groups.
Ten-point and one-hundred-point scales introduce arbitrary precision. A respondent marking 7 versus 8 on a ten-point scale is not making a meaningfully different judgment. For a detailed look at scale design for surveys, see What is a Likert Scale?.
The Neutral Option
Always include a neutral midpoint for attitudinal questions. Removing the neutral option forces respondents with genuinely neutral views into either a positive or negative category, introducing systematic bias toward whichever end feels safer.
The exception is when you are deliberately forcing a directional choice, for example, in a preference task. In those cases, the absence of a neutral option is intentional, and you should make that explicit to respondents.
Question Order Effects
The order in which questions appear affects how respondents answer later questions. Early questions prime respondents to think in certain ways, which shapes their answers to subsequent questions.
The most common order effect is asking an overall satisfaction question after a series of detailed questions about specific service elements. Respondents who have just spent five questions thinking about problems with your service will rate overall satisfaction lower than they would have before the detailed questions.
Best practice: ask the overall question first, then the detailed questions. This is called the funnel approach: broad to narrow.
Acquiescence Bias
Acquiescence bias is the tendency for respondents to agree with statements regardless of their actual views. Some respondents will say “yes” or “agree” to almost any statement simply because agreement feels more socially comfortable than disagreement.
You can reduce acquiescence bias by using balanced forced-choice questions rather than agree/disagree scales. Instead of “Do you agree with that [statement]?” use a comparison: “Which of the following best describes your view: A or B?”
Another approach is to reverse-code some items: include statements where agreeing actually represents a negative view. If a respondent is genuinely agreeing with everything, their answers to reverse-coded items will contradict their answers to standard items, flagging them as unreliable data.
Social Desirability Bias
Social desirability bias occurs when respondents answer in ways they believe are socially acceptable rather than honest. This is particularly common for questions about sensitive topics: income, health behaviour, financial decisions, or views on politically loaded issues.
For sensitive questions, anonymous survey formats produce significantly more honest answers than researcher-administered interviews. Where the respondent knows they are talking to an AI rather than a human (conversational AI interviews) can also reduce social desirability effects, because the judgment of a human interviewer is removed.
Testing Your Survey Before Launch
No matter how carefully you write your survey, test it before publishing it. There are two levels of testing: cognitive testing and pilot testing.
Cognitive Testing
Cognitive testing involves walking a small number of respondents (typically five to ten) through the survey and asking them to think aloud as they answer. The question is not whether they give the “right” answer but whether they understand the question the way you intended it.
You will almost always discover at least one question that respondents interpret differently from your intention. Cognitive testing reveals these problems before they contaminate your dataset.
Pilot Testing
A pilot test runs the survey with a small sample (50-100 respondents) drawn from your target population. Analyse the results as you would a real wave. Look for: questions with very low variance (everyone answered the same way, which suggests the question is not discriminating), high “not applicable” rates (the question does not apply to your respondents), and unexpected answer distributions.
For guidance on sample size decisions before launching a survey, see Sample Size for Qualitative Research: How Many Respondents Do You Need?.
A Practical Pre-Launch Checklist
Before your survey goes live, check every question against this list:
- Does the question contain evaluative language that could signal a preferred answer?
- Does it ask about more than one thing at once?
- Does it assume something the respondent might not agree with?
- Is the response scale balanced and appropriate for what is being measured?
- Is this question placed in the right order relative to the questions around it?
- Have you cognitively tested it with real respondents?
Rigorous question design is the foundation of any survey study. Sample size, analysis method, and presentation can all be improved after the fact. The questions cannot be changed once the data is collected.