Innovation Advisory · Framework 09
6 questionsThe Survey Bias Audit
Six questions to stress-test a survey instrument for the biases that survive the pilot. For evaluation designers, MEL teams, and reviewers about to sign off on a household questionnaire.
Who is not represented in the sample frame — and why?
Every frame excludes somebody. Missing from most household surveys: unhoused people, migrant labourers between locations, women whose husbands answer the door, houses without a listed head. Name the excluded groups on the first page. If you cannot name them, the frame is doing that work invisibly.
Does the question form the answer it invites?
‘Do you agree that women should have equal opportunities?’ will get a different answer than ‘Do you think a wife should give up her job if her husband asks her to?’ Both are about gender norms. One flatters the respondent's self-image. The other does not.
Do the response categories cover the reality you are measuring?
‘Very satisfied / Satisfied / Neutral / Dissatisfied / Very dissatisfied’ assumes the respondent has a strong preference. ‘Own / Rent’ forgets the majority of Indian housing arrangements. Missing categories are a hidden proxy for whose reality the survey was designed for.
Would a different interviewer get a different answer?
Enumerator caste, gender, age, dress, accent, and even mood all shape which answers respondents give. Randomise the enumerator-respondent pairing. Track answer variance by enumerator. Report the between-enumerator variance in the final tables. If nobody is asking, the answer is: yes, a different interviewer would get a different answer.
What would the respondent be embarrassed to say out loud, to you, in this house?
A woman asked about intimate partner violence in front of her mother-in-law will report the community average. A farmer asked about pesticide use in front of a health worker will underreport. The audit is not "did we ask the question" but "did we ask it in a way the truthful answer could survive."
What language is the survey in, and whose language is that?
Translation is not the same as adaptation. ‘Empowerment’ does not translate cleanly into most Indian languages. ‘Household head’ assumes a household structure. If the translation was done by an urban graduate student and the survey runs in a Santhal village, the survey is in Hindi, not in Santali — even if the words are Santhali.
What the six questions are doing
Read together, the six locate the six places bias hides in a survey instrument that has already survived a review. The frame catches sampling exclusions. The item wording catches leading questions. The answer options catch structural exclusions in the response categories. The enumerator catches interviewer effects. The social desirability catches self-presentation bias in the room the survey happens in. The language of the room catches the translation-versus-adaptation gap that turns a survey into a foreign-language survey without anybody noticing.
Use it after the piloted instrument has been shortlisted and before the enumerator training. Pair with the Indicator Test (VALID) for each question's indicator, and with the Causal Pathway Review (CLEAR) for the theory the survey feeds into.