Innovation Advisory · Framework 09
6 questionsThe Survey Bias Audit
Six questions to test a survey instrument for the biases that survive the pilot. This canvas is written for evaluation designers, MEL teams and reviewers about to sign off on a household questionnaire.
Who is missing from the sample frame, and why?
Every frame excludes somebody. Most household surveys miss unhoused people, migrant labourers between locations, women whose husbands answer the door, and houses without a listed head. Name the excluded groups on the first page. If you cannot name them, the frame is excluding them without anyone seeing it.
Does the wording of the question shape the answer it invites?
‘Do you agree that women should have equal opportunities?’ will get a different answer from ‘Do you think a wife should give up her job if her husband asks her to?’ Both are about gender norms. The first flatters the respondent's self-image and the second does not.
Do the response categories cover the reality you are measuring?
‘Very satisfied / Satisfied / Neutral / Dissatisfied / Very dissatisfied’ assumes the respondent has a strong preference. ‘Own / Rent’ leaves out the majority of Indian housing arrangements. Missing categories show whose reality the survey was designed for.
Would a different interviewer get a different answer?
An enumerator's caste, gender, age, dress, accent and even mood all shape the answers respondents give. Randomise the pairing of enumerators and respondents, track the variance in answers by enumerator, and report the between-enumerator variance in the final tables. If nobody is checking, assume that a different interviewer would get a different answer.
What would the respondent be embarrassed to say out loud, to you, in this house?
A woman asked about intimate partner violence in front of her mother-in-law will report the community average. A farmer asked about pesticide use in front of a health worker will underreport. The audit asks whether the question was put in a way that let the truthful answer survive, which is harder than asking it at all.
What language is the survey in, and whose language is that?
Translation and adaptation are different tasks. ‘Empowerment’ does not translate cleanly into most Indian languages, and ‘household head’ assumes a particular household structure. If an urban graduate student did the translation and the survey runs in a Santhal village, the survey is effectively in Hindi, even if the words are Santhali.
What the six questions are doing
Together, the six questions locate the six places where bias hides in a survey instrument that has already passed a review. The frame question catches exclusions from the sample. The item wording question catches leading questions. The answer options question catches exclusions built into the response categories. The enumerator question catches interviewer effects. The social desirability question catches self-presentation bias in the room where the survey takes place. The language of the room question catches the gap between translation and adaptation, which can turn a survey into a foreign-language survey without anyone noticing.
Use it after the piloted instrument has been shortlisted and before enumerator training. Pair it with the Indicator Test (VALID) for the indicator behind each question, and with the Causal Pathway Review (CLEAR) for the theory the survey feeds into.