Measurement work usually goes wrong at one of two moments. The first comes when the boxes and arrows of a Theory of Change are already on the wall and nobody wants to ask whether the logic holds. The second comes when somebody proposes an indicator and the room nods along, because nobody wants to say that it does not measure what it claims to measure.
This note describes two reviewer’s checklists for those two moments. Both are on the Canvas now, free and printable, one page each.
CLEAR, for the pathway
CLEAR is used when a Theory of Change or causal pathway is under review. It asks five questions, and each letter does a separate job.
- Causal logic. Does the pathway show how one step leads to the next, and where does the logic jump? Read the if-then between every pair of steps. If a reasonable reader could say “and then a miracle happens here”, you have found the jump. Either fill it with an intermediate step or admit that the leap is an assumption and move it to the assumptions row.
- Level clarity. Are activities, outputs, intermediate outcomes and outcomes each placed at the right level? The most common error is presenting an output as an outcome. Trainings delivered is an output, and changed practice is an outcome. If the implementer alone could tick a row off, it is probably the work that produces an outcome rather than an outcome itself.
- Essential missing links. Were the key stakeholders and systems considered when the pathway was designed? A pathway that depends on a frontline worker, a panchayat, a school principal or a district officer should name them somewhere. If the people and structures the pathway runs through do not appear on the diagram, the pathway is relying on their effort without acknowledging it.
- Assumptions and risks. Which assumptions need to be stated, and where could they fail? Every arrow hides an assumption. Take the strongest three out of the diagram and write each one as a full sentence. Then ask under what conditions it would stop being true.
- Repetition / overlap. Is anything repeated in another pathway, and should it be merged, moved or removed? Organisations that run several pathways in parallel often count the same outcome twice from different angles. A clean pathway claims only what it produces by itself.
The full one-pager is at /canvas/causal-pathway.
VALID, for the indicator
VALID is used on each indicator that goes into the logframe. It also asks five separate questions.
- Valid. Does the indicator measure the construct named in the ToC, or only a convenient proxy for it? Write the construct in plain language (“do mothers feel supported during the perinatal period”) and put the indicator beside it (“count of calls received”). If a stranger reading the two would not believe that the indicator captures the construct, you have a proxy problem. The proxy may still be useful, as long as you call it a proxy.
- Actionable. Does a result on this indicator tell the implementer what to do next? If the answer is “we report it” and never “we do X”, the indicator serves the dashboard and does nothing for the work.
- Linked. Does the indicator belong to a specific node in the causal pathway, or to the project as a whole? An indicator that floats above the whole programme, such as “lives improved” or “outcomes achieved”, measures a general impression. Anchor it to one step in the chain.
- Independent of gaming. How hard is it to move the indicator without producing the underlying change? This is Goodhart’s law applied to daily operations. If a frontline worker under pressure can shift the number without doing the work, the indicator will be gamed long before anyone evaluates the programme.
- Disaggregable. Can the indicator be broken down by the equity dimensions that matter for this programme? An average improvement can hide inequities. Decide at the start which cuts the indicator must support (caste, class, gender, geography, age, disability) and check that the data system can produce them.
The full one-pager is at /canvas/indicator-test.
How they fit together
CLEAR applies to the boxes and arrows. VALID applies to the metric inside any one box. Run CLEAR on the pathway first, because it is pointless to ask whether an indicator is valid for a construct that is misplaced or unsupported. Once the pathway holds, run VALID on every indicator proposed for any of its boxes.
Together they catch the two failures that The Measurement Trap tries to make visible: a pathway that does not say what it depends on, and an indicator that does not measure what it claims to. Most of the problems found at evaluation were already built in at one or both of these moments.
The frameworks are free to use, and the page sources are linked above. If you apply them to a real pathway and find a question missing, or one that is doing the wrong job, please write in.