A state government has commissioned a randomised controlled trial of a critical thinking curriculum in its government schools. The RCT is expensive, correctly designed, and ready to go. There is exactly one question left to answer before the first classroom is randomised.
What are we going to measure?
This sounds like it should have been settled at the proposal stage. It usually is, and it usually is not. Most curriculum evaluations arrive at the measurement question late, with the instrument chosen from a shortlist and the construct treated as obvious. Both of those choices carry more of the eventual finding than the intervention does.
The construct is not the instrument
Critical thinking is a construct. It is not one skill; it is a family — evaluating a claim against evidence, holding uncertainty without collapsing it into an answer, distinguishing a question from a demand, noticing when an argument’s structure disagrees with its confidence. The construct is real. It is also fuzzy at the edges.
The instrument is a specific test. A short-answer probe with rubric scoring. A multiple-choice test built on Watson-Glaser or one of its descendants. A performance task where students argue a position and get scored on structure. Each of these captures part of the construct and leaves other parts on the floor.
The trial cannot make itself honest if the two are treated as the same thing. The evaluation may show the intervention moves the instrument. What that tells you about the construct depends on how tightly they were welded to each other, and that welding is a decision you make now — before the pilot, before the randomisation, before the enumerator training.
What the RCT can and cannot conclude
Assume the intervention is delivered well and the trial is powered. There are four kinds of things the study can conclude, and it is worth being precise about which is which before the first classroom is randomised.
One. The intervention moves the instrument. This is the easiest and most common finding, and it is the one most reports treat as the whole answer.
Two. The intervention moves the construct, if the instrument captures the construct well. This is what people usually want to say. It is only defensible if the instrument was validated — separately and honestly — against the construct in this population, in this language, in this grade band.
Three. The intervention moves the construct in ways the instrument was not built to detect. This is the finding evaluators are most likely to miss, because the trial is not designed to see it. Building a small qualitative panel alongside the RCT is how you make this finding possible.
Four. The intervention teaches to the test. If the instrument leaked into the curriculum — even by structural resemblance rather than by copied items — the study measures alignment, not learning. This is a common failure mode with critical-thinking programmes because the curriculum designers are usually the same people who know the instrument.
The instrument review needs to distinguish these four before the trial begins, and the report needs to be honest about which one it can defend at the end.
What the advisory work looks like
The design decisions that pay off before the intervention runs are unglamorous. They usually look like this.
The construct definition, written in three sentences, in the language of the state curriculum’s own frame. Not aspirational, not literary — matching the frame the trial will be judged against.
A short pilot of the instrument in classrooms not part of the trial, with two things measured: internal consistency (is the instrument reading its own construct in this population), and floor/ceiling effects (does the test have room to move where the intervention is aimed).
A companion qualitative protocol — six to eight classroom observations and student interviews, on a randomised sub-sample — that can catch the third-kind finding above. This is cheap relative to the RCT and is usually the first thing cut for budget reasons. It is also usually the thing whose absence hollows out the final report.
Pre-registration of the primary analysis. Not because the state will read the registration, but because the act of writing it forces the team to name the finding it hopes to make. Once that finding is named, the honest question of “what would change our mind” becomes tractable.
Why this matters more than usual
Critical thinking sits at the centre of a set of curriculum debates that will still be running long after this trial ends. If the study concludes the intervention worked and the construct was fuzzy, the finding will get generalised into policy for a decade. If it concludes the intervention did not work and the instrument was miscalibrated, an actually useful curriculum will be abandoned.
The construct fix is not a step to complete before the trial. It is the trial’s most consequential decision, dressed up as preparatory work.
Run The Indicator Test (VALID) on your instrument before you commission the trial. Run The Causal Pathway Review (CLEAR) on the pathway from curriculum to construct to instrument to policy. If either canvas fails, the trial is not ready — even if the randomisation code is.