A state government has commissioned a randomised controlled trial of a critical thinking curriculum in its government schools. The RCT is expensive, correctly designed and ready to go. Before the first classroom is randomised, the team still has to decide what it is going to measure.
That decision should have been made at the proposal stage. On paper it usually is, and in practice it usually is not. Most curriculum evaluations reach the measurement question late. The instrument is chosen from a shortlist and the construct is treated as obvious, and those two choices shape the eventual finding more than the intervention does.
The construct and the instrument are different things
Critical thinking is a construct. It is a family of skills: evaluating a claim against evidence, holding uncertainty without collapsing it into an answer, telling a question apart from a demand, and noticing when an argument’s structure does not support its confidence. The construct is real, and its edges are fuzzy.
The instrument is a specific test. It might be a short-answer probe with rubric scoring, a multiple-choice test built on Watson-Glaser or one of its descendants, or a performance task in which students argue a position and are scored on its structure. Each of these captures part of the construct and misses other parts.
If the two are treated as the same thing, the trial cannot report accurately on either. The evaluation may show that the intervention moves the instrument. What that says about the construct depends on how closely the two are tied, and the team decides that now, before the pilot, the randomisation and the enumerator training.
What the RCT can and cannot conclude
Assume the intervention is delivered well and the trial has enough power. The study can reach four kinds of conclusion, and the team should be precise about which is which before the first classroom is randomised.
The first is that the intervention moves the instrument. This is the easiest and most common finding, and most reports treat it as the whole answer.
The second is that the intervention moves the construct, provided the instrument captures the construct well. This is what people usually want to say. It can only be defended if the instrument was validated separately against the construct in this population, in this language and in this grade band.
The third is that the intervention moves the construct in ways the instrument was not built to detect. Evaluators are most likely to miss this finding, because the trial is not designed to see it. A small qualitative panel run alongside the RCT makes it possible to see.
The fourth is that the intervention teaches to the test. If the instrument leaked into the curriculum, even through structural resemblance with no copied items, the study measures alignment with the test and says little about learning. Critical thinking programmes often fail this way, because the people who design the curriculum usually know the instrument well.
The instrument review needs to separate these four before the trial begins, and the final report needs to say plainly which of them it can defend.
What the advisory work looks like
The design decisions that pay off before the intervention runs are unglamorous. They usually include the following.
A definition of the construct in three sentences, written in the terms of the state curriculum’s own framework, since that is the framework the trial will be judged against. It should be plain and literal.
A short pilot of the instrument in classrooms outside the trial, measuring two things. One is internal consistency, which shows whether the instrument measures a single construct in this population. The other is floor and ceiling effects, which show whether the test has room to move where the intervention is aimed.
A companion qualitative protocol of six to eight classroom observations and student interviews on a randomised sub-sample, which can catch the third kind of finding above. It is cheap compared with the RCT and is usually the first thing cut when the budget is tight. Its absence is also usually what leaves the final report thin.
Pre-registration of the primary analysis. The state may never read the registration, but writing it forces the team to name the finding it hopes to make. Once that finding is named, the team can ask in concrete terms what result would change its mind.
Why this matters more than usual
Critical thinking is at the centre of several curriculum debates that will continue long after this trial ends. If the study concludes that the intervention worked while the construct was fuzzy, the finding will be generalised into policy for a decade. If it concludes that the intervention did not work because the instrument was miscalibrated, a useful curriculum will be abandoned.
Fixing the construct is the most consequential decision in the trial, even though it looks like preparatory work.
Run The Indicator Test (VALID) on your instrument before you commission the trial. Run The Causal Pathway Review (CLEAR) on the pathway from curriculum to construct to instrument to policy. If either canvas fails, the trial is not ready, even if the randomisation code is.