Innovation Advisory · Framework 15
5 questionsThe Curriculum–Construct Fit
Five questions before you evaluate a curriculum. For research advisors, evaluation designers, and state teams about to commission an RCT of a teaching programme.
Can you write the construct in three sentences, in the frame the state will judge it against?
'Critical thinking' is a family, not a skill. So is 'life skills', 'resilience', 'grit', 'agency'. Write the construct in plain language, in three sentences, matching the state or funder's curriculum frame. If the sentences drift into aspiration, the instrument will drift too — and the trial will end up measuring something the report cannot defend.
Does the instrument capture the construct, or the nearest convenient proxy?
Short-answer probe with rubric scoring; multiple-choice Watson-Glaser variant; performance task with argument-structure rubric — each captures part of the construct and leaves other parts on the floor. Pick the one whose omissions you can defend. Report the omissions in the design document, not the appendix.
Was the instrument built by the same people who built the curriculum?
If the curriculum designer and the instrument designer are the same team, the trial will measure alignment rather than learning. Even structural resemblance — question format, phrasing style, item order — leaks. Separate the two teams by intent, not just by convenience. Document the separation before the trial begins.
Does the instrument have room to move in the direction the intervention is aimed?
A test that most students already pass leaves no ceiling for a good intervention to show effect. A test that almost no student can pass leaves no floor for a poor intervention to demonstrate anything at all. Pilot the instrument in classrooms outside the trial; check the score distribution; adjust before randomisation.
Is there a small qualitative panel running alongside the trial to catch what the instrument was not built to detect?
The most consequential curriculum effects are usually the ones the instrument was never designed to catch — changes in how students talk about disagreement, how teachers scaffold arguments, what the classroom's silence means. Six to eight observations plus student interviews on a random sub-sample is cheap relative to the RCT, and is the first thing cut for budget reasons. It is also usually the thing whose absence hollows out the final report.
What the report can and cannot conclude
Assume a well-delivered intervention and a powered trial. There are four things the study can conclude: the intervention moved the instrument; the intervention moved the construct (if the instrument captures it well); the intervention moved the construct in ways the instrument was not built to detect (only visible if there is a qualitative panel); or the intervention taught to the test (only visible if leakage was checked).
The instrument review needs to distinguish these four before the trial begins, and the report needs to be honest about which of them it can defend at the end. This canvas is the pre-mortem for that report.