Innovation Advisory · Framework 15
5 questionsThe Curriculum-Construct Fit
Five questions to ask before you evaluate a curriculum. This canvas is written for research advisors, evaluation designers and state teams about to commission an RCT of a teaching programme.
Can you write the construct in three sentences, in the frame the state will judge it against?
‘Critical thinking’ is a family of skills, and so are ‘life skills’, ‘resilience’, ‘grit’ and ‘agency’. Write the construct in plain language, in three sentences, in the terms of the state's or funder's curriculum frame. If the sentences drift into aspiration, the instrument will drift too, and the trial will end up measuring something the report cannot defend.
Does the instrument capture the construct, or only the nearest convenient proxy?
A short-answer probe scored with a rubric, a multiple-choice Watson-Glaser variant and a performance task with an argument-structure rubric each capture part of the construct and leave other parts out. Pick the one whose omissions you can defend, and report the omissions in the body of the design document.
Was the instrument built by the same people who built the curriculum?
If the same team designs the curriculum and the instrument, the trial will measure how closely the two are aligned and will miss whether students learned. Even structural resemblance (question format, phrasing, item order) leaks. Separate the two teams deliberately, and record the separation before the trial begins.
Does the instrument have room to move in the direction the intervention is aimed?
If most students already pass the test, a good intervention has no room at the top to show an effect. If almost no student can pass it, a poor intervention has no floor against which to show anything at all. Pilot the instrument in classrooms outside the trial, check the distribution of scores, and adjust before randomisation.
Is there a small qualitative panel running alongside the trial to catch what the instrument was not built to detect?
The curriculum effects that matter most are usually the ones the instrument was never designed to catch, such as changes in how students talk about disagreement, how teachers scaffold arguments, and what silence in the classroom means. Six to eight observations plus interviews with a random sub-sample of students cost little next to the RCT, and the panel is still the first thing cut when the budget tightens. Its absence is also usually what leaves the final report thin.
What the report can and cannot conclude
Assume the intervention was well delivered and the trial had enough power. The study can then reach one of four conclusions. The intervention moved the instrument. The intervention moved the construct, if the instrument captures it well. The intervention moved the construct in ways the instrument was not built to detect, which shows up only if there is a qualitative panel. Or the intervention taught to the test, which shows up only if leakage was checked.
The review of the instrument needs to separate these four before the trial begins, and the report needs to say plainly which of them it can defend at the end. This canvas is a pre-mortem for that report.