Measure Once, Cut Twice: Using Data for Continuous Improvement and Impact Evaluation in Education Programs
By Sarah Williams, research and learning officer, The Centre for Youth Impact
Tuesday, November 26, 2019
This research paper describes a “quality to outcomes” (Q-O) evaluation design that was used to evaluate out-of-school time provision for young people in the US.
- C Smith, S Peck, L Roy, L Smith (April 2019)
The Q-O design offers an alternative to “gold standard” counterfactual studies, such as randomised control trials, by integrated impact evaluation and continuous improvement within the same youth organisation. Through combining measures of quality and outcomes, the approach can offer detailed information about practice that is needed for continuous quality improvement alongside insight to help understand impact.
Aims and methodology
The purpose of Q-O design is twofold: first, to identify and differentiate subgroups of provision for young people according to the quality of practice that is provided; then, to compare the social and emotional skill growth of individual young people across those subgroups. In other words, the evaluation design assesses “skill growth by level of quality” across a cohort of youth organisations.
In this paper, “quality” is based on four domains: basic safety, basic learning conditions, social interaction and advanced learning conditions. Measures of practice quality are at the core of the Q-O design, including the Youth Program Quality Intervention (YPQI). In the Q-O design, the prevalence of low- and high-quality across a cohort of providers is identified: organisations are classified into distinct “profiles” of quality based on their practice in the key quality domains.
The “skill growth” of individual young people, on the other hand, is based on changes in social emotional learning (SEL) over time. SEL skills are broadly understood to refer to a young person’s ability to self-regulate and demonstrate agency. In the Q-O design, skill sets of individual young people are measured at two or more time points, typically through measures that use behavioural indicators of underlying mental skills. The distribution of low and high SEL skill exhibited by young people is identified at a baseline, and change is measured from baseline to a second time point. “Low” SEL skill indicates higher support needs among young people. “High” SEL skills means offering support to young people to make sure they maintain and practise their skills.
The Q-O design uses “path impact analysis” to “connect the links in a hypothesised cascade of effects”: it predicts that exposure to high-quality practice causes greater SEL skills growth in young people. This skills growth transfers to other areas in young people’s lives, where improvements in academic performance and other outcomes are experienced over subsequent years. The Q-O design builds on an existing body of research that suggests SEL skill-building transfers across settings to improve skill learning and outcomes in a variety of contexts for young people.
Key findings
This report uses example data to illustrate the evidence that is produced from Q-O designs. Some themes in this evidence are set out here, reflecting patterns that are frequently seen in other studies that use this evaluation design (e.g. Smith, Roy, Peck, Macleod, & Helegda, 2017; Smith, Roy, Peck, & Macleod, 2018).
Distribution of quality profiles
Provision that is characterised by the highest quality represents examples of best practice fit to local circumstances and populations – where practitioners and organisations know young people, communities and context really well. Provision characterised by the lowest quality may not be producing positive effects on young people’s learning, and should be a clear target for both resources and improvement.
Shape of quality profiles
The shape of the quality profiles is as important to consider as the overall quality score. For instance, experience suggests “middle quality” provision tends to fit a “direct instruction” profile, with high support for learning but minimal support for interaction and reflection. Identifying such patterns, which may be an intentional instructional style chosen by staff, provides a useful basis for continuous quality improvement at the individual and collective organisational level.
Equity effects
The Q-O method provides an approach for “evaluating the equity of how outcome effects are distributed across young people”. Specifically, it can reveal the extent to which young people with lower SEL skills, who are offered high-quality practice, demonstrate greater skill growth than young people in higher skill subgroups. This is where a key purpose of youth provision policy is achieved, where vulnerable young people are supported to further engage with the world.
Implications for practice
- The Q-O design is distinct from “impact studies” that produce evidence of outcomes through counterfactual reasoning (that is, the difference between the average “change” for young people engaging with a particular project or organisation, and the average “change” for those young people not engaging).
- Whilst these studies are often desired by policymakers for accountability and funding purposes, they are not always well-suited to understanding complexity and “seldom offer useful information to practitioners”.
- The Q-O design offers practitioners detailed information about practice and process that is required for continuous quality improvement, resulting in tangible improvements in the quality of provision to levels at which outcomes are known to occur.
- Various studies using the Q-O design have been published by the David P. Weikart Centre, all undertaken in the context of US out-of-school programmes. As yet there is no evidence base to support the use of the evaluation design in the UK youth sector, though emerging research is being undertaken through the Youth Programme Quality Intervention (YPQI) UK Pilot led by the Centre for Youth Impact.
