One of the most comprehensive research projects ever undertaken to measure the impact of open-access youth work has found that young people who took part saw greater improvements in social and emotional learning, with the highest quality provision delivering the most benefits.
The evaluation of the three-year, £40m Youth Investment Fund (YIF) programme was carried out by New Philanthropy Capital (NPC) and the Centre for Youth Impact. Its final report, published last month, sets out the characteristics of effective open-access youth provision and puts forward a framework for charities and funders to use to generate their own evidence of impact.
High satisfaction marks
The YIF programme saw funding from the Department for Digital, Culture, Media and Sport (DCMS) and the National Lottery Community Fund (NLCF) allocated to dozens of youth organisations across Liverpool, Tees Valley and Sunderland, east London, Bristol and Somerset, West Midlands, and Eastern Counties (see in numbers). Earlier analysis of the programme showed that the projects funded through it were well received by young people using them, and achieved high satisfaction marks for safety, enjoyment and trust.
In addition to helping youth work providers gather evidence of impact, experts say findings from the YIF Learning Project should strengthen the argument for greater backing from policymakers and funders, and help the sector focus more on delivering quality provision (see expert view). Here, experts analyse the key findings from the evaluation.
IN NUMBERS
- £40m funding for the programme
- 400,000 young people attended sessions run by YIF providers
- £3-13 possible return on every £1 invested in open-access youth provision
Source: YIF Learning Project, May 2021NPC and Centre for Youth Impact
Evaluation practice
The YIF evaluation approach gathered five types of quantitative data – beneficiary, engagement, quality, feedback, and outcomes. It explored the links between them revealing the relationships between quality of provision, the experiences of young people and impact.
The evaluation framework used for the project enabled triangulation of data across multiple settings and provided for in-depth testing of hypotheses. The research team say adopting this approach is feasible for youth organisations of all sizes, both individually and collectively. However, collecting outcomes data for young people over long periods of time remains a challenge.
Another obstacle to evaluation included variation across YIF grant holders in terms of evaluation capacity, resources and motivation to take part in the evaluation. Where resources were stretched, the demands of frontline delivery took priority.
Kings College London academic Tania de St Croix, who is also researching youth work impact, says the project’s focus on taking into account the voices of young people and youth workers in its evaluation methods is “something to celebrate”. She also praises the project for making it optional for grantees to undertake questionnaires of participants, something that many young people object to doing. “Most funders wouldn’t be that flexible,” she adds.
The evaluation estimates that over a 25-year period the Treasury could save between £3-£13 for every £1 invested in the YIF projects, via increased taxes or reduced cost of services. However, de St Croix says putting a monetary value on the impact of youth work is still relatively new and sends a message “that this is desirable, which is worrying”.
“If we’re going to do it we need to include caveats: that there’s also a value to youth work that cannot be measured and monetised, and that this is equally important,” she adds.
Lessons for youth providers
The evaluation found that the quality of provision and young people’s experiences play a critical role in developing social and emotional learning skills – capturing data on these aspects of provision gives a much fuller picture of how and why change might be happening for young people.
It recommends that youth organisations continue to listen to young people and embed systematic collection of feedback into practice.
Pauline Daniyan, interim director of programmes at London Youth, says inclusive practices are “a core principle of the sector”.
“It is a timely reminder for a broad offer that is inclusive of young people from all genders and ethnic backgrounds,” she says of the evaluation findings.
“Across London Youth’s network, we know there is a slight bias towards organisations working with young men and boys. Our programmatic work, particularly in sports and mental health, has clearly demonstrated the effectiveness of actively broadening our offer to be inclusive to young women and girls.
“We fully endorse the recommendation to embed listening and responding to young people into professional practice. It is vital that engagement with young people is genuine dialogue with two-way communication.”
Learning for funders
The YIF Learning Project recommends a focus on quality provision and youth voice in grant holder learning evaluations and to encourage a culture of continuous learning and commitment to evaluation among providers.
It also calls for it to be made easier for organisations to collect and share data. “This requires a culture of ‘low stakes accountability’ in which learning and improvement is prioritised over results,” it states.
The focus on quality of provision is important from a symbolic perspective, says Kevin Franks, chief executive of Youth Focus North East.
“Although this may seem obvious, it is pleasing to see this supported with evidence from an evaluation project of this scale and scope,” he adds.
However, Franks says that such a focus will need a shift in the way funders require providers to demonstrate their impact and value to them.
“The vast majority of young people, especially those who engage with open-access provision, also engage with a range of other providers and services,” he explains. “How many of these services are asking young people to complete evaluation forms, feedback data? Many are asking the same questions and using different tools to collect this information.
“It is clear that there needs to be an improved emphasis on more shared evaluation approaches…which bring youth organisations together with funders and young people so that measurement frameworks and related tools are in place before grants are awarded.”
Ultimately, he says, there needs to be a stop to short-term funding and a greater focus on how to deliver services in collaboration, “and funders need to think in the same way”.
Implications for policy
Leigh Middleton, chief executive of the National Youth Agency, says the YIF evaluation “makes a compelling case” for more investment in high-quality open-access youth work.
“There is a clear link between trained staff (high-quality practice), young people’s recognition of their enhanced learning (quality) and the impact outcomes achieved by young people (outcomes),” says Middleton.
“The challenge is how to get the government, and Treasury in particular, to listen and act on the evidence presented. Yet when evidence is delivered, it seems it is not enough and is all too often overlooked. That cycle has to end.”
To illustrate this, Middleton points to the two-year delay to the £500m pledged by the Treasury for investment in youth services. Initially earmarked for open-access provision, the Treasury now want at least half of the money to be spent on buildings and facilities and the rest on “innovation” only, says Middleton. “The Treasury has set a course for its promised fund that runs counter to the evidence,” he adds.
The cry from policymakers and funders for the sector to improve its use of impact evidence works both ways, says Middleton.
“As a sector we can – and must – use the evidence to share and improve youth work practice, and to help transform young lives through youth work,” he says. “However, to be effective, we need a shared commitment and significant investment from government and other funders, clearly based on the evidence presented.”
- YIF Learning Project resources from https://www.youthimpact.uk/yif-learning-project
EXPERT VIEW
LET’S EVALUATE WHAT’S AT THE HEART OF GOOD YOUTH WORK: TRUSTED RELATIONSHIPS, YOUTH VOICE AND SAFE SPACES
By Bethia McNeil, chief executive, the Centre for Youth Impact and Karen Scanlon, principal – funder impact, New Philanthropy Capital
One of the inherent challenges with evaluation is the risk that we treat it as an affirmation of what we think we already know, rather than an opportunity to pause and think more deeply. Instead, we should think of an evaluation not as the full stop at the end of a project, but a semicolon: an invitation to rethink what we think we know, and a call to action to improve what we do.
This is the opportunity presented by the YIF Learning Project: an emerging evidence base that identified a promising connection between the quality of youth provision and young people’s outcomes and gave us an approach to shared measurement co-produced with youth workers that has proven to be appropriate and meaningful for measuring the impact of open-access youth provision. This call to action involves all of us: practitioners, evaluators, funders, and young people.
We believe the next steps must be collective. They should include a re-framing of evaluation in youth provision to focus on a shared understanding of quality youth work; a recognition of the role of evaluation in supporting reflective practice; a commitment to common measures that highlight the consistent features of good youth work practice; and investment in long-term learning partnerships between funders, evaluators and delivery organisations. Committing to all this together would be nothing short of transformative.
The YIF Learning Project suggests that what we collectively believe to sit at the heart of good-quality youth work (trusted relationships, youth voice and safe spaces) are the most important “predictors” of impact. This is incredibly exciting. It could enable us to refocus evaluation on what matters, and simultaneously work in partnership with funders to advance understanding of and investment in quality youth work.
It has arguably never been more important to commit to collective action with and for young people. We’re sharing all the tools and resources from the YIF Learning Project openly and have already published updated versions based on our learning. We’re particularly focusing on what the evaluation told us about equity and young people’s voice, the “equity effect” of high-quality youth work, and the importance of listening routinely and systematically to young people’s feedback and then acting on it.
We urge youth organisations to consider ways to reach and engage young people who would benefit most from their support, focus on the quality of their provision, and provide different opportunities to listen and respond to the voices of young people.
