Children's minister Tim Loughton delivered a stark message to children's and youth organisations at the Funding the Future conference in London last month. He told delegates that government support will focus more on "quality of outcomes," adding, "it is, in other words, about producing more evidence-based projects".
His warning came less than two months after Labour MP Graham Allen's report Early Intervention: The Next Steps urged the government to end the culture where "billions of pounds are paid out year after year; indeed decade after decade, often without the faintest acquaintance with an evidence base".
John Copps, head of sector research at New Philanthropy Capital (NPC), believes such comments are an overdue wake-up call for organisations reliant on public funds. His organisation has long advocated that charities, in particular, need to improve the way they measure their effectiveness and communicate the value they bring to society. "There is a limited budget and it is a stark fact that charities are going to have to get better at proving they are capable and will be a success," says Copps.
But the cost of independent evaluations can run into thousands or even tens of thousands of pounds per programme. So where does this leave all those organisations that lack both a strong evidence base and the resources to pay for research?
Louise Morpeth, co-director at the Social Research Unit, which gathered evidence for Allen's report, says that one way organisations can cut evaluation costs is to use an off-the-shelf programme that comes ready with robust research. She cites the 19 top-tier projects identified in Allen's first report which include Incredible Years in children's centres and Multidimensional Treatment Foster Care.
"Charities need to be realistic," she says. "If they do not have the evaluation, they can pick a project that does."
A small charity that has evidence of strong links with its community can be attractive to commissioners and grant givers, she adds. Compiling such evidence can often be as simple as collecting testimonial quotes from local community groups.
Use existing evidence
A cost-effective way to impress commissioners can be to tap into the wealth of evidence and evaluation that already exists, argues Richard Piper, head of strategy at the National Council for Voluntary Organisations (NCVO). For example, it would be perfectly valid for a detached youth work project to refer to Joseph Rowntree Foundation's 2004 study The Role of Street-based Youth Work to support an application for funding.
"Arguably, we need more of this intelligent use of a limited number of high-quality studies, rather than a thousand poor quality evaluations that no-one ever reads or shares or uses or learns from," says Piper. "If a report from 2004 still represents the best thinking about a particular intervention or field, then use it."
Morpeth adds that organisations need to show commissioners and grant givers they have a clear focus, saying that it is surprising "how many are unclear about what they do".
Copps says a fundamental question charities should ask themselves is how they, as an organisation, can improve. "Those giving grants are impressed by honesty, an acknowledgement of mistakes and evidence that action is being taken to improve," he says.
Education select committee chair Graham Stuart MP voiced his frustration at the inability of youth sector organisations to articulate the difference they make. During the first sitting of the inquiry into services for young people in January, he described the inability of the youth sector to present robust evidence for their work as an "extraordinary failure".
But politicians, commissioners and grant givers also need to improve their understanding of evaluation, says Ian Joseph, social researcher at the University of East London. He is particularly concerned about commissioners and grant givers demanding statistics rather than softer, qualitative evidence to show that programmes make a difference.
Joseph also questions Allen's focus on programmes proving their future savings to taxpayers through measurements such as cost-benefit analysis. "This kind of social cost analysis can be useful, but it is just one indicator of success. Also, it is complex work and it is still difficult to prove long-term gains effectively," he says.
To fill the void in evidence, a number of evaluation tools have been developed to help organisations monitor their performance (see below). NCVO's Piper believes such tools offered by his charity and others like NPC offer products that are "high quality, simple and good value".
But Joseph questions the value of such toolkits that offer templates for surveys, describing them as "useful, but only up to a point".
He says: "The focus on evidence is to be welcomed and there is a clear need for support. With a before-and-after survey, for example, setting the right questions does require certain skills. But more is needed than just a survey. There needs to be more detail, perhaps running focus groups as well."
According to the Social Research Unit's Morpeth, commissioners and grant givers put too much focus on improving outcomes for young people as a measure of success. "Take a child's play area," she says. "It is impossible to prove it is cutting obesity, for example. To ask for such evidence would be unfair."
Customer surveys
More realistic evidence of the success of a play area would focus on whether it is safe or well used through a customer satisfaction survey of parents and children, she argues.
Piper says commissioners and grant givers should be clear about the evidence they expect to see and that providers need to negotiate if they think evaluation being asked for is unreasonable. "There is too much waste, with stuff collected that no one actually wants, or needs, or uses," he argues. "Some might say this is a scandal."
While he concedes that some grant givers and commissioners can be too obsessed with statistical evidence, he says many continue to respond well to qualitative data and case studies and accept these are often a more appropriate way for certain charities to prove their success. For example, quotes from children and young people as well as photos still have their place, Piper argues.
Somerset Youth Volunteering Network is only too well aware of the differing evidence required by funders. It receives money from around 20 different funders a year to maintain its £400,000 income, which helps pay for a 12-strong team that supports 3,500 young people a year across the South West. Recent grant successes include £97,000 from the government's Transition Fund, which chief executive Scott McMillan admits required the organisation to provide statistical data about the sex, ethnicity and religion of those taking part in its programmes.
McMillan advises organisations to build evaluation into their day-to-day work. "What we have learned over the past 12 years of dealing with funders is to sit down at the start of the project, find out exactly what they want evidence of and look to compile that as we go along. What you do not want to end up doing is rushing around at the end of the project," he says.
Young people's views
He believes that robust evaluation does not need to be expensive and estimates that the average cost to evaluate a project can be as little as three per cent of expenditure. New Philanthropy Capital advises the amount that charities should set aside depends on the size of the organisation and the work they do.
McMillan adds: "What we have learned is that evaluation is not all about producing a glossy brochure costing thousands of pounds or getting universities involved. It is often more about getting a clear picture from funders about what they want from you and making sure you think about evaluation in everything you do."
CYP Now is holding its annual Impact and Outcomes conference in London on 19 May. The conference will include sessions on measurement tools as well as assessing long-term outcomes. To find out more, visit www.measuringoutcomes.co.uk
TOOLS FOR GATHERING EVIDENCE
Wellbeing measure
This tool, developed by New Philanthropy Capital, focuses on a before-and-after survey suitable for 11- to 16-year-old users and based around seven outcomes. These are: self-esteem, resilience, emotional health, quality of relationships with friends and with family as well as satisfaction with school and with their community.
Prices start at £375 for a simple before-and-after survey but can rise to around £3,000 depending on the number of surveys used and the number of young people surveyed.
www.philanthropycapital.org/ how_we_help/big_ideas/wellbeing_project.aspx
Soul Record
Targeted at small charities, councils, schools and children's centres, this toolkit is used mostly in one-to-one sessions, but can also be used with groups. It seeks to measure improvement in people's lives after taking part in programmes where there is a lot of face-to-face contact over a period of time, such as a family support project. Questions are designed for either adults, young people or children.
Project Oracle
Launched in 2008 by the London mayor's office, the project aims to support the youth sector to tackle youth violence. This summer it will launch an evaluation toolkit in conjunction with the Social Research Unit that aims to create standard measurements of effectiveness that groups and charities can compare themselves against.
This self-assessment-focused approach will also offer guidance on how to make improvements.
www.london.gov.uk/priorities/crime-community-safety/ time-action/project-oracle
VIP Impact Tools
The National Council for Voluntary Organisations has developed a package of evaluation tools to help umbrella bodies that support or represent other charities.
This can assess their impact on an organisation with which they are working as well as specific sectors, such as youth work or a geographical area. A three-year package of support, including training and access to a system to use the tools online to track progress, is offered by the NCVO.
