Evaluating early intervention

Whether diverting a child away from gang culture or helping a young person combat drug addiction, early intervention schemes have enormous potential to transform lives.

But evaluating such initiatives is no easy task, with commissioners and funders looking for clear evidence of success from support that is complex and often time consuming.

Evaluation can also be an expensive business that few providers can afford, especially in tough economic times.

Nevertheless, many of those involved in early intervention work are meeting such challenges head-on and successfully demonstrating the value of their work.

Effective evaluation is "good for the bean counters" when it comes to securing precious funding, according to Dez Holmes, director of consultancy Research in Practice.

"But more importantly, it is also good for families," she adds, as it helps ensure they get support from proven schemes.

Seeking the gold standard

The Early Intervention Foundation (EIF) sets out "standards of evidence", which provide a useful guide on the most effective forms of evaluation.

Near the summit is the use of randomised control trials (RCTs) that can prove those taking part had better outcomes against a comparison group. At the very peak are established schemes where multiple RCTs have taken place in different areas, including other countries.

Research in Practice's Dez Holmes says this kind of robust global evaluation is important as there is a danger of assuming good results from abroad can easily be replicated at home. The UK's distinct welfare state and early school starting age are among key differences to consider when using evidence from abroad.

While RCTs are the pinnacle of success, EIF director of implementation Donna Molloy is keen to point out that those unable to afford this form of complex evaluation "should not just pack up and go home". Many commissioners and funders are still impressed by evaluation that shows potential for success, especially through good feedback from service users, she stresses.

Charitable funders, such as trusts, are among those to be most impressed by evaluation that shows potential. "They are often just as interested in finding out if a good idea works as they are in gathering strong evidence," says Dan Corry, chief executive of think-tank New Philanthropy Capital (NPC).

Expensive RCTs can still be attainable for cash-strapped providers and commissioners by teaming up with universities and academic research bodies, says the EIF, which supports councils to forge such links.

These include the London Borough of Croydon, which was looking for help to evaluate its Functional Family Therapy (FFT) service. This offers intensive support to families where a teenager is at risk of offending.

The council was matched up with Queen's University in Belfast, which was keen to investigate whether the US-originated FFT could be a success in the UK. Through this partnership, a successful bid for £160,000 to the Economic and Social Research Council was made, which is now funding a three-year RCT.

Taking a logical approach

Logic modelling is another aspect of evaluation that scores highly in EIF's set of standards.

This focuses on a robust "theory of change" that shows to funders and commissioners what a particular problem is, how the scheme aims to address that and the outcomes that can be expected.

A recent example of an early intervention project where strong logic modelling impressed funders is the Better Start scheme to improve the lives of vulnerable families with babies and young children.

Last year, five areas secured a total of £215m from the Big Lottery Fund to implement Better Start. One was an NSPCC-led partnership in Blackpool, involving Blackpool Council and Blackpool Hospital Trust, which was awarded £44.9m over the next 10 years.

Merle Davies, director of the partnership's Centre for Early Child Development, explains its logic model's starting point was to highlight the high prevalence of "stress points" for local parents, including high rates of drug and alcohol problems, domestic abuse, isolation and mental ill health.

The partnership then gathered a range of academic evidence showing how such stresses harm outcomes for children. Finally, it proposed a range of interventions to tackle these issues and set out the expected results.

Some of these interventions were backed with strong evaluation involving RCTs from similar population groups in the UK, such as the Family Nurse Partnership scheme, which targets intensive support to young parents.

Other projects such as the Survivor Mums Companion - which has strong evidence of success in the US but not in the UK - is to be piloted from September 2016 in Blackpool, with input from its developers at the University of Michigan.

A good logic model also needs to be backed with cost data, says Holmes. "Having a good logic model with good evidence of success is no good to a commissioner if it ends up costing £400,000 to help one child and is unaffordable," she says.

The best logic models are also upfront about the fact that each early intervention scheme is part of a broader array of support and events in a local area. Holmes cites the example of an employability scheme she was involved in for parents in an area of high unemployment. "Overall, 25 jobs were created for parents and the project wanted to be able to say 'Look at the difference we've made'," she says. "But at the same time, and completely separate to the project, a new supermarket was opening, which was arguably a bigger driver for getting them jobs."

Tackling short-termism

Too often, funders and commissioners crave quick evaluation, showing good results within a short space of time. The reality is vulnerable young people and families with complex problems often require years of support.

Holmes says the challenge of securing funding for early intervention schemes "where the impact may not be seen until much further down the line" cannot be underestimated.

To counter short-termism, Holmes recommends building proxy indicators within evaluation to show funders an initiative has strong potential to change lives.

For example, for a scheme that aims to improve educational achievement of children through early years support, proxy indicators could include whether parents are more engaged in education or attendance levels when children start primary school. Proxy indicators can act as a stop-gap ahead of key evidence of success to be gleaned years later when children take their GCSEs or look to access further and higher education.

Holmes says such indicators can be part of a "journey of change" and can be "marked off as a project progresses".

Not all funders are fixated on the short-term. For example, the Big Lottery Fund has long-term grants available, especially for schemes where long-term support is required.

The Big Lottery has committed funding for up to eight years in the five Better Start areas to acknowledge more time is needed to effectively assess the life-long success of a scheme working with children from birth.

Jonathan Rallings, assistant director for policy and strategy at Barnardo's, believes more funders should follow Big Lottery's lead. "Effective evaluation of initiatives supporting people with deeply entrenched problems will not get results immediately," he stresses.

Working in partnership

It took more than just a strong logic model to ensure Blackpool received its Better Start funding. According to Merle Davies, the Big Lottery was also impressed by the bid's focus on close partnership work across health, social care, and charity and justice sectors to improve families' lives, as well as pool data more effectively for evaluation.

From next year, a data sharing initiative with Blackpool Hospital Trust will enable the partnership to gather data on things such as accident and emergency visits, and alcohol-related health problems, to evaluate how interventions to combat domestic violence and addiction issues are performing.

This data is anonymous to allay concerns from health professionals around privacy.

Stockport is another area where evaluation of early intervention work is being supported by strong partnerships across children's professionals. Health visitors, family support workers, the youth offending team, substance misuse specialists and youth workers are part of an integrated children's services department. From April next year, this will be expanded further to include children's social care.

Viki Packman, Stockport's head of integrated children's services, says this combined approach means a wealth of data can be gathered to better evaluate whether families' lives are being turned around by early intervention work.

Data around breastfeeding, obesity rates and youth offending as well as information to evaluate the combined approach itself are all being used to monitor the success of its interventions.

This focus on information sharing across agencies is of particular interest to the Big Lottery, which is keen to measure the "collective impact" of how organisations work together to improve the lives of young people and families, says its knowledge and UK portfolio director Joe Ferns.

Involving service users

Qualitative evidence, where service users reveal how an early intervention initiative has turned their life around, can offer compelling evidence of success.

Gathering this kind of evidence is an important element of Barnardo's Five to Thrive scheme. The intervention was first developed in Hertfordshire in 2011 and supports parents to improve their attachment with their babies based around the themes of talk, play, relax, cuddle and respond.

Evaluation by Barnardo's to date has focused on interviews with parents to examine its impact on their lives, as well as with children's professionals to assess how well the approach is being adopted.

Interviews carried out at Barnardo's Kendal West Children's Centre in Cumbria and its Manchester-based Benchill Children's Centre were released in November 2015, and showed parents had a good understanding of the Five to Thrive messages and were actively acting on them at home.

"Having people telling you how Five to Thrive is changing their lives is very powerful stuff," says Barnardo's assistant director, policy and research, Jonathan Rallings.

But the EIF warns those using qualitative measures to avoid being too subjective.

It advocates the use of established measurement tools that have been carefully designed to avoid subjectivity, such as the Strengths and Difficulties Questionnaire used to measure emotional wellbeing.

Going it alone with your own measurements rarely works, says the EIF's Donna Molloy. "Examples of subjective measurement include asking if a family is stressed or whether their house is clean," she says. "That is based on a subjective baseline of what family stress is or how clean a house should be, so won't produce any reliable evidence."

NPC's Dan Corry advocates the use of both qualitative and quantitative evaluation as it gives a clearer picture of success to commissioners and funders.

"It can be a problem if you haven't got real life examples to back up your data," he says. "If you simply stick to quantitative data, you won't impress people who are moved by emotion."

Admitting mistakes

The Big Lottery wants those evaluating early intervention schemes to ensure they are open and honest about their failures, so others can learn from their mistakes, says Joe Ferns.

Being honest in this way will be seen favourably by the Big Lottery, which prioritises wider sector learning when assessing evaluation from schemes it is funding or looking to fund.

There are three levels of evaluation for the Big Lottery, he explains. First, the fund is looking for evidence that recipients are a "good grant holder" by proving they have spent the money in the way they said they would when making the bid. The second is about showing intervention has had a positive impact on people's lives. And third, the Big Lottery is interested in whether a scheme "provides something the wider community can learn from".

He believes a focus on shared learning and admitting mistakes requires something of a culture shift for many involved in early intervention schemes "who don't like talking about when they fail".

"We need to be more mature as a sector about how we talk about things that haven't worked," he says. "Organisations should be able to say 'here's what we did and it didn't work'."

USEFUL RESOURCES

The Early Intervention Foundation Guidebook

This interactive tool from the Early Intervention Foundation offers advice on evaluating early intervention initiatives and features its colour coded "standards of evidence". It includes references to latest research on early intervention schemes, as well as advice on funding and theories such as logic modelling.

guidebook.eif.org.uk

Preventing Gang and Youth Violence: Spotting Signals of Risk and Supporting Children and Young People

For those working on crime prevention-focused early intervention initiatives, the Early Intervention Foundation has produced this useful report looking at the best ways to identify children at risk of gang culture and youth violence. It also contains detailed evidence from the most successful programmes working in this field.

eif.org.uk/publication/preventing-gang-and-youth-violence

Making the Most of Funding: Learning to Improve Impact and Effectiveness

The Big Lottery Fund offers a handy guide on evaluating impact and measuring outcomes for those looking for funding early intervention projects. The guide also focuses on the importance of ensuring evaluation provides lessons to the wider voluntary sector.

biglotteryfund.org.uk/research/making-the-most-of-funding

Focusing on Impact: What do Trustees Need To Know?

This guide by New Philanthropy Capital is aimed at improving charity trustees' understanding of the importance of evaluation. This includes guidance on embedding "a culture of impact practice", case studies and advice on tackling the barriers charities face in effective impact measurement.

www.thinknpc.org/publications/focusing-on-impact-what-do-trustees-need-to-know

The Value of Early Intervention

This analysis of Barnardo's children's centre services provides a useful case study in ensuring cost-benefit analysis is a key focus of evaluation.

www.barnardos.org.uk/the_value_of_early_intervention.pdf