Evidencing outcomes on the agenda as the next Spending Review looms
Tuesday, February 18, 2014
CYP Now's Outcomes Conference earlier this month brought together leaders in children's services across the voluntary sector, local authorities and government to debate the way ahead for evidencing what works.
With public services and voluntary organisations continuing to feel the squeeze of financial austerity, the need to make every penny count is greater than ever.
CYP Now's Outcomes Conference on 5 February brought together an array of figures in the sector to discuss best practice on delivering services for children, young people, and families. Delegates heard from leading figures in the children and young people's sector about how to get the best out of services through use of evidence and expertise.
Making better use of resources for troubled children
Andrew Webb, president of the Association of Director's of Children's Services (ADCS), said local authorities must focus on restructuring the highest areas of spending – namely residential care and youth custody – if they are to be able to meet the challenge of delivering improved outcomes at reduced cost.
"The challenge for us at a local level is how do we reimagine what we are trying to achieve and then reimagine how we do it," he said.
Webb said most areas conduct at least five population-level needs assessments that will feed into anywhere between six and 10 strategic plans, informing the activities of eight or more commissioners across health and wellbeing, youth justice and a range of other services.
"When we talk about reimagining, there's a plethora of services," he said. "If we just look at young people involved with crime, we get competing services within the same priority area. Child and adolescent mental health services and substance abuse services compete for resources to work with the same children. Many of those children will be at schools that have invested a lot of pastoral time in supporting keeping them learning while they are in school, but with no connection to what's going on outside the school gates.
"And the fragmentation of schools has driven a further wedge between schools and the services that are outside."
He went on: "We need to turn the huge number of competing needs assessments and strategies into a single design process."
"Services that currently have competing agendas could be brought into one?
"How might that make a big difference for far less money to that overlapping, dangerous behaviour cohort that every area has? That's the real challenge we have to address quickly, because the Spending Review of 2015 isn't very far away, and if we don't have some models that can do this within the next few months, I have no idea what we are going to do. We cannot continue to do what we are currently doing. If you strip some of our services down to being any smaller than they already are, they simply won't be worth having."
Not measuring for measuring's sake
Kathy Evans, chief executive of Children England, said the voluntary sector is supportive of an increased focus on outcomes, pointing to the fact that prominent examples of evidence and evaluation already come from the sector.
"Family intervention programmes - pioneered by Action for Children - are incredibly well evaluated and the evidence is now informing troubled families," she said.
But she went on to warn that too much of a focus on outcomes and evidence by local authority commissioners can deter prospective voluntary sector providers.
"At the moment, my sense is we are quite deluged and frenzied in the challenge of evidencing what our impact is," she said.
"The voluntary sector has exactly the same belief as the policy agenda behind outcomes for children - that really making a difference in children's lives is the only goal in town.
"All of our donors have always wanted evidence and demanded examples.
"But I remember one organisation offering £20,000 and asking how you will deliver outcomes on crime reduction, reduced drug and alcohol use, and education and learning attainment, in one year, and how you will prove it?
"This was about being clear that those were the council's objectives.
"But for £20,000, that was one of the most deterrent exercises I can possibly imagine."
She added that while the sector is prepared to work to targets, a lot of what it achieves is less tangible.
"Much of what the voluntary sector is there to do is not a service," she said.
"We are not a school. We are not set up to take a child at one point, do something sustained and see how it looks at the end.
"We are doing something to fill a gap.
"It may be providing someone to talk to unconditionally.
"We all know the examples of Samaritans and ChildLine. Do we want to test these provisions against the question of whether they actually prove they save lives in the end, or do we instinctively know that's a useful thing to be available regardless?
"There are some parts of what the voluntary sector is there to do that are about being there and doing the right things.
"Not everything we do is about achieving an outcome."
Ensuring what works can be replicated
Youth charity Catch22 is currently leading the consortium delivering the Realising Ambition scheme, which is investing £25m over five years in projects to prevent young people becoming involved in criminal activity and antisocial behaviour.
The scheme is specifically backing intervention programmes with a track record of success, either at home or abroad, in order to gain evidence of how they can be successfully launched in other areas.
Sally Morris, director of children, young people and families at Catch22, said: "We are trying to understand what the significant factors are that help and support effective replication.
"It is all very well having an evidence-based programme that worked in a place in America three years ago, but if you actually re-provide that in a context in Hull or Salford, you will obviously come across significant differences that you will need to adapt and take account of."
Morris said the learning to date is that there is a lack of evidence-based programmes to choose from.
"Those that meet the highest standard randomised controlled trial experimental standard are very few and far between," she said.
"You have a whole range of evidence in between that, whether it is feedback from service users, feedback from observations or other kinds of evidence you might collect.
"The number is less than 50 that are actually recorded for children and families nationally for those high levels of evidence."
Morris added that although Realising Ambition hopes to create a clear framework for how to set up successful projects in different areas, it does not want to create an environment where working to a textbook becomes the norm at the expense of initiative.
She gives the example of a young mother who pinpoints the moment her life changed when a social worker spent a couple of hours teaching her to cook a meal that her late mother used to make.
"That had proved to be a real turning point," she said. "Just because it was the worker noticing the key thing that was going to help turn things around. For us, that's the kind of things we want to be able to say to practitioners.
"These are the sorts of things you should be looking for, this is the sort of context you are working in and these are the kind of practices we think you should be making reference to.
"You are drawing on a range of support and resources that are defined, but you still have the ability to react in a human way.
"The worst thing you could take away from this is that we want people to be operating in a robot fashion."
Comparing your outcomes with others
Delegates also heard about research being carried out by independent consultant John Freeman into annual spending by councils on children's services.
Since 2010, local authorities have been required, under section 251 of the Apprenticeships, Skills, Children and Learning Act 2009, to submit annual budget returns to the Education Secretary, detailing planned and actual expenditure for their education and children's social care functions.
Freeman said that interim findings of his research on the returns show that, even though the government often uses the data to make funding decisions, it is not particularly reliable.
He cited data from the latest returns that show spending on social care per child ranges from £400 to £1,500 across all local authorities, while the cost per year of children in care ranges from £35,000 to £70,000.
Freeman said this can lead to a belief in government that savings can be made in areas that appear to spend more for certain services, but explained that the truth is not as simple as that.
The Section 251 returns mask a number of variations between different areas - "extrinsic factors" such as size, numbers of children, demographic change, population churn (the number of people moving in and out of the area) and rurality.
There are also "intrinsic factors" such as the amount an area invests in early intervention, and how it uses residential accommodation or its fostering policy.
Freeman said that understanding the context of the figures is essential
"We need to include activity levels and outcomes - we can't just base it on spend," he said.
"We need to get comparisons as good as possible because measuring against others is a valuable tool," he added.
Gillian Hillier, deputy director at the Department for Education's young people division, had earlier said that her department will be working with the Department for Communities and Local Government, local authority chief executives, the Local Government Association and ADCS over the coming months in order to improve the reliability of data.
"We are conscious we have more data than we use, and financial data is not as reliable as it could be and often emerges after the date it applies to," she said.
The vision, she added, is to create cost and outcomes data that combines information on spending and outcomes in a way that allows authorities of similar size and scope to compare themselves.