First produced as a briefing for Alliance for Useful Evidence/SOLACE Foundation policy seminar
As the axe swings on funding, we need the best available evidence to tell us how to get the most from our budgets. If the grave Audit Commission predictions are correct, local authorities will need to find 26% further cuts in 2014/15, on top of the ones that have already kicked-in. Robust research and analysis can help point to where we might do more with less.
But what evidence do we need? Grand scale high-cost evaluations with armies of analysts are not be feasible in a time of austerity. Similarly, evidence that is not robust will not help us find interventions that work. We are going to need to be realistic about what evidence we can marshal to help local services. The evidence has to be proportionate to the challenge.
Fortunately, some evidence is free and accessible. The Education Endowment Foundation, for instance, has a ‘pupil premium toolkit’ that is popular with schools and local decision-makers at it gives an easily-digested but robust snapshot of the evidence of class sizes, teaching assistants, homework and other features on educational attainment. But some councils have taken a system-wide approach to using evidence to improve local services. Birmingham City Council has hardwired research into decision-making. At the highest level in the city – and across political parties – it has been recognised that current decision making, especially on resource allocation is partially ‘flying blind’, without the evidence needed to work out priorities between major spending programmes. What Birmingham has done is to move towards ‘outcome focussed planning’. To put it simply, this means starting with desired outcomes, testing different approaches and then re-designing services, rather than the more traditional focus on inputs and processes. Evidence has to underpin all of this.
An example of this in practice in Birmingham can be seen in Shard End Community budget pilot that helps families with complex needs. Based in one of the most deprived areas in the east of city, Birmingham has implemented a variety of family interventions based on a depth of evidence, such as ‘Triple P’, ‘Incredible Years’, ‘PATHS’ and Family Nurse Partnerships. Hopefully it will be a win/win situation – not only will this result in better outcomes for families, it will create long term financial benefits for the programme. They have mapped out a whole range of longer term fiscal savings through these early interventions across all the Birmingham wards.
Question 1: what evidence do we need in a time of austerity to help local decision-making?
More local bang for the buck
The Birmingham approach is ambitious in its desire to transform thinking on social outcomes. But there can also be smaller, targeted interventions based on technical analysis that can help save money. The waste company WRAP have used data analysis to help West Oxfordshire District Council save around £500,000 a year on an integrated refuse, recycling and waste collection service. WRAP are a good example of an external body providing practical, research-based advice, creating smarter waste and recycling services, and also helping with procurement strategy and tender documentation.
We also need evidence to see if high profile cost-saving initiatives are actually doing their job – and saving us money. Do we really know, for instance, if shared services make a difference? A report from the Local Government Association says that five local shared services arrangements have saved a total of £30m through a range of measures, including integrating IT systems and better procurement. The report was the first to provide a detailed insight into the scale of savings that have been achieved through sharing back office functions like IT and legal, and teaming up to deliver frontline services like waste disposal and road maintenance.
The significance of this LGA report is that it highlights how we must evaluate and test local initiatives – sometimes ‘in-flight’ rather than waiting until it’s too late at the end – to see if efficiency savings are really being made. The evidence form this LGA report is useable across local government – alongside the report, they built a new, accessible, Excel-based prototype tool to understand the benefits of sharing front and back office services. It will help others develop baseline information to track the level of savings and performance benefits that can be achieved.
Local authorities have a good story to tell central government in raising productivity. We also need to learn from the bad stories of where things aren’t working. In these difficult financial times, we can’t afford the luxury of ploughing on with things that don’t work. It will, of course, take considerable political mettle and media management to shut down projects, programmes and policies that are not working. Trying to explain that particular services will be cut because of a robust cost-benefit analysis will not always cut it with voters, journalists or the political opposition. But without credible evidence it may be much harder to make the case for the cutting of particular services.
Tapping into new sources of income
There is another financial bonus to evidence. Councils are making evidence work for them by finding alternatives sources of income through social finance. In order to scale up community-based budgets like Shard End, Birmingham are working with the Cabinet Office to develop evidence to make the business case for social impact bonds. Birmingham’s ‘Brighter Futures’ project uses robust cost-benefit analysis to inform future investments. Based on methodology from the Washington State Institute for Public Policy, it will seek ways of reallocating resources away from expensive late interventions to greater investment in early intervention. It is hoped that the model will be used by other councils to predict savings, track financial efficiencies, and provide an evidence base to inform future commissioning.
There are, however, significant challenges with this social investment model. As the chief executive of Birmingham Stephen Hughes pointed out in a recent speech, it is not always clear if the evidence exists to back up social investment bids. And if we do have good evidence to help better outcomes, is that always translated into predictable cost savings?
Family Nurse Partnerships is another example of using evidence-based approaches to access new national resources. Run by the NHS and local authorities, the Government has committed to doubling the evidence-based capacity to at least 13,000 places by 2015. This scheme is a highly effective approach for young first time mothers. It was inspired by 30 years of evidence from the US on the benefits of intensive home visits by specially trained family nurses, from early pregnancy until the child is two. The outcomes are wide ranging, from improved early language development, to reductions in children’s injuries and abuse. Although we don’t yet know what the savings might be for UK local authorities, cost savings in the US are around US$ 3-5 return for every dollar invested. A positive recent formative evaluation in England pointed to the potential for early impacts and for cost savings. We will also find more about its value to local authorities in 2013, when a large scale research trial will report early findings. But the benefits look highly promising and could work across local areas.
Sources of information on ‘what works’
It can be a bemusing experience finding sources of information on ‘what works’. Take education as an example. There are a plethora of online resources, ranging from targeted easy-to-read online resources like the Pupil Premium Toolkit, to great big online libraries of meta-analysis, such as the EPPI Centre. The Department for Education itself does not have a single bank of evidence available across children’s services, but there are various resources that local areas can access. In particular, the Centre for Excellence and Outcomes reviews, interprets and publishes the evidence base online. Evidence is also being driven through commissioning and productivity by the Children’s Improvement Board (that includes SOLACE, LGA, Department for Education and others). The Board’s remit includes reviewing evidence on ‘what works’. But how many people are aware of these resources?
Some analysts in government departments also provide a ‘what works’ function. The Department for Work and Pensions, for example, publishes a wealth of evidence. The 160-page DWP report into ‘What Works for Whom’ reviewed the stock of evaluation in order to identify which interventions have worked most effectively for key customer groups, such as young people and long-term unemployed. An informal group of central government analysts working on localism and decentralisation has also been set up – the Localism Analysts’ Group. They may have a role in supporting evidence for local needs.
Question 2: would a single online evidence hub pointing to all available resources on ‘what works’ be useful for local decision-makers?.
One new source of evidence is on the horizon. New ‘what works’ institutes in key areas of social policy are being developed by the Government. Trailed in in the Civil Service Reform Plan and Open Public Services White Paper, the proposed new infrastructure is sometimes referred to as ‘NICE-type institutes’, following the National Institute for Health and Clinical Excellence in the health service. The plans for this new infrastructure are still being worked out by the Cabinet Office and others – including the BIG Lottery, ESRC and Nesta. Current thinking is that there will be sector-specific centres – perhaps with one central coordinating body – and that they will synthesise and communicate the evidence base to decision makers, commissioners and practitioners. Although no institutes have been finalised, there is currently exploratory work underway on sector-specific ‘what works’ institutes focusing on ageing well and local economic growth. These institutes would seek to synthesise and disseminate evidence relevant both to national policymakers and local commissioners and providers.
Question 3: what would be the most useful function these national ‘what works’ institutes could provide to help local decision-makers?
Collaborations and scaling up
A common theme for many successful local evidence-based initiatives is the need to work with partners. Success – and new sources of income – often comes through working alongside other national and local bodies. New Economy Manchester – a commission to advance economic growth and prosperity across Greater Manchester – have committed to using national partners to drive the adoption of new approaches, and have been working with Whitehall,other large cities and international partners, such as the Washington State Institute for Public Policy. They have been in the vanguard, along with peers such as Birmingham and Westminster, in developing an evidence-based approach to the design, implementation and funding of local services, using new methods of cost-benefit analysis, Randomized Controlled Trials and other techniques. Their use of evidence was a vital part of the debate around Manchester’s ‘city deal’. Manchester focused on how to extract local revenues, such as the proposals to ‘earnback’ taxes from central government and plough back into local economy. Evidence was a key part in their success in making the case for their ‘city deal’ and raised the quality of debate about how to boost growth, based on sound research.
One challenge is that lessons of ‘what works’ from local areas may not be applicable elsewhere. Examples of social interventions in say, Fife, may not be relevant to Birmingham’s Shard End. Smaller sample sizes mean lessons learnt will be specific to local context and not generalizable to other areas. There may also not be consistent ways of measuring and collecting data – leading to even less chances of comparing local areas to see if things are working. There are therefore potential economies of scale and benefits from ensuring, perhaps by some national body, that information and evidence is produced and disseminated effectively.
Collaboration between local projects and programmes in gathering evidence must be one way forward. The Mayor of London’s Project Oracle is one attempt to bolster collaboration between social intervention projects, government and academia. Not only have they brought together disparate groups to figure out ‘what works’, they have also set down a common standard of evidence for everybody to buy into.
Project Oracle arose from the Mayor’s ‘Time for Action’, a programme to tackle serious youth violence and to ‘understand and share what really works’. It was set up in response to a recognition that there was not a clear understanding of what programmes work to tackle youth violence. They are building a sustainable body of evidence to help future policymaking, and have a consistent method and standard of evidence. Oracle employs an online ‘refereed’ resource to share examples of good practice across the capital. So single projects with names such as The Gang*Star Project Growing Against Gangs and Violence, Safer Children Project and many more, are working to an agreed set of standards of evidence, so that they can collectively figure out ‘what works’, alongside the Met Police, Youth Justice Board, Director of Children’s Services – and academia. It’s a remarkable collaboration and may offer a model for collaboration for other local social initiatives
Question 4: what collaborations are needed to help provide better evidence on ‘what works’?
Question 5: what can central government do to support greater local evidence-informed decision-making and practice?
Despite the need for research, local analytical capacity has reduced over recent years as headcount falls. Now, more than ever, analytical teams have to justify their existence to senior officers and councillors. At a recent conference of local analysts and researchers, the chief executive of Coventry Council and President-elect of SOLACE Martin Reeves said that researchers must “focus on outcomes for the people we serve – nothing else matters”. Analysts and researchers should view this as a “moment in time to transform the profession”, using their insights to provide evidence on effectiveness, not just efficiency, manage and reduce demand and design new services. There are many challenges in finding ‘what works’ in local areas. Decision-makers in local government, charities or business may not have the capacity or capability to access and interpret evidence on ‘what works’. Central government could have a role here, supporting local areas by providing more access to data and knowledge. For example, there might be some centralised standards for evaluation, kitemarks on effective interventions or user-friendly synthesis on the latest research. The forthcoming ‘what works’ institutes flagged up in the recent Civil Service Reform Plan may help provide this service to local authorities.
But there is no point getting the supply of evidence right if there is no demand for it in the first place. If we can get the incentives right on the demand-side, we may also find a body of useful evidence grows of its own accord. Payment by Results and social investment models may provide the motivation/incentives, rewarding local areas for achieving outcomes based on evidence-based approaches.
We have to make every pound work harder and research, evaluation and evidence are key to this. We need to make sure we gather intelligence that can really help to make a difference to people on the ground. It needs to help make savings and efficiencies, striving for more for less. We can’t afford the luxury of ivory towers and research for research’s sake. But intelligence and analysis can help focus our minds – and shrinking resources – on ‘what works’.
Jonathan Breckon, Manager, Alliance for Useful Evidence
 Bovaird, Tony and Kenny, Richard (2012), Modelling Birmingham: Using Strategy Maps to Compare Outcome Pathways, Paper presented to Performance Management Association Conference, University of Cambridge, 11-13 July 2012.
 This research follows the recent publication of the Commons Public Accounts Committee report into shared services which demonstrated that local authorities are significantly outperforming central government in this area. http://www.parliament.uk/pac
 See for example, the 160 page review of evidence and meta-analysis for policy areas covered by DWP http://research.dwp.gov.uk/asd/asd5/rports2007-2008/rrep407.pdf
 A report based on interviews with 25 Chief Executives concludes that “monitoring and evaluation roles appear to be one of the foremost casualties of back office efficiencies in many, if not all, authorities” Serving deprived communities in a recession, Annette Hastings, Glen Bramley, Nick Bailey and David Watkins, January 2012, Joseph Rowntree Foundation http://www.jrf.org.uk/publications/serving-deprived-communities-recession