The evaluation of three 'Communities that Care' demonstration projects

Community-based early intervention and prevention programmes aim to tackle future social problems. The development of such programmes has become an important part of government's approach to community capacity building. This evaluation by the University of Sheffield of the JRF's Communities that Care (CtC) programme shows how one such approach can be implemented in deprived communities, what impacts it is having after five years of implementation and how these can be measured.

• The impact of CtC on levels of risk and protection was difficult to measure. In one project there had been a reduction in risk factors but it was not possible to attribute this to the influence of CtC. In a second, evaluation of local initiatives suggested that these new interventions might be making a difference. In the third, there was no evidence of impact.
• Success in implementing CtC varied. In one project implementation was broadly successful; in another implementation was partial; and in the third, the project remained largely unimplemented over the course of the research.
• Successful implementation was associated with the presence of strong partnerships, active communities and good leadership prior to the start of the programme.
• Having active and supportive 'champions' of the project among the senior executives of key local agencies was critical.
• It was important to have a wide range of professionals involved at operational and managerial level from the beginning - especially where specific programmes were to be implanted, such as in schools and social services.
• Partnerships worked best where structures and processes allowed for working between the strategic and the operational levels. When this happened, the pace of implementation increased.
• Early and comprehensive induction for new partners and staff was crucial for keeping projects 'on track'.
• Project co-ordinators were critical for success: maintaining momentum, increasing the active involvement of partners, and keeping projects focused on core objectives.
• Moving from planning to delivery was the most difficult part of the process. Success was more likely where the programme was built on the consent and active involvement of all involved.
• Money and resources were critical. Staff and local people found it frustrating when time and effort spent devising a plan were not matched with the resources to implement the work.

Background

In the mid-1990s the Joseph Rowntree Foundation (JRF) funded the Communities that Care (CtC) prevention initiative. This early intervention programme targets children living in communities and families that are deemed to put them at risk of developing social problems. The CtC approach focuses on small geographical areas and involves bringing together local community representatives, professionals working in the area and senior managers responsible for service management. Participants are given training, and provided with evidence of the levels of risk and protection in their community. From this they design an action plan that seeks to enhance existing services or introduce new ones likely to reduce risk. CtC is therefore not simply a service delivery programme, but a process leading to the identification of a programme of work, and a method of facilitating the delivery of well-co-ordinated services that reduce risk and increase protection. CtC does not deliver services itself, but facilitates and activates change in a local area.

In March 1998, JRF funded three demonstration projects in the UK to test whether this approach could be successful. The three project areas were unique and had their own characteristics, but were selected because they were identified as deprived areas with evidence of professionals and local people being committed to work together in tackling local social problems.

The evaluation

A research team from the University of Sheffield was commissioned to undertake an evaluation between 1998 and 2003, and report on the success of the programme. The research looked at the process by which the three projects developed and implemented CtC, and measured outcomes in terms of changes in risk and protective factors by means of school surveys undertaken before and after implementation.

The school surveys included questions about: pupils' personal and social circumstances, their families, neighbourhoods, and school experiences; the availability and use of alcohol, tobacco and other drugs; delinquent and anti-social behaviour; and spare time activities. Identical questionnaires were used for both surveys. Their responses to the various questions were put together to compose 16 'risk factors' and seven 'protective factors'.

The outcome evaluation involved comparing changes in the proportion of children at risk for each risk factor, or protected by each protection factor. The analysis compared those children who went to schools serving a CtC project area who actually lived in the CtC areas with those pupils who went to the same schools, but did not live in the CtC areas.

What was implemented over the five years?

Over the life of the programme each project developed differently. Although each project used the CtC method to measure risk and protection and to develop a programme of work, each one adapted the method to meet local circumstances. Across the programme the following were identified:

• All three projects managed to identify risk and protective factors, involve a wide range of partners and local people, and develop an Action Plan for delivery.
• Thirty-three initiatives or programmes of work had been planned in the three project Action Plans. Fifteen of these were delivered in total, eight of them being in one project.
• One project delivered only three limited programmes of work, all of which had a short life (less than six months), and therefore failed to implement as intended.
• Given the overall aim of preventing risk behaviours, the majority of implemented initiatives focused either on the parents of young children, or on children of primary and pre-school age.
• The number of parents and children who came into contact with CtC programmes was small in two of the projects. In the third project there was little monitoring of information about throughput, so it was difficult to assess the level of contract parents and children had with new services.
• In two of the areas primary schools were difficult to bring into the programme. This resulted in problems implementing services targeted at schools.

Impact of CtC

The evaluation covered a three-year period of programme implementation. However, the CtC approach is based on changing factors that affect childhood development, which are likely to take many years to bear fruit. Therefore it needs to be borne in mind that this evaluation could only ever give some early indications.

Two of the areas did not implement the CtC programme in the way intended, and it was therefore not possible to use them as a test of the CtC model. For the area where the CtC programme had been implemented as intended, there was evidence that life for children in the CtC area was improving. In general risk factors were on the decline, especially community- and family-based risk factors. This suggested that something positive may have been happening in the CtC area that was not happening elsewhere (or that something negative was occurring for children from other areas that was not affecting CtC children). However, it was not possible to be sure that this could be attributed to the intervention of CtC. In addition, there were positive developments in educational achievement in the CtC area in that Key Stage 2 tests in the local primary schools showed significant improvement during the CtC period of intervention. Again, although this could not be directly attributed to CtC, it was an indication of positive development for the future.

What worked in helping implementation?

Community readiness
It was clear from the research that the different starting points of each project affected how far it was able to make progress. The background of each project was very different, especially in terms of partnership working, geography and community involvement. For example, in one area having an active community development project that had good partnership working already increased the active engagement of local people and professionals in the CtC project. This helped improve implementation.

Partnership working
Having a wide range of partners involved in the programme is also critical. One of the unique qualities of CtC is that it recognises that making changes to service delivery or bringing in new practices and services within a geographical area needs to include people from all levels. This has four dimensions.

• Key leaders (strategic partners such as Heads of Education and Social Services, Chief Executives, etc.) are critical to this process.
• Having a wide range of key personnel involved at the operational and managerial level throughout the programme is important. It is especially valuable to have groups involved who are responsible for the delivery of services.
• Being 'joined up' - crossing not only agency boundaries but also hierarchical institutional boundaries between strategic and operational functions and/or the local community - is critical
  if successful implementation of early intervention and prevention is to be achieved.
• CtC puts much store on inducting participants into the programme early. Early intervention and prevention approaches to tackling social problems are still in their infancy. It has only been in recent years that prevention has become an issue for policy and practice. As a result the knowledge base and practice of professionals are limited, so having mechanisms for building up knowledge and understanding is critical if prevention is to become more established in service delivery.

Project co-ordinators
How projects are managed also affects implementation. Because CtC is a multi-agency programme that involves people from all levels of professional practice, arrangements for communication amongst those involved are especially important. If problems exist with communication then, as this evaluation showed, problems can emerge with the practice. The role and management of co-ordinators are critical here. They are essential for helping the programme make progress and maintain a focus on key objectives. This was most apparent when co-ordinators left a project. How each project managed the loss of co-ordinators greatly affected how fast the project could develop and deliver.

Moving from action plan to delivery
This stage of the process is one of the most difficult, full of potential problems that can undermine any hopes of success a project might have. Moving from planning to action requires substantial effort and a lot of hard work on the part of all parties involved. An Action Plan has to be constructed as a consensus of all parties and agencies at both operational and strategic levels, otherwise the chances of successful implementation are limited. It is also the case that constructing an Action Plan that has a wide range of interventions allows for failure. Even when it seems that no problems exist, problems can emerge that undermine implementation. Constructing an Action Plan that recognises this likelihood helps to ensure that the programme is not threatened.

Conclusion and lessons for policy and practice

The evaluation remained inconclusive about the impact the three CtC projects had on levels of risk and protection. At one level this is unsurprising as CtC aims to be a long-term community intervention programme and it is unlikely to show its effects at this stage of the process. But it is also the case that evaluating community-based programmes is problematic: it remains difficult to isolate the impacts a programme like CtC has had on levels of risk and protection.
However, even if the level of impact remains unclear, evidence indicated that, if implemented well, the CtC approach could make a long-term contribution to the development of services and maybe also on levels of risk and protection. Its strong commitment to evidence-based approaches in defining the problem, in identifying programmes and in implementation provides a real opportunity for future success. While there is still much to learn about measuring and reducing risk and implementing these types of programmes, the results of this evaluation show that a national policy of increasing resources towards this form of evidence-based prevention, at both national and local level, could well pay long-term dividends.

The evaluation also showed that there are positive lessons for local policy-making and for professional practice:

• The CtC approach to using evidence gathered locally clearly offers an opportunity to build a strong evidence base that will help local policy-makers and practitioners develop effective measures of risk and protection.
• While CtC UK still needs to resolve technical problems, its risk assessment model and, in particular, the school-based self-report survey do offer a way of providing evidence of risk and protection in the locality. Collecting self-report data from children and young people about their behaviour and attitudes is, as the evaluation shows, a potentially powerful tool. It also offers the opportunity of long-term measurement and evaluation.
• The process of auditing risk itself is also a very powerful tool. Local professionals and communities find the process of analysing the data and making priority decisions based on evidence very useful in helping them construct services that are evidence-based.
• Being involved in the process is also beneficial for participants: as people become more involved in the programme of assessment, they also become more knowledgeable about risk and protection. CtC offers a route into developing local capacity and knowledge about local levels of risk.
• While it seems to be stating the obvious, CtC has also shown that the process of measuring risk and developing and implementing interventions in Children's Services is complex, time-consuming and requires strong leadership from above.

The CtC approach also shows how some of the problems highlighted in multi-agency practice can be overcome.

• Evidence from this evaluation shows how the process of assessment, action planning and implementation can aid multi-agency practice by giving a forum for joint working around objectives that are relevant to all partners.
• Professional workers and local people also highlighted the importance of having training and support in this work, recognising that professionals need access to other forms of information to ensure that best practice is achieved.