Annex E - Evaluation Framework
Introduction
- In line with the Review of the Rural White Paper, the Rural PSA Delivery Plan and Lord Haskins' comments on the importance of Defra's evaluation role, this section sets out an evaluation framework between 2004-8 for measuring outcomes under the Rural Strategy 2004 [Footnote].
Background
- Evaluation is the process which objectively judges the actual outcomes, including any unintended side effects, of a policy or group of policies against the policy objectives, or intended outcomes, and the resources that are used in policy delivery (see Figure 1 below). Evaluation forms an essential part of the Government's Rural Strategy 2004.
Figure 1: Summary of evaluation process
- In designing the rural policy evaluation framework a number of key themes, common to most evaluation activity, need to be addressed:
- Relevance - is our approach to rural policy consistent with the problem?
- Effectiveness - how far have our rural policy objectives been achieved?
- Additionality and side effects - to what extent would the rural policy outcomes have been achieved anyway and are there any unforeseen side effects?
- Efficiency - what have been the costs (and on whom do these fall) of achieving the rural policy objectives and what have been the causes of any inefficiencies?
- Coherence - do the individual policy and programme measures combine to achieve the higher level rural policy objectives in a coherent fashion and are there any synergies or inconsistencies between policy instruments?
- Durability - will changes brought about by rural policies be self-sustaining and continue in the long run or will there be an ongoing need for policy intervention?
- A number of steps are necessary to ensure that any evaluation activity is able to generate useful results in addressing these key themes:
- a coherent set of structured and agreed rural policy objectives, specified in terms of intended outcomes, needs to be clearly defined;
- well defined links must be established between intended rural policy outcomes and measurable indicators which provide evidence of the extent to which the outcome is actually being achieved and how specific policy measures are contributing to it;
- the results of monitoring activities and other relevant data sources will need to be easily available for analysis in order to provide objective evidence of how effective rural policy has been in delivering its intended outcomes, whether these could have been delivered more efficiently and if there have been any unforeseen side effects;
- stakeholders who have an interest in the policy area, including those affected by the policy, and those who designed, managed and implemented the policy (such as other Government Departments, Agencies and regional and local partners), need to be actively engaged in the evaluation process; and
- a mechanism needs to be established which provides for the results of evaluation activity to feed back into the policy process in order to improve future policy design and delivery.
Evaluation framework rationale
- The starting point for the development of a coherent policy evaluation framework will be clarification of the intended outcomes and the way in which these relate to the policies in place. This process will inform the selection of indicators against which to measure progress. In addition, the large number of programmes impacting in rural areas means that a prioritisation exercise will be carried out to identify the key policies for the achievement of the intended outcomes; these policies will be the subject of programme-based evaluation.
- A number of indicators will be used to monitor progress towards achieving rural policy outcomes. However, changes in the indicator values will reflect a large number of factors beyond the scope of policy. Thus, further work will be done aimed at establishing the extent to which rural policy is making a difference. Attitudinal assessment surveys will help to enrich our analysis along with the potential development of longitudinal, qualitative data sources such as the British Household Panel Survey.
- Similarly, evaluations of individual schemes and the rural proofing of evaluations of other Government Departments' programmes will not, in isolation, tell us whether rural policy is working as a whole. Therefore, additional work will be needed to ascertain whether policies in rural areas are mutually reinforcing and contribute to the achievement of Defra's high level priorities. One option will be for case studies in rural areas to look at the cross-cutting impacts of rural policy as a whole, set within a sustainable development context, reflecting the balance between environmental, economic and social factors.
Evaluation overview
- Taking these points into account, the approach taken to evaluating rural policy contains a number of distinct work streams led by Defra:
- statistical development to establish a baseline and to monitor over time a range of indicators for key policies which will then feed into their respective evaluations. The Rural Evidence Hub will act as a repository of this statistical data;
- monitoring progress in achieving rural policy outcomes against a basket of indicators identified in the Rural White Paper and which will be reviewed by the Rural Evidence Research Centre;
- monitoring progress in achieving Defra's Rural PSA against a proxy headline productivity measure and a suite of intermediary indicators;
- working with other Government Departments and Agencies in the programme-based evaluation of their policies to ensure that full account is taken of rural impacts. The large number of policies affecting rural areas means that Defra's resources will be focussed on those areas where policy is likely to have had the greatest impact;
- reporting on an England-wide study of the knowledge, take-up and attitudes to local services by both individual residents and business managers, following its initial run earlier this year. It will compare rural areas not close to services with more accessible rural areas and with urban areas; and
- a longitudinal study (a survey following a cohort of individuals through time, such as the ongoing British Household Panel Survey) to further our understanding of the socio-economic effects of policy on households in rural England may be used. Alternatively, case studies based around a small number of rural communities may be another way of providing depth to our analysis. These possibilities are subject to a scoping exercise currently being undertaken by the Rural Evidence Research Centre.
- An overarching rural policy evaluation will bring together the information and analysis collected through the six workstreams outlined above. This overarching evaluation will be carried out in time to inform the 2008 Spending Review. It will provide the first full evaluation of the Rural Strategy as well as an assessment of rural policy since the publication of the Rural White Paper in 2000.
Stakeholder engagement
- The evaluation process requires Defra to work alongside other Government Departments, Agencies and other stakeholders in order that full account is taken of the impact of policy in rural areas. In addition, and in accordance with standard Defra practice, the results of evaluation studies will be placed in the public domain generally via the Defra website (www.defra.gov.uk).
Page last modified:
19 May, 2005
Page published: 21 July, 2004