Department for Environment, Food & Rural Affairs

Financial Management and Policy Review of the
Royal Commission on Environmental Pollution

3. Achievements

3.1 To assess the Commission's achievements, we wanted to review the extent to which it has had an impact on real world outcomes, including influencing the considerations which are taken into account in the development of policy. This involves the difficult task of trying to disentangle the specific contribution of the Commission from the underlying matrix of shifting concerns and priorities. The 1998 research report by Susan Owens and Tim Rayner identifies a variety of subtle ways in which the Commission's work makes an impact. These include visible action generation (such as changes in legislation, procedures or institutional structures) enlightenment (changing belief systems and policy frameworks), opening up policy communities (allowing new voices to be heard), doing good by stealth (where Commission interest in a topic can push issues up the Whitehall agenda and stimulate learning in government departments), and education.

3.2 There is no easy or mechanistic way to gauge the impact of Commission reports. As Owens points out, different audiences - including Government Departments, industry and environmental groups - apply different criteria when judging the quality and usefulness of the Commission's reports, and those that are considered mould breaking and radical by one audience may be dismissed as ill conceived and impractical by others. Furthermore, given the wide range of issues addressed by the Commission, the intended impacts of reports may differ, so an evaluation model developed for one report will not necessarily be appropriate for another. Another issue is timing: initial reactions to the Commission's ideas can often be critical, but as issues are debated and policy developed, the impractical can come to look like the obvious. Reports therefore tend to look better with a number of years of hindsight.

3.3 There is at present no formal process for evaluation of reports. The Commission has announced its intention to introduce an evaluation process to take place three years after publication of each report, and we welcome this insofar as it should provide a structured basis on which to reflect on the impact of reports, while also serving to refocus attention on any outstanding recommendations.

3.4 We consider it is important that the evaluation process should be firmly rooted in the Commission's planning and review processes. It should be possible to link the evaluation of a report (or other Commission output) back to the stated purpose of the study; the originally intended impacts and influences (including indicators and timescales); and the strategy adopted for achieving those impacts and influences (including definition of audiences). The Commission's current corporate planning arrangements do not encompass all these elements, nor the linkages between them. We return to this topic in Chapter 7 in discussion of corporate planning and in Chapter 9 on working methods.

3.5 In the absence of an evaluation process, we looked at the Commission's various outputs over the last seven years. The most tangible of these is published reports and we questioned interviewees about perceptions of the value and influence of those produced over the review period. We also looked at sales of these documents to try and get some feel for the extent to which these might be reaching outside government and the policy community. We examined other significant outputs, most notably Commission statements and contributions to consultations, and activities aimed at ensuring the thinking in Commission reports can be more widely understood.

Published reports
3.6 With the caveats in 3.2 in mind, we asked for reactions to Commission reports published over the last seven years. Our discussions identified four cross-cutting themes, and these are discussed below:

(i) A significant number of interviewees outside RCEP commented on the exemplary quality of the Commission's scientific work. We understand that the science base of reports has never been subject to criticism, although the interpretation the Commission puts on findings and how this translates into recommendations has often been the subject of legitimate and heated debate. Scientific excellence was seen by many as crucial to the Commission's credibility and vital to maintain. This has implications for membership criteria and for the composition of the Secretariat, which are discussed in Chapters 10 and 11 below.

(ii) There was concern among several interviewees that although the Commission does seek to integrate the economic, social and moral dimensions of environmental issues into its deliberations, it has not always been successful in reporting clearly the tensions, sometimes apparently irreconcilable, between different perspectives. However, we note that in recent reports the Commission has devoted considerable effort to exposing the differences between perspectives - for example a chapter in the first transport report set out six different perspectives on the future of surface transport, and the whole purpose of the Standards Report was to subject different perspectives to critical examination. Underlying the concerns of interviewees may have been the real difficulty of integrating or reconciling the different perspectives in a way which makes possible practical recommendations on the way forward, as well as fears both that a particular perspective might be overlooked and that in any recommendation there might be losers as well as winners. We return to this and related issues in Chapter 9 on working methods.

(iii) Some interviewees felt that Commission reports had remained rather more UK-centric than was justified by the increasingly European and international context of environmental policy making, though we were not given concrete examples of relevant international work that had not been taken into account in preparing reports. Critics did however recognise that the need to look outwards had to be balanced against the risk of duplicating efforts of international research organisations, e.g. IPCC on climate change, and acknowledged the difficulties of keeping up to date with the vast numbers of players in the field. By contrast, Commission members and the Secretariat pointed to the efforts made in the Transport and Standards reports to take account of developments outside of the UK, and told us of several initiatives to improve links with bodies such as the European Commission, and with the advisory bodies of other European countries. The issue is how best to build up these links without excessive diversion of the Commission's resources. We return to this in the chapter on working methods.

(iv) There was concern among some interviewees about the subjects the Commission selected for study. Some felt there were surprising gaps in the Commission's coverage of issues - climate change, fisheries and food safety were all mentioned as obvious topics which the Commission could have addressed. However, we understand that the first of these was not considered for study because significant work was already taking place elsewhere (IPCC), and the second was examined on the last two occasions when a subject for study was selected (as part of a wider project on marine environment) but did not emerge as the front runner.

3.7 Interviewees had varying degrees of awareness and involvement with the individual reports produced by the Commission during the review period. The comments made to us are insufficient to provide a detailed assessment of each report. However, we set out below a summary of comments put forward.

Sixteenth Report, 1992, Fresh Water Quality
3.8 After the longest period of gestation of any RCEP report, this publication was considered by government to be mis-timed and over long. Some RCEP members accepted this was fair criticism, and commented that one outcome had been to focus attention on the need for improved strategic planning within the Commission. Nevertheless, the precautionary approach the report espoused was welcomed by environmentalists and eventually fed into the regulatory structure.

Seventeenth Report, May 1993, Incineration of Waste
3.9 Some interviewees considered this report was influential in the development of new regulatory regimes. It is credited with ensuring that the Government published a waste management strategy, and that HMIP took a robust line on closing down incinerators which did not comply with EC standards. A more general view, held by environmental groups and Commission Members, was that the narrow focus of the study, in not looking at all aspects of waste management including recycling, had limited its value. Moreover, the report failed to foresee the use of waste solvents as fuels in cement works, which has remained a controversial issue ever since. This experience emphasised the difficulties of reaching robust conclusions when considering only one aspect of complex issues, and the Commission drew from it important lessons on scoping of studies. Some people within government felt the report's contribution to the policy debate had been small and made less impact than reports by the Parliamentary Select Committee on the Environment led by Sir Hugh Rossi.

Eighteenth Report, October 1994, Transport and the Environment
3.10 The first report on transport and the environment exemplifies the way perceptions of the Commission's work change over time. When first published, many people had questioned the report's practicality, and there was particular criticism of unrealistic targets undermining the credibility of the report. Five years on, there was virtual unanimity that the report had made an important contribution to unblocking politically stuck positions by focusing attention on the longer term problems.

3.11 Interviewees from within DETR acknowledged that the report had strongly influenced Transport Ministers within the previous and present administrations. They described the report as mould breaking and responsible for introducing a new element of realism into the public debate - the proof of which lies in the Transport White Paper "A New Deal for Transport: Better for Everyone". Other Departments were critical about what they saw as the impracticality of the targets, and regretted the lack of experienced transport professionals among the Commission's membership. But they acknowledged that the Commission had highlighted the existence of a fundamental problem and set the agenda for debate, and that quantified targets usefully provided something to bite on irrespective of whether they were subsequently adopted or rejected. Likewise environmental groups considered the report to be highly influential, although they believed that wider interest was also stimulated by high profile campaigns underway at the same time such as protests against the Newbury bypass. A small number of those who were members at the time of drafting reported reservations about the targets and recommendations, although there was no attempt to introduce a minority report. Other members pointed out that several targets which had been scorned at the time of publication had now been achieved.

3.12 Overall, there was a general consensus that the 1998 Transport White Paper owed much to the Commission's work and to the constructive debates which it had helped to generate. The White Paper itself specifically acknowledges the Royal Commission's Eighteenth and Twentieth Reports as key influences.

Nineteenth Report, February 1996, Sustainable Use of Soil
3.13 This completed the Commission's look at the three media, air, water and soil. After the high profile response to the Transport study, it appeared to make less of an impact, and several of our interviewees confessed they were unaware of its recommendations. One Department was critical of what it saw as the Commission's capture by the soil science industry, which it believed resulted in too much precaution based on too little science. But it acknowledged that the report integrated and articulated the existing position in a useful and coherent way, and helped enforcement bodies to develop a rational approach to soil issues. Other government interviewees saw the report as directly responsible for the first soil protection strategy, but thought insufficient time had elapsed to evaluate its full impact.

Twentieth Report, September 1997, Transport and the Environment - Developments since 1994
3.14 The report on developments since 1994 on transport and the environment was produced in 1997 following much debate within the Commission. On one hand it was argued that the recommendations of the original report were still on the table and might lose their force if amended by a subsequent report. On the other hand was the view that a second report might provide a new impetus to the new Labour administration to give further serious consideration to the issues and recommendations which still awaited a detailed response from Government. Most interviewees saw the report as giving a new push to the initial report rather than as a fresh or free-standing enterprise. The 1998 Transport White Paper includes a two page summary of the report's main conclusions.

Twenty-first Report, October 1998, Setting Environmental Standards
3.15 This document elicited some strongly polarised views among consultees. Industry and environmental organisations we spoke to saw it as leading the way in articulating a conceptual framework for environmental standards and providing tools for policy development which would take account of public values. A Commission member predicted it would provide a framework for government to address complex and contentious issues such as those raised by current concerns about the long term effects of GMOs. One trade association described it as prescient and timely and highly relevant to industry's concerns about social acceptance of its activities. But others, such as those interviewed from the enforcement agencies expressed disappointment with its philosophical approach, when they had expected something with more direct practical application. One interviewee advised that his organisation had expected some "hard science" on particular environmental standards which the report failed to deliver. There was also criticism from within the Commission itself with some past and existing members expressing disappointment with specific aspects. There was acknowledgement that there had been difficulties in developing a common language across disciplines where measures and concepts are not always compatible. However, this was seen as one of the challenges of the multi-disciplinary approach which gave long term added value to the Commission's work. Overall, interviewees felt it was too soon to evaluate the report's impact.

Other Commission Statements
3.16 In addition to its main reports the Commission produces on a selective basis responses to consultation documents or ad hoc statements. These average three per year - recent examples include responses to DETR's consultation on Sustainable Production and use of Chemicals, and a response to an OST (Office of Science and Technology) consultation on the Review of the Framework for Oversight of Developments in Biotechnology. The Commission also submits memoranda to Parliamentary Committees, usually two or three per year. Over the last year one has gone to the Commons Select Committee on Science and Technology, and two to the Lords Committee on Science and Technology.

3.17 The Commission sees such statements as key contributors to disseminating the messages contained in Commission reports - focusing clearly on those responsible for policy development at the critical moment when changes in policy direction are being actively considered. Their publication also serves the secondary purpose of helping to keep the Commission in the public eye, to which the Commission attaches importance given the infrequency of publication of main studies (generally one a year).

3.18 Demand for Commission input to these exercises far exceeds its capacity to respond. The Commission limits its involvement to topics on which it has developed expertise through previous studies, or more general issues where it feels it has a particular interest or expertise. But we were told that resource limitations prevented the Commission responding to some consultations where it would otherwise wish to do so. Examples quoted were the review of the National Air Quality Strategy and the review of Company Law.

3.19 Because the Commission applies its normal standards of intellectual rigour to responses, preparation inevitably involves significant resource inputs both from the secretariat and from members, and diverts effort away from main studies. Whether the work can be justified by the end result is not clear as the Commission has no internal systems for evaluation. We therefore sought feedback from recipients about the impact of the two responses listed in 3.16 above.

3.20 On both of these, the policy divisions concerned received over one hundred responses. In neither case did the divisions consider that changes to policy had been made solely on account of points made by the Commission, although the Commission's comments had added weight to arguments made in responses. Policy divisions felt the Commission would have most impact on issues where it had strong grounds for opposing the proposed policy direction, but this had not arisen in either of these cases. There was some criticism that impact of contributions was reduced if they arrived late.

3.21 We look further in Chapter 7 on corporate planning and Chapter 9 on Working Methods at the need to set out clear objectives for these activities and to evaluate their impacts.

International work
3.22 The Commission has developed contacts with counterparts in Europe through meetings of the European Environmental Advisory Councils. Particular areas for co-operation have included transport (on which the Secretary to the Commission has chaired a workshop) and pursuit of environmental objectives within liberalised energy markets (on which a joint meeting has been held with the Commission's German counterpart, and a wider working group is planned).

3.23 We were told that the demand for input to international meetings and workshops by Commission Members and Secretariat exceeds the ability to deliver, and is likely to increase particularly on the European front. This demand has potential to divert members and the Secretariat from work on main reports. On the other hand, collaboration with European counterparts is seen by the Commission as being valuable in increasing its potential impact, given the increasingly European and international context of environmental policy making - and in response to the charge by some interviewees that the Commission has in the past been too UK-centric (see para 3.6 above).

3.24 We consider that the objectives and intended impacts of participation in such work should be clearly set out, and achievements evaluated, in the context of the Commission's corporate planning process. This is discussed in Chapter 7.

Research
3.25 We noted at para 2.19 that the Commission's role in relation to its remit 'to advise on the adequacy of research' is currently unclear.

3.26 We asked interviewees about this aspect of the Commission's remit. Several interviewees believed that it had been allowed to wither to the point where it could not be considered a significant part of the Commission's work. They felt the Commission was well placed to draw attention to gaps in knowledge on issues covered in their main studies, both current and those likely to arise in the longer term, and that it should give higher priority to this function. Commission members we spoke to had no feel for how influential the research related comments in their reports had been, nor for the extent to which they had impacted on the formulation of research programmes.

3.27 We tried to get some impression of impact through discussions with DTI staff involved in the Foresight programme7. Those responsible for panels dealing with transport, the natural environment, and the food chain and industrial crops had heard of the Commission but none could say that the Commission's output influenced Foresight's discussions. We also spoke to staff at a Research Council, but again the Commission's influence was not thought to be tangible.

3.28 In Chapter 9 we look at possible options for clarifying and developing the Commission's role on research.

Disseminating the Commission's work
3.29 We reported in para 2.15 above the Commission's commitments to achieving maximum impact for its reports and to promoting wider awareness of the Commission and its role. This section reports on achievement of these commitments.

3.30 We note that in the policy statements quoted at para 2.15, the Commission does not define what it means by a wider public, nor does it differentiate the intended impacts on different sections of its audience. We think this matters, because unless the Commission has a clear vision of the various constituents of its target audience it will not be able to plan effectively the style and content of reports and statements, and the strategy for dissemination, so that its impacts can be maximised.

3.31 We therefore welcome the dissemination strategy which the Commission drew up in 1998 with the aim of maximising the impact of the Standards Report. It represented a significant advance on previous practice in that it entailed consideration of the range of different types of activity which could help disseminate messages. These activities included discussions with individuals, speaking at conferences, formal presentations to Parliamentary Committees, seminars, bilateral meetings with selected organisations, and contributions to books and journals and to government consultations on related topics.

3.32 However there it did not contain a systematic analysis of the range of target audiences, the priority to be given to reaching each, the range of means of doing so, and the nature of the impacts intended, with a planned timetable of activities. In the absence of such an analysis, and in order to try and assess the effectiveness of the Commission's public presentation strategies over the past seven years, we have looked at sales of published reports, as one indicator, and have asked interviewees about their perceptions in terms both of awareness of the Commission and of the dissemination strategies for reports.

Published Reports: Sales
3.33 One indicator of the Commission's effectiveness in public presentation is the sales of its published reports, though we accept that sales figures for reports cannot be used as an accurate measure of the extent of dissemination of the Commission's messages, nor a measure of its influence. The broad picture is a progressive decline in sales from the first report (31,000 copies) and the second report (14,000 copies) to a plateau of about 2000 copies from 1989 onwards, of which 400 -500 are purchased by the Commission itself for free distribution. Exceptions have been the sixth report on nuclear power (12,000 copies) and the eighteenth report on transport (nearly 6000 copies, plus 1375 of an edition published by OUP); and on the other side the fourteenth report on GENHAZ (less than 1000).

3.34 Interviewees considered there were several reasons for limited sales of reports:-

Activities directed to awareness of the Commission and its role
3.35 The most significant recent initiative on raising awareness of the Commission has been the creation of a Royal Commission web site following the Internal Review of Working Methods last year. The site includes information about the Commission and its members, and about past reports, current studies and future plans, as well as press releases and minutes of meetings. This has improved the accessibility of information about the Commission and widened the potential audience for its work. Visits to the site are currently averaging 9 per day.

3.36 Nevertheless, we found in the course of the review that a surprising number of people, even within Government Departments, and also within the academic community and industry, were only vaguely aware of the Commission or had only sketchy knowledge of what it did, even though the nature of their jobs might have suggested otherwise. Although our contacts did not constitute a representative survey, we gained the impression that knowledge of the Commission and its work did not permeate government or the policy, academic and industry communities, as comprehensively as it might. We cannot say how far the general public has in practice become better informed about the Commission.

3.37 As the Commission pointed out, one difficulty in the context of raising awareness is the relative infrequency of Commission publications. We return in Chapter 9 to the issue of what might be done at least within defined key audiences, to avoid the risk of the Commission's influence being lessened through lack of awareness.

Activities directed to disseminating messages in reports
3.38 In addition to the formal activities described in 3.31 the Commission uses a variety of other techniques to put its message across. For example, Commission reports are normally launched at a press conference organised by the Secretariat. The degree of media interest is variable, depending largely on the extent to which the issues addressed are the subject of live political controversy. The desire to secure publicity while at the same time standing back from the political arena involves a tension which is not easy to manage, particularly given the media liking for setting up adversarial debates.

3.39 Much information dissemination also takes place as a result of the 'permeability' of members. As respected people at the top of their professions, usually with connections to a variety of influential associations, members have a key role in filtering information and ideas in a two way flow to raise levels of awareness about the issues tackled by the Commission and their conclusions and recommendations.

3.40 The formal and informal communication flow between the Commission (and its secretariat) and Government Departments was seen by interviewees as important and influential and was valued by members and officials alike - although some officials we spoke to felt the relationship would bear more frequent contact, without the danger of the Commission being captured.

3.41 While we welcome the attention given by the Commission to its dissemination strategy on the Standards Report, the concerns we expressed at para 3.36 about low levels of awareness even within government and the policy community apply also to awareness of the messages in Commission reports. Several interviewees said that part of the reason for this might be the lengthy nature of reports and their cost, as well as the absence of separate free-standing summaries.

3.42 We return in Chapter 9 on working methods to the issue of how the dissemination of Commission messages might be improved, and the impact on wider audiences increased.

3.43 Conclusions: the majority of consultees including individuals from government, enforcement agencies, industry, environmental groups and the National Academies regard the Commission's reports as scientifically authoritative, providing a valuable long term contribution to policy development. This is in spite of variable views on individual reports produced since 1992.

3.44 There is a need for the Commission to develop a process for evaluating the outcome of reports, linking back to its original definition of objectives and intended impacts. We recognise that there is no simple mechanistic way of evaluating reports, that the process may need to be different for each publication and that the full impact of a report may not be felt for a number of years. A more systematic approach to defining objectives and intended impacts, and evaluating outcomes is also needed for the Commission's other activities. We look further at this issue in Chapter 7 on corporate planning.

3.45 Although the Commission is well placed to comment in its main reports on the adequacy of research on issues raised within those reports, the extent of its influence in practice on research programmes is unclear, but was thought by many to be limited.

3.46 While the creation of a web site, and the implementation of a dissemination strategy for the Standards Report, are welcome developments, the levels of awareness of the Commission's work and the messages in its recent reports are surprisingly low, even within government. There is a need for clearer definition and prioritisation of target audiences within and outside government, to maximise the effectiveness of public presentation. We return to this in Chapter 9 on working methods.

7 Foresight aims to promote wealth creation and better quality of life by looking at future needs, opportunities and threats, and how developments in science could help the UK meet these challenges.

[ Previous ] [ Contents ] [ Next ]

Published 19 April 2000 / Updated 11 May 2000
Royal Commission on Environmental Pollution Index
Environmental Protection Index
Defra Home Page