Department for Environment, Food & Rural Affairs

Financial Management and Policy Review of the
Royal Commission on Environmental Pollution

9 Working Methods

9.1 This chapter looks at the working methods adopted by the Commission in the conduct of its various activities, but with emphasis on main studies. Its aim is to draw together comments we received from interviewees and written contributors on the way in which the Commission works, and to draw broad conclusions about whether current practice is appropriate. In making suggestions for change, we acknowledge where the Commission has already introduced improvements following implementation of the "Guidelines for Conduct of Studies", including those where, because of time lags, the effects of the Guidelines may not have been fully apparent to interviewees. We also identify where changes to working methods will flow logically from introduction of a strengthened corporate planning system.

9.2 In writing this chapter we had in mind that the philosophy of strategic sponsorship requires NDPBs to determine for themselves how best to conduct their day to day activities. Once objectives are set, it is for each individual body to determine the detail of how these are met. The suggestions put forward here are for the Commission to consider, and to adopt or adapt as it sees fit.

9.3 For clarity, we look first at the way in which main studies are carried out and consider each of the significant stages of a study; we then consider how other Commission activities are conducted, and look at evaluation mechanisms. Finally, we address openness and transparency which emerged as an important cross cutting theme.

Background
9.4 Since the late 1980s, the Commission has experimented with a number of different ways of managing its work in order to improve quality and timeliness. One significant change involved the introduction of short studies to allow coverage of a wider range of subjects and a more rapid response to topical issues. Experience showed there were a number of drawbacks to this approach9, and the Commission decided the length of subsequent studies should not be predetermined but should reflect the complexity of the issues involved. Changes to programming were introduced to improve the regularity of output, and an internal target of publishing one report a year was established. To achieve this studies had to overlap, so that the preparatory phase of the following study was started well before the previous one was complete.

9.5 The drivers for these changes initially came from government10 but over the last two years the Commission has been pro-active in seeking to rationalise its working procedures. We referred in Chapter 2 to the fundamental review of working methods carried out by a working group of Members and Secretariat in 1997-8, and the resulting "Guidelines for the conduct of Commission studies" (Annex 10). In addition to strengthening of planning and reporting procedures, the Guidelines recommended new methods of working to be adopted for subsequent studies. These are comprehensive, and cover selection of topics for study, gathering of information, processing of information, deliberation and drafting, and presentation of the Commission and its work. The Energy study (which commenced in 1998 and is still underway) provided the first opportunity to apply these new procedures.

9.6 We consider that when implemented in full, the Guidelines will represent a significant step forward in updating the Commission's working methods to ensure they incorporate current good practice in the public sector. The benefits will include improvements in clarity and openness (by making transparent the processes involved in the preparation of reports, making planning documents and minutes of meetings available through a Web site, widening consultation on topics for future study) and enhanced effectiveness (through setting objectives for evidence gathering, increasing accessibility for reports through use of summaries and prioritising recommendations, and introducing a dissemination strategy for each report). We were told that a review of the working and effectiveness of the Guidelines is planned for the end of the Energy study in Spring 2000. In conducting the review, the Commission may find it helpful to be aware of the perceptions and experience of those we interviewed.

9.7 We welcome the proactive approach taken by the Commission in reviewing and updating its working arrangements, and recommend that priority should be given to ensuring that the Guidelines for the Conduct of Commission Studies are implemented in full. We also endorse the Commission's proposal to review the working of the Guidelines at the end of the Energy study.

Main Studies

Selection of topics for study
9.8 All interviewees agreed that the Commission should retain responsibility for selecting topics for study, and should determine the breadth of work and timescales. Those we spoke to felt other options - such as giving responsibility for selection to government - would be seen as undermining the core purposes (see paragraph 2.7) of bringing in to political focus environmental issues that might otherwise be ignored, or which might not be readily identifiable from within government. Ministers' right to ask the Commission to look at particular topics was acknowledged, but the fact this had been used sparingly to date was welcomed as underlining the Commission's independence.

9.9 There was widespread welcome for the consultation round introduced through the Guidelines. The Commission now consults annually with government Departments, regulatory agencies, non-governmental organisations and European bodies on potential subjects of study, and a shortlist of topics is available for public comment in the Commission's Forward Look (available on the Website). Consultation was seen as improving transparency and engendering confidence in the value of topics selected, although there were inevitable calls for the Commission to give greater weight to the views expressed by some consultees.

9.10 We noted in 3.6 (iv) some criticism of the Commission's choice of topics for study and felt this arose in some cases from a lack of understanding about the selection criteria (e.g. that the Commission should not consider topics where significant work was already going on elsewhere) and in others from lack of information about why one short listed topic was preferred to others. The first has been addressed through publication of the criteria (now incorporated in the Guidelines and available on the Web), and the second will be dealt with by inclusion in the Forward Look of a rationale for the Commission's choice of new topics. We suggest the rationale should also include an explanation of the added value the Commission can bring to the issue.

9.11 We endorse the Commission's proposal for explaining the rationale for topic selection in its corporate planning documentation, and recommend that this is applied to future studies.

Scoping of studies and initial preparation
9.12 This phase involves establishing the extent of current knowledge on a subject, identifying different perspectives and points of view, and reaching preliminary decisions on the issues a report should cover. The importance of competent scoping has been emphasised by the Commission in recent years, but there was criticism from a variety of sources about poor scoping of the Standards study, and a feeling that frequent changes of direction had led to uncertainty in the outside world about the focus of the final report. Similar complaints were also made about the Energy study, which was described by one interviewee as needing an earlier axe. There was no desire to see the Commission straightjacketed by its original scoping but a conviction that to produce timely reports it was important to make hard decisions on scope and stick to them. Making other organisations aware of decisions on focus was also seen as important in managing expectations. The role of the Chairman in ensuring members stuck to the agreed brief and timetable was considered to be vital.

9.13 We support the Commission's continuing efforts to improve the scoping of studies to help clarify focus, plan timetables and manage external expectations. We welcome the commitment to publicise decisions on scope and timetables.

9.14 In light of our comments in Chapters 2 and 3 about audience, we believe the scoping phase also presents an opportunity for the Commission to consider how to maximise the impact of reports by identifying as far as possible groups likely to have an interest in particular studies, and to explore options for report formats that will maximise accessibility for these groups.
Potential audiences may include:

9.15 We note that the Commission Guidelines also suggest the audiences for reports should be identified in the scoping phase, and welcome the introduction of this arrangement. The Commission told us that this was done in the early stages of the Energy report. We recognise that groups identified as relevant at the outset of a study may well change or be added to as work progresses, and that the Commission's strategy for maximising accessibility may need occasional adaptation to reflect this.

9.16 We support extending the scoping phase of studies to include consideration of the intended impact and hence potential audience for each report, and to explore options for report formats that will maximise accessibility for these groups. This also flows from the recommendations made in Chapter 7 about a strengthened corporate planning process.

Evidence
9.17 The second phase, information gathering, involves invitations to submit evidence, and commissioning research by consultants on particular aspects of a study. Requests for written evidence go to a wide variety of relevant organisations, and a news release and adverts in newspapers are used to reach a wider audience with an interest in the subject. Oral presentations are sought from those with a particular contribution to make on specific issues.

9.18 There was mixed experience among interviewees about both written and oral evidence. On the former, there were complaints from a minority of organisations about the volume of work generated by producing evidence for the Commission, but others felt the discipline of the exercise was invaluable in refining their own thinking on particular issues.

9.19 We noted in one meeting we attended that Commissioners were critical of consistently poor written evidence from some organisations. We were told that the Commission had devoted considerable effort over the last couple of years to improving the quality of contributions by setting out clearly what it wants when inviting evidence, and briefing groups of organisations (such as through the Society for the Chemical Industry) about the sorts of evidence that will be most useful. We commend these proactive approaches and suggest they should continue and be further developed. We consider there would also be value in adopting retrospective techniques, such as giving feedback to major contributors on their submissions, and providing more references in reports to the evidence received.

9.20 On feedback, several interviewees were critical of the absence of comment from the Commission on the value or otherwise of their written contributions. Their experience was that it was demoralising for staff to expend time and effort on preparing evidence only to have no idea whether it was helpful or not, or why particular ideas were accepted or rejected. The absence of comment also made it impossible for them to assess the value of their work, and so to justify further work (and expenditure of resources) for subsequent studies. One organisation said its contributions "disappeared into a black hole" which added to perceptions that the Commission remained opaque. We were sympathetic to this concern, and agree that in not giving feedback the Commission may be missing a further opportunity to improve the quality of what is sent to them.

9.21 In recommending that feedback should be given, we are aware that commenting on all contributions would place unreasonable burdens on the Secretariat. We therefore suggest that feedback should be confined to those organisations who have made a substantial contribution to a study, and who are likely to contribute to future studies i.e. those with whom the Commission has (or wishes to develop) a long term relationship. The type and timing of feedback would be for the Commission to judge. In some cases this might be appropriate after evidence has been considered by Commissioners; in others it might be best at the conclusion of a study and combined with dissemination activities.

9.22. At the Commission meeting we attended, we were conscious of the volume of evidence that Commissioners had to consider, and the vast amounts of paper this consumed. We look further in Chapter 11 at how electronic mail might be used to reduce copying costs and conserve resources.

9.23 We recommend that the Commission should continue to seek means, both proactive and retrospective, to improve the usefulness of written evidence. We suggest this should include a mechanism for giving feedback to organisations who have submitted substantial written evidence to studies, and are likely to do so in future.

9.24 Formal oral evidence sessions also had a mixed reception. Some of those giving evidence commented that the unstructured nature of these sessions made them less productive than if questions from members were more clearly set out in advance, providing an opportunity to think about the issues beforehand (whist not constraining Commissioners from exploring other aspects on the day). By contrast, members generally felt that the flexibility of structure gave an opportunity to develop new ideas and themes, and that the "brainstorming" element would be reduced by too much advance prescription. It may be that different types of oral evidence are appropriate for different stages of studies. More formal prepared presentations might be most useful early on when the focus is on information gathering, whereas wider ranging discussion sessions might be more appropriate at the stage of refining Commission thinking.

9.25 The Guidelines refer to the need for oral evidence sessions to be carefully planned and targeted with clear objectives and expectations. We understand these new procedures were applied to evidence taken for the Energy study. In reviewing this aspect of the Guidelines, the Commission may wish to ask members and recent contributors whether they felt sessions for the latest study were well planned and effective, or whether there is still more that could be done to further improve the process.

9.26 We support the Commission's desire to be clearer about the purpose of each oral evidence session and the broad themes to be covered, and to communicate this to presenters so that they can prepare themselves adequately in advance. We suggest that the effectiveness of new procedures for doing this should form part of the Guidelines review.

Dealing with differences of views
9.27 Our interviews gave rise, perhaps inevitably, to criticism of the Commission's handling of the views of some experts. Comments were made about failure to give sufficient weight to particular viewpoints (Soil and Standards reports) and failure to seek out a sufficiently wide range of expert views to inform recommendations (targets in the transport report). These concerns may reflect the fact that views or theories advanced by particular contributors were not accepted, or that particular individuals were not called upon to contribute. We acknowledge that it is not possible for the Commission to satisfy all of the experts all of the time, but believe the comments do raise some serious questions about how best to handle expert views, and in particular conflicts between views. This applies to both evidence from outside experts and conflicts of views between members of the Commission. Relevant questions include:

9.28 The Guidelines set out the Commission's approach to the first question. Strategies for ensuring a sufficiently wide call for evidence include maintaining a database of previous consultees who can be written to individually for subsequent studies, publishing a news release inviting evidence, and placing advertisements about new studies in various UK publications. All of these seem sensible and we propose they should continue. In addition, we suggest that for some studies, placing advertisements in European and other international journals may also be appropriate; and organisations such as NERC suggested they may be able to help with developing consultee lists for specific studies to ensure coverage is sufficiently international.

9.29 On the second question, we noted in Chapter 3 criticism from some interviewees that the Commission had not always been successful in reporting tensions between different perspectives, and the Commission's counter argument that in recent studies (Transport and Standards) it has done exactly that. We endorse the Commission's approach of attempting to take account of the full range of perspectives in drawing up reports and referring to these explicitly in the text - which is in line with the philosophy in the Standards report. We see this as applying not only to scientific and technological questions but to those covering economic, social and moral issues as well.

9.30 The Commission currently exercises quality control over how it deals with expert views through mechanisms such as refereeing or peer review. This involves experts in a particular discipline (mainly academics, although some will be professionals working say within government) being asked to review either specific chapters or groups of chapters of a report or the whole report. Other control mechanisms include internal checks by authors of chapters and members of the Secretariat, and external checks by relevant government departments. These arrangements for ensuring the quality of reports seem to us to be comprehensive, and we welcome their continuation.

Checking credibility of conclusions
9.31 The Guidelines recognise the need for a "sounding board" process, and propose the Commission should rehearse its arguments and findings before an expert audience of individuals and representatives of organisations. This has a number of benefits: it tests credibility of what is being proposed; it makes the Commission aware of possible counter arguments; and it improves openness by giving the invited audience an opportunity to contribute to reports at a stage where they can still influence the final outcome.

9.32 The categories of individuals or groups who might be invited to participate in this process are not specified in the Guidelines (presumably because they will vary from study to study) although the declared intention is for a range of viewpoints and perspectives to be presented and discussed. We see merit in opening up this part of the process to the representatives of the audience that the Commission hopes to influence. This may include representatives from industry, environmental groups, academics, central and local government and in some cases members of the public.

9.33 Commission studies will inevitably give rise to differences of views between experts, including between members of the Commission. A raft of strategies are needed to ensure these are fairly represented including procedures for identifying and consulting experts with a wide range of perspectives, recognition of the importance of articulating different perspectives and explaining the limits of expert knowledge in each report, quality control procedures to check accuracy of factual matters, and a sounding board to check the credibility of innovative concepts and ideas. These have been recognised in the Guidelines and are being put in place as the Energy study progresses.

Dissemination strategy, including format of reports
9.34 In Chapter 3 we reported that in spite of their exemplary academic quality, readability and valuable innovative thinking, levels of awareness of the Commission's work and of the messages in reports were surprisingly low even within government. We suggested that there was a need to define more clearly the target audiences for reports, the intended impacts on them and most appropriate format for reports and how these should be disseminated. Defining the audience and impacts has already been addressed in 9.14 above; we look further at format and dissemination below.

9.35 On format of reports, some interviewees suggested that the current practice of producing a single weighty tome was unimaginative, especially if the Commission really wanted to reach a wider audience. Possible alternatives suggested included:

9.36 We consider that giving wide circulation to copies of the summary chapter of reports, aiming at people at senior and middle management levels in targeted organisations, would help raise awareness of the key messages within defined target audiences at little cost to the Commission. This would be in line with the 1998 Guidelines. We understand that the Commission is considering the possibility of going further by publishing a separate summary of the Energy Report at the conclusion of the study. We suggest that the Commission should try to monitor as far as possible the impact of any initiatives taken on this issue, and the extent to which wider audiences are reached.

9.37 A further issue on the format of reports which was raised by some interviewees concerned the presentation of conclusions and recommendations. Some alleged the Commission made too many recommendations and failed to give recommendations a ranking or weighting. The effect was for more minor issues to cloud major messages which further reduced accessibility and the impact of reports. More cynical interviewees felt a large number of recommendations gave governments freedom to ignore major points of concern, whilst claiming to have implemented a majority of (albeit minor) recommendations. We note that the Commission's recent practice is rather different. The Soil Report did prioritise and group its recommendations, and the two subsequent reports (the second Transport Report and the Standards Report) have not made recommendations but merely reached conclusions; conclusions being more difficult to summarise and target. We endorse continuing attention to user-friendly presentation of conclusions and recommendations in the text of reports, in line with the 1998 Guidelines.

9.38 Many interviewees felt that the Commissions arrangements for disseminating the messages in reports could be improved, and in Chapter 3 we recognised that the Commission had already taken this on board in the Guidelines and had produced a dissemination strategy for the Standards Report.

9.39 We noted in Chapter 2 (para 2.17) that some interviewees suggested the Commission should work in partnership with other organisations to help improve dissemination of messages. Underlying this suggestion was a recognition of the potentially large costs involved in this work, but also the possibility that some organisations are better placed to communicate with a wider public than the Commission, and the importance of avoiding duplication of effort and confusion of purpose with other environmental advisory bodies. This would be in line with co-operative partnership arrangements advocated in the Modernising Government White Paper. We suggest that the Commission should think creatively about how to work in partnership with others to get its message across while minimising costs. Possibilities might include:

9.40 We welcome the attention now being given by the Commission to the presentation of the conclusions and recommendations of reports and to the production of summary chapters, in line with the new Guidelines, as well as the possibility of producing a separate summary of the Energy Report. We recommend that further attention be given to achieving wide distribution of summaries within target audiences. We welcome the production of a dissemination strategy for the Standards Report and suggest this should be developed for future studies to include not only identification of target audiences and means of reaching them, but a planned timetable of activities. We also recommend that the Commission should explore opportunities for partnerships with other bodies to help improve dissemination of messages to a wider public. Care is required to ensure that partnerships are selected in a way that does not undermine the impartiality and independence of the Commission.

Commission statements
9.41 In Chapter 3 we reviewed the effectiveness of two recent Commission responses to consultation exercises. We were concerned about the need to manage the tension between, on the one hand, seizing opportunities to disseminate key messages from past reports, and, on the other, being inappropriately distracted from work on the current main study. With increasing numbers of consultations and competing calls on resources, we suggest the Commission needs to identify consultation rounds to which it can give most added value. It may be helpful to develop criteria to assist with selection. These might include subjects where the Commission's multi disciplinary approach would bring particular benefit (and would otherwise be absent from policy makers considerations), or where the Commission either strongly support or disagree with what is being proposed.

9.42 On the format of statements, policy sections commented that it is helpful when:

9.43 As with other activities, we suggest that statements should be subject to an evaluation process.

9.44 We recommend the development of criteria to assist with identifying consultation rounds to which the Commission can bring most added value, and the extent to which the Commission's intervention will make a difference to the outcome. These might include subjects where the Commission's multi-disciplinary approach would bring particular benefit (and would otherwise be absent from policy makers considerations), or where the Commission either strongly support or disagree with what is being proposed.

International activities
9.45 We noted in other chapters the increasing demands on the Commission to contribute to a wide range of other work, such as discussions with European counterparts, and participation in workshops and working groups on topics that have been the subject of reports. We accept that the Commission must be able to consider the EU dimension of any study it undertakes, and must address the Europe wide implications of its proposals; and we agree that doing so effectively depends on maintaining contacts with European institutions. But, because this kind of liaison activity can easily proliferate, with potential to distract the Commission from production of main reports, we recommend that the Commission sets clear aims and objectives for this work and devises a means of evaluating when and whether these have been achieved. We recommend that the Commission should set objectives for work such as liaising with European counterparts and consider how to evaluate effectiveness.

Research
9.46 We reported in Chapter 2 that the Commission's role on research is currently unclear, and in Chapter 3 that it is also unclear to what extent the Commission's references in reports to the adequacy of research have had any influence. In our interviews some interviewees suggested to us that the Commission should seek to develop closer relationships with the Research Councils in order to be able to influence research decisions. We wonder if this is realistic. We think the kind of advice on research which the Commission is best placed to offer will flow from its work on main studies, and, insofar as those studies are primarily concerned with the process of policy development within Government, we think that such advice is likely to bear most directly on the research programmes of Government Departments, particularly DETR, and the enforcement agencies. We think that diversion of effort to target the Research Councils may be unlikely to have as much impact as would developing closer links with the relevant research programme managers within Government, where they are identified as target audience for a particular study. We see the role of advising on research as continuing to be subsidiary to the main role of producing reports on major policy issues.

9.47 We recommend that the Commission should consider further how to maximise the impact of the views on the adequacy of research which are included in its reports, including through developing closer links with the relevant research programme managers within Government Departments and the enforcement agencies.

Evaluation
9.48 In Chapters 3 and 7 we referred to absence of a process for evaluating Commission reports and other activities and highlighted the importance we attach to introducing such an arrangement. We also drew attention to the difficulties involved in this task, but welcomed the Commission's commitment to undertake it. There is no mechanistic formula which can be adopted for this work and the Commission will need to develop tailor-made solutions for each report and activity. Evaluation procedures need to be seen as an integral part of the new corporate planning process.

Evaluation of Commission Reports
9.49 Factors the Commission may wish to take into account when evaluating reports fall into two broad categories: real world outcomes and subjective perceptions. Examples of the former include:

Subjective assessments include

Evaluation of other Commission activities
9.50 We have also placed emphasis in previous Chapters on the importance of identifying audiences for each report and devising strategies to ensure key messages reach relevant groups. In due course, the Commission will also need to evaluate the effectiveness of its dissemination strategies, so that conclusions can be fed back into the process to inform future dissemination plans. We welcome the recent first shot at evaluating the dissemination strategy for the Standards report, although the low profile release of the Report means this is limited to a description of what was done rather than an assessment of its effectiveness.

9.51 Evaluation mechanisms will also be needed to consider the effectiveness of Commission statements and its role in advising on research, and to assess the benefits arising from its international work.

Openness and Transparency
9.52 Many of those we spoke to acknowledged that the Commission had changed significantly over the last five years becoming increasingly transparent and open in line with developments elsewhere in the public sector. The Guidelines on the Conduct of Studies were widely welcomed (though not everyone was yet aware of them), as were the Website with minutes of Commission meetings and material about individual studies (although these need to be kept up to date), and the range of meetings encouraging participation of a wider audience. Most interviewees want to see this trend continued and developed further, and a number of suggestions were put forward as to how this could be done. These are reproduced below and we recommend that the Commission considers to what extent they can be accommodated.

9.53 Making decisions on the scope of studies more widely known was raised by a number of interviewees. The importance of this has already been recognised by the Commission and information on scope of future projects will appear in the corporate planning documentation (see paragraph 9.13). It is also addressed in formal invitations to submit evidence and the accompanying news releases.

9.54 Several of those we spoke to suggested the Commission should produce interim statements to show emerging thinking. We note that this latter point was also raised during the Commission's internal Review where a number of respondents indicated they would welcome progress reports. Arguments in favour include:

Arguments against are:

9.55 We believe there would be advantages in the Commission providing updates on progress of reports, and support inclusion of these in corporate documentation (such as the Annual Report), or in other awareness raising documentation such as newsletters. To minimise the risk of progress reports being misused, we suggest they should be confined to coverage of topics/issues under consideration rather than developing arguments.

9.56 Opening up the Commission's deliberations to a wider audience was also raised in our discussions. It was generally accepted that the Commission's internal deliberations should be held in private. However, some of those we spoke to felt the Commission should follow the lead of other public bodies and open sessions such as oral evidence to the public. Those against the proposal felt it would make contributors reluctant to attend and unwilling to be innovative in responses, and would open up the possibility of meetings being hijacked by groups keen to exploit what was said for their own ends. Those in favour felt it would make contributors more conscious of what they were saying and so improve quality, and would give a chance for a wider audience to feed in views on what was said. A compromise position thought credible by many was to follow the example of Select Committees and presume that all oral evidence sessions would be open, but allow some sessions to be closed if there was sufficient reason.

9.57 We consider the approach used by the Select Committees in opening up oral evidence sessions to the public provides an appropriate model for the Commission and suggest that members consider this possibility for future studies.

9.58 Other activities aimed at opening up the Commission's processes include organisation of meetings at which the public are invited to contribute their views on the subject under study. To date the Commission has held four public meetings, one in England, two in Scotland and one in Wales.

9.59 We support the Commission's moves to greater openness, and suggest that experience in their operation be reported in the Annual Report to provide a basis for review and development.

9 Short studies were not thought to make best use of the Commission's broad approach to complex problems: apparently narrow subjects often proved more complex than they originally appeared (e.g. Incineration of Waste); it was difficult to produce persuasive and robust conclusions and recommendations if reports considered only one aspect of a topic; and there was a perception that short studies increased reliance on outside experts and the risk of capture by one perspective.
10 DOE had expressed concerns about low productivity and lack of a proper planning process (the latter in the 1991 Audit report), and Treasury had questioned the purpose of the Commission after considerable delays in producing the report on Freshwater Quality.

[ Previous ] [ Contents ] [ Next ]

Published 19 April 2000 / Updated 11 May 2000
Royal Commission on Environmental Pollution Index
Environmental Protection Index
Defra Home Page