Science Quality - Peer review of completed Defra-funded research
Background
Defra's Chief Scientific Adviser is required to provide an independent challenge to the science commissioned by Defra, to assure himself that policy is informed by a high-quality, robust evidence base and to raise the perceived standard of Defra-funded science in the eyes of the external communities with which we interact.
Whilst peer review is used widely in Defra as a method of assessing the quality of research proposals, its use to determine the quality of the final reports received by the department has previously been sporadic.
Defra has established a unit to assist in assuring that Defra policy is informed by high quality science. The Science Quality and Priorities Team (SQP), which is distinct from existing science units within Defra and reports to the Chief Scientific Adviser, is charged with carrying out this work and co-ordinates the review by external experts (peer reviewers) of completed science programmes.
Programme selection
Projects from four or more science programmes will be reviewed per year, including at least one from each of the major science funding policy areas (Sustainable Farming, Food and Fisheries; Animal Health and Welfare; Natural Resources and Rural Affairs; Environment). Both R&D and non-R&D programmes will be considered by this process.
Programmes are selected on the following criteria:
- Public/political profile
- Value of programme
- Time since or to the last review carried out by independent experts
- Current level of independent challenge (e.g. peer review of proposals)
Click here to see programmes reviewed and programmes for forthcoming review
Project selection
The choice of projects for peer review will also be determined by value and impact/profile. Therefore it may not be appropriate to peer review all projects in a given programme. It is intended that for each programme between 15 and 20 projects will be reviewed.
Because it is not the intention of Defra to look back at historical projects, all the projects selected for peer review will have reported no more than 18 months prior to the start of the review.
It is also not the intention of Defra to repeat the work already carried out by others. Therefore, if a project can be demonstrated to have undergone thorough independent peer review - e.g. through publication in high quality peer reviewed literature, this would be taken into consideration when selecting individual projects.
Peer reviewer selection
Peer reviewers will be well qualified in the areas that they are reviewing
and have current or recent active research experience. Potential peer
reviewers will be asked to declare any vested interests. An assessment
will be made regarding the suitability of the peer reviewer in light of
this declaration.
Peer reviewers will be paid £130 per report reviewed.
The peer review process
Final reports (SID5 forms as published on the Defra web site) will be
sent to the peer reviewers. At least two peer reviewers will be asked
to comment on each project. In many cases an additional peer reviewer
- a statistician will be used. The peer reviewers will be asked to comment
and score reports according to a set of specified, published, criteria
and enter their comments onto the following form:
Final Report Appraisal Form:
(R&D projects) - QALT3
(non R&D projects) - QALT4
SEE ALSO: Notes for Guidance - QALT
3A (relevant to both Qalt3 and Qalt4 forms)
The peer reviewers' comments on individual projects will be sent to a co-ordinating peer reviewer (who themselves may be peer reviewing projects). The co-ordinating peer reviewer will produce a short document summarising the findings of the peer reviewers for each project and an overall summary highlighting any trends across the programme.
The co-ordinating peer reviewer will be paid £500 per day.
The summary report will be sent to all research contractors involved in the review and the reports on individual projects sent only to the institute that conducted the research, along with the peer reviewers original comments.
The research contractors will then have an opportunity to respond to the reports on both the individual projects and the summary. These comments will be returned (via Defra) to the co-ordinating peer reviewer who will liase with the other peer reviewers and will consider modifying the reports and the summary in light of these comments. The revised reports will be sent to Defra.
Concluding Report and Recommendations
The report returned to Defra will consist of four parts:
- A programme level summary document
- A summary report for each project reviewed
- The appraisal forms for each project
- The comments made by contractors to the initial peer reviewers' comments.
Publication of reports and confidentiality
Defra intends to actively publish the summary of the programme review as prepared by the co-ordinating peer reviewer (see above for completed reviews). The individual reports on each project and the appraisal forms will also be available on request to Defra and will be circulated to the researchers responsible for the work, the funding policy division within Defra and Defra's Chief Scientific Adviser.
In line with Defra's policy on openness and transparency, the peer reviewer's name will normally appear on the appraisal as a matter of course. If a box at the top of the form is ticked the reviewer's name will not normally be disclosed to the researchers. However, peer reviewers will need to be aware that we may still be required to disclose their name in response to a request, including under the Environmental Information Regulations, the Code of Practice on Access to Government Information and the Freedom of Information Act 2000. Defra would, as far as practicable, seek to consult the peer reviewer if such a request were received.
In the majority of cases the final reports that will be peer reviewed will be already published on the Defra web site. Therefore, confidentiality issues relating to the results and patents/IP will have already be resolved. In cases where research has not been made publicly available, peer reviewers will be required to sign confidentiality agreements.
Conflict Resolution
It is possible that peer reviewers and research contractors will disagree over the final comments regarding individual projects and the programme summary. If the contractors wish to, they may make a further set of comments available to Defra in response to the reviewers' reports. In such cases, these comments will not be passed to the peer reviewers (their job will have been done at this stage), but will be circulated to the policy division responsible for the science. Comments on the programme summary may be published along side the reviewer's summary on the Defra web site if considered appropriate.
Where there is disagreement between the peer reviewers, it will be the role of the co-ordinating peer reviewer to find a consensus. Where appropriate, with liaison with Defra, project reports may be sent to one or more further peer reviewers.
Follow-up
When the review process has completed four reviews, peer reviewers, policy divisions, contractors and others in science directorate will be consulted on their opinions of the peer review process. The above processes will then be revised or updated accordingly. In addition, Defra will assess how the peer reviewers' recommendations have been followed-up and implemented into current commissioning of relevant research.
Page
last modified: 21 November 2005
Page published: 21 November 2005