Independent Audit of Badger Trial: Audit of the Surveying and Social Group Delineation SOPs
Executive Summary
This document sets out the results of an independent evaluation of field operations associated with the MAFF badger culling trial. This element of the audit was intended to assess, on a sample basis, the effectiveness of surveying and social group territory delineation carried out in accordance with the prescribed Standard Operating Procedures (SOPs). The audit was based on careful and comprehensive re-surveys of a selected sub-sample of setts and inferred social group territories from a total of six Treatment Areas.
Care was taken to ensure that, as far as possible, this sample was fully representative of variables which could affect the accuracy and efficiency of surveys, assessments and social group territory delineation. To optimise survey efficiency and objectivity, the general approach was to issue surveyors with the original maps showing sett locations but without the detailed sett description, classification and assessment records, to avoid unconscious bias. To enable a more rigorous analysis, the re-survey aimed to classify every sett according to the full list of accepted criteria ('main', 'annexe', 'subsidiary' and 'outlying'), supported by detailed surveys of sett size and activity.
The intention was not simply to repeat MAFF's original field surveys. The principle aims were to find all of the setts recorded on the original survey maps and any others that may have been missed or which may have appeared between surveys, and to assess critically the status of each sett. In many cases bait-marking was used to confirm 'main' sett classification, particularly when a sett appeared possibly to have been mis-classified during MAFF's original surveys.
A total of in excess of 300 setts were involved in the re-surveys, of which 58 were 'main' setts. A sub-sample of these setts were investigated further to assess the effectiveness of delineating social group territories. This involved a total of five clusters of putative badger social groups from four different Treatment Areas. The clusters were chosen in order to include areas with a range of BRO histories, and to include territories which appeared to be relatively stable, with well-defined boundaries, as well as those where badgers' likely social organisation was less clear-cut.
All of the re-survey data were then compared to MAFF's original results. The degree of agreement/disagreement between the surveys was assessed, and every effort was made to investigate the reasons for any discrepancies.
The sett records were classified into three agreement categories, identifying those for which there was general agreement, those where moderate discrepancies were identified, and those where substantial discrepancies were highlighted. The imposition of these different categories inevitably, and unavoidably, masked the complexities in the discrepancies between the surveys. For example, in one instance a sett appeared to have been mapped inaccurately but classified 'correctly' as a main sett by the field surveyors, however this was then 'incorrectly' down-graded on review. Clearly assigning this record to one of three categories masks the detail, but such an approach was necessary to discern patterns of discrepancies in the data. Attempting to amalgamate and classify the different reasons for these discrepancies exacerbated this effect. Nevertheless it was possible to identify what appeared to be the most important reasons for differences between the surveys. There was general agreement over the majority of the sett records (a little over 60%). Approximately 15% were identified as showing moderate discrepancies between the original results and the re-survey, and in just under 25% substantial discrepancies were highlighted.
The majority of substantial discrepancies (approximately 75%) related to outlying setts identified during the re-surveys but which did not appear in MAFF's original results. Few larger setts appeared to have been missed. However, this included four main setts (one of these had subsequently been identified during a repeat survey). In all, eight larger setts appeared to have been missed. Some main setts were also mis-classified (approximately 20% of main setts had not been classified as such).
The substantial discrepancies relating to smaller setts could have been as a result of finding but ignoring a small disused sett, the sett having appeared in the short interval between the two surveys or the sett simply having been missed. It appeared that in the case of the larger setts, these simply had been missed. Significantly more setts appeared to have been missed in difficult habitats (eg. woodland and scrub), and this appeared to be an important contributing factor.
The majority of moderate discrepancies related to main setts and comprised a combination of mapping discrepancies, substantial differences in size and activity, and apparent misclassifications which were not confirmed by bait-marking. In addition to these two categories, a small number of rabbit burrows and fox earths appeared to have been mis-classified as badger setts.
The apparent mis-classification of setts was, in part, attributable to surveyor error but also to drawbacks in the operating procedures: particularly the restriction of survey activities to discrete land-ownership 'parcels'; the need for surveyors initially to ascribe setts as 'main' setts or 'others' on the basis of their survey information alone; the use of this 'main' sett or 'other' classification rather than a more comprehensive (and hence informative) system; and the lack of 'feed-back' as the survey data were compiled and the tessellations drawn.
No significant differences could be found between the performance of the different surveyors/pairs of surveyors, with regard to any of the perceived errors. In addition, very few administrative errors were revealed by the Audit.
The bait-marking investigations were largely successful: although several of the bait-marked territories were characterised by the presence of relatively few large 'typical' boundary latrines and other features characteristic of stable, well-defended territories, nevertheless, reasonably distinct territories could be identified, with at least some boundaries being regularly marked and relatively stable. For some territories it was necessary to repeat bait-marking during the subsequent season in order to help resolve issues where the badgers' social organisation remained unclear. Significant discrepancies were found between the tessellated boundaries and those derived from the bait-marking investigations. The vast majority of the discrepancies appeared to be a direct result of the differences in main sett classification between MAFF's original results and the re-survey. Where the initial classification of main setts was 'correct', there was good agreement between the tessellated and bait-marking derived boundaries and 100% of setts were assigned to the 'correct' territory. Notwithstanding the clear differences in boundaries caused by 'incorrect' sett classification, more than half of the setts involved were still assigned to the 'correct' territory and the use of different reliability categories during the process did tend to highlight correctly those setts for which assignment to a given territory may be unreliable.
Overall, the Audit did not reveal fundamental flaws in the SOPs or their application. Clearly, it is important that as many as possible of the setts (and crucially all of the larger setts) are found and classified correctly, in order that all of the other elements of the Trial which depend on these data can be pursued in an effective and reliable manner, and in this regard opportunities for improvement were identified. In addition, correctly classifying main setts is also a key factor, since the tessellation procedure and ascribing sett 'ownership' is fundamentally dependent upon this, as will be many related elements of the Trial relating to badger density etc., and once again the Audit highlighted improvements which could be made. It is important to note that the Audit period covered the early stages of the trial when staff would be least familiar with the processes and techniques and the SOPs were not finalised. It is understood that refinements have already been made to many areas covered by the Audit.
Finally, as part of the Audit several recommendations were made:
- surveying each Treatment Area should be an iterative process;
- survey larger parcels of land as integral units and avoid piecemeal surveys delimited by farm boundaries;
- adopt a more comprehensive (and generally accepted) approach to sett classification;
- adopt a staged, precautionary approach with regard to classifying larger setts;
- continuously review the survey data as they are collected;
- use tessellation as a helpful tool, early in the process, and critically review the size and shapes of putative territories;
- encourage a team approach to surveying difficult habitat features;
- consider the restricted use of bait-marking techniques to resolve doubts of the classification of potentially important setts;
- encourage the checking of the final data/maps by the field staff responsible for them;
- with regard to the adjustment of tessallated boundaries, critically review the identification of topographic features which are thought to limit territories;
- provide additional training, particularly relating to sett classification in different habitat types and population densities, and the use of tessellation as part of an iterative approach to surveying the Treatment Areas;
- incorporate further auditing as appropriate.
Page last modified:
12 August 2003
Page published: 5 February 2003