Independent Audit of Badger Trial: Audit of the Surveying and Social Group Delineation SOPs
2.0 Assessing the effectiveness of the 'Surveying for badger activity' SOP.
2.1 Methodology
2.1.1 Field surveys
This assessment has been based on careful and comprehensive re-surveys of a selected sub-sample of setts and inferred social group territories from within the treatment areas. Surveys were undertaken in the Survey Only and Reactive Areas of triplet B, and in all three treatment areas of triplet A.
In the case of triplet B, two clusters of 4-6 putative badger social groups in each area were chosen for re-survey, including at least one large area from which no main setts were recorded. Care was taken to ensure that, as far as possible, this sample was fully representative of the range of:
- surveyors undertaking the work;
- BRO histories;
- apparent badger main sett densities; and
- difficulties (associated with different habitat types) with finding the setts.
For triplet A Treatment Areas, a single cluster of social groups was chosen. Although a different group of surveyors had obviously been involved in MAFF's surveys, and those surveys had been undertaken at a different time of year, triplet A clusters were chosen to supplement the range of other variables (e.g. BRO history) 'covered' by the triplet B sample. For operational reasons surveying was aborted in part of triplet A and because these data were incomplete and not subject to the same follow-up investigations, they have not been included in Section 2.2 (below). However, the cluster of approximately 10 social groups surveyed in triplet C has been substituted. The total number of main setts/putative badger social groups re-surveyed represents a more than adequate sample of the numbers of main setts found during MAFF's original surveys in order to detect infrequent errors in the survey procedures. It has, however, been necessary to re-survey a greater number of social groups in order to 'balance' the different variables described (in points a-d) above. In total, approximately 60 social groups have been involved in the re-survey; significantly more than was envisaged at the inception of the Audit.
To optimise survey efficiency and objectivity, the general approach has been to issue surveyors with the original sett records, but without the detailed sett description, classification and assessment records, to avoid unconscious bias. To enable a more rigorous analysis, the re-survey has aimed to classify every sett according to the full list of accepted criteria (see below), supported by detailed surveys of sett size and activity.
Although the intention was not simply to repeat MAFF's original field surveys, the re-survey nevertheless involved searching for the characteristic signs of badger activity including setts, latrines, paths, footprints, hairs, and feeding signs. Particular effort was directed at finding all of the setts recorded on the original survey maps and any others that may have been missed or which may have appeared between surveys. Any badger sett identified was classified according to the definitions given in Appendix I. The current level of activity of each sett entrance hole was also classified (definitions of these terms are also given in Appendix I).
In order to confirm 'main' sett classification and, in particular, when a sett appeared possibly to have been mis-classified during MAFF's original surveys, bait-marking investigations were undertaken. In some cases these were limited exercises which aimed simply to determine sett 'ownership', but in other cases the re-survey cluster fell within an area earmarked for a full bait-marking survey. Bait-marking methodologies are explained in more detail in Section 3.1.
2.1.2 Assessment methodologies
Once the re-survey data had been collected, mapped and collated, the results were compared to MAFF's original field survey results, both in the form of the definitive map for each treatment area, and the individual landowner-based survey maps and sett record sheets. In the case of each sett record, the degree of agreement/disagreement between the surveys was assessed, and every effort was made to investigate the reasons for any discrepancies. In order to aid analysis and interpretation, the sett records have been classified into 'agreement categories'.
These have been defined as follows:
- Category 1
- Setts or sett records for which substantial discrepancies exist between MAFF's initial surveys and the re-surveys. For example, where setts have not been classified correctly as 'main' setts, and where the audit survey classification has been confirmed by bait-marking; where there are substantial mapping errors; where there are large differences in sett size; or where setts have not been located by one or other survey.
- Category 2
- Setts or sett records for which moderate discrepancies exist. For example, where sett classification appears to be inaccurate but this has not been confirmed by bait-marking; where there are reasonably significant mapping errors; or where there are moderate differences in sett size that appear unlikely to be the result of natural variation during the interval between surveys.
- Category 3
- Setts or sett records for which there is general agreement between the surveys. Any mapping errors are insignificant, and any variations in size would be expected to be the result of natural variation between surveys.
NB. With regard to some Category 1 records, during the process of consultation and review of this element of the audit it became clear that, in some cases, whilst the bait-marking investigations had revealed (or confirmed) that certain setts should be classified as 'main' setts, such a classification would not necessarily have been obvious to the original surveyors, on the basis of the size and activity of the setts (assuming that these sets were indeed identified during MAFF's original surveys). One example which illustrates this issue is the small 'main' sett of Group 5 on figure 5. At the time of MAFF's initial surveys, this sett comprised three partially-used or disused holes. At the time of the audit surveys the holes had all become conspicuously well-used and the bait-marking revealed the apparent presence of a small, distinct social group. During follow-up surveys later in the audit, activity at this sett had declined and by the time that MAFF performed their own re-survey the sett had apparently become disused once more. In order not to over-emphasise discrepancies between MAFF's original surveys and re-surveys carried out as part of the audit, an amended series of Category 1 scores are also presented, with this small number of potentially misleading sett records deleted.
It is important to stress that the imposition of these different categories inevitably will mask the complexities in the discrepancies between the surveys. Attempting to amalgamate and classify the different reasons for these discrepancies (which was also necessary to facilitate analysis and reporting) would also be expected to mask some of the details. Wherever possible, any apparently important reasons for differences between the surveys have been highlighted and included in Sections 2.2 and 2.3 (below).
One potentially important factor was the habitat in which the various setts were recorded. In order properly to 'stratify' the sample areas prior to the re-survey, data were collated from existing Phase I habitat survey information and notes on the MAFF sett record forms. In seeking to assess the importance of this factor, these data were combined with field observations compiled during the re-survey.
2.1.3 Consultations
Various ad hoc discussions took place with supervisory and administrative staff during the project, in addition to a small number of more formal meetings. The intention of this aspect of the Audit was to obtain a picture of the various administrative and operational activities associated with surveying procedures, both in an attempt to identify any additional potential sources of error or inconsistency, and to ensure that the recommendations set out in Section 2.3 were appropriate and could reasonably be applied in practice. There was also a short series of more detailed review meetings at the end of the audit period, during which these recommendations were discussed.
2.2 Results and discussion
2.2.1 Accuracy of identification and classification of setts
A total of in excess of 300 setts (or sett records) were involved in the re-surveys, of which 58 were 'main' setts. The overall proportions of the different 'agreement categories' is set out in Table 1.
In addition, a further 24 'sett' records were identified during the re-survey as relating either to rabbit burrows (the majority) or to fox earths, each without any conclusive signs of past badger activity.
It is important to re-state that the imposition of the different 'agreement categories' to some extent masks the complexities involved in disentangling the discrepancies, or otherwise, between MAFF's original field surveys and the re-surveys. Attempting then to categorise the reasons for these discrepancies exacerbates the problems of imposing categories on what are a complex mixture of sett-specific issues. However, without adopting some form of 'reductionist' approach, interpreting these data would be impossible.
As can be seen from Table 1, the majority of the Category 1 records relate to outlying setts, and in the vast majority of cases these are setts which were recorded during the re-survey but do not appear on the original MAFF field survey maps. In a small number of cases these setts have since been recorded during repeat surveys undertaken within the Reactive Treatment Areas.
A significant proportion of the Category 1 records also relate to main setts; however, in this case the majority of these are the results of mis-classification (12 out of 16). Of these 12, six were classified largely on the results of bait-marking alone and (as discussed in Section 2.1.2 above) there were some concerns as to whether MAFF's original surveyors would have reasonably been expected to identify these as main setts (or as possible main setts). For this reason these setts have been deleted from the figures for 'Revised Category 1'. Nevertheless, in each case notes were made during the audit surveys to the effect that the status of these six smaller setts should be checked during the bait-marking investigations. For the remainder, classification was more straightforward, with these setts being recorded during the audit surveys either as main setts (without equivocation) or as 'main/subsidiary setts' (with a note that their status as main setts should be confirmed by baitmarking).
NB. The main sett numbers in parenthesis (Categories 2 and 3) indicate the adjustments as a result of the revised Category 1 figures.
A total of four main sets appeared to have been missed during MAFF's initial field surveys. One of these had subsequently been identified during a repeat survey (as for the outlying setts above). In all, eight larger setts appeared to have been missed: the four main setts, plus one relatively large annexe sett and three of the larger subsidiary setts. The remaining annexe and subsidiary sett in this category were small.
In contrast to the Category 1 records, the majority of Category 2 relate to main setts. These represent a combination of mapping discrepancies, substantial differences in size and activity, and apparent misclassifications which were not confirmed by bait-marking. In this regard, this category has, to some extent, been inflated by records from the triplet C cluster, where no bait-marking has taken place. A total of eight main setts were recorded as Category 2 from this area; at least three of these appeared almost certainly to have been mis-classified and hence are equivalent to those main setts included under Revised Category 1 (above).
2.2.2 Reasons for apparent discrepancies
Setts 'missed' during initial surveys
Very few setts that had been identified during MAFF's initial surveys could not be located during the re-surveys (a total of four, all smaller setts). Thus, the majority of this category relates to setts not appearing on the initial field maps.
In the case of outlying setts (and the few small annexe and subsidiary setts), this could be for a number of reasons, primarily:
- the sett appeared disused, with few, if any, obvious signs of past badger activity and was seen but ignored;
- the sett had appeared in the short interval between the two surveys; or
- the sett was simply missed.
It is very difficult to say with any degree of certainty what proportion of the discrepancies are accounted for by each of these reasons, but the apparent age of the spoil heaps outside many of the holes in question suggest that a significant proportion must fall into categories (a) and (c).
In the case of main setts (and the larger annexe and subsidiary setts), fewer options are available, and these setts appear simply to have been missed. Contributory factors are discussed below.
Main setts mis-classified during initial surveys
It is likely that these mis-classifications arose as a result of surveyor 'error', but also as a direct result of failings/drawbacks in the operating procedures. These are dealt with in some detail in Section 2.3 and include:
- the restriction of survey activities to discrete land-ownership 'parcels';
- the surveying of contiguous boundaries often by different surveyors / pairs of surveyors and often at different times;
- the need for surveyors initially to ascribe setts as main setts or others on the basis of their survey information alone;
- the use of a 'main' sett or 'other' classification rather than a more comprehensive (and hence informative) system; and
- the limited amount of practical 'feed-back' in the process as the survey data are compiled and the tessellations are drawn.
Sett records falling into Category 2
As with small 'missed' setts, it is possible that the size and activity of setts changed substantially between the surveys, but this is the least likely reason for the Category 2 records. In most cases these would be expected to be the result of surveyor 'error', compounded in many cases by the factors identified for mis-classified setts (above).
Mis-identification of setts
During the re-survey, each of the smaller setts and other holes were inspected carefully for confirmatory signs of badger activity. Only those with conclusive signs of past activity (or those likely beyond reasonable doubt to have been used by badgers) were recorded as setts. Thus setts, or other holes dealt with in a more cursory manner during MAFF's initial surveys may inadvertently have been mis-identified and contribute to the 24 'sett' records not recorded as badger setts during the re-survey. Alternatively, it is possible that changes took place between the surveys, substantially altering the appearance of the holes (e.g. partial collapses), and all that was visible by the time of the re-survey were signs of the current occupants (i.e. foxes or rabbits). However, on the basis of the re-survey results, this appeared to be the least likely option.
2.2.3 Contributing factors
Further analysis of the categories identified in Table 1 was undertaken in an attempt to discern patterns in the data. The effects of the most (potentially) important factors are identified below:
Surveyors
Whilst the sample areas were chosen to ensure that examples of the work of as many surveyors / pairs of surveyors as possible were included in the Audit, and that the results would not be biased by the preponderance of one particular survey team, the number of sett records from any one surveyor was insufficient to discern significant differences between surveyors. The apparent discrepancies appeared to be spread widely across the surveyor pairs. Every surveyor, and pair of surveyors, provided at least one Category 1 or Category 2 record, and the trend was for multiple Category 1 and 2 records for each pair. Whilst some pairs provided greater proportions of Category 1 records, none of the differences were significant.
Habitat
There was a significant difference in the overall proportion of setts 'missed' in woodland and scrub compared with all other habitats, with a greater than expected proportion of all setts being 'missed' in the woodland and scrub. When looking at the small number of large setts 'missed', the pattern is even more striking, with all of the 'missed' setts occurring in woodland or scrub. In addition, all but two of the setts where significant differences in size were noted were also recorded in woodland or scrub.
Timing of survey
There were no significant differences associated with the timing of MAFF's initial surveys (although there was relatively little variation: the majority of the audited surveys were undertaken during autumn and winter). However, in the case of the triplet C sample, a much larger number of rabbit burrows appeared to have been mis-identified as badger setts, and, in parallel with this, fewer outlying setts appeared to have been 'missed'. There appeared to be no explanation for this observation, other than perhaps a more diligent/cautious approach to recording outlying setts in this area.
Administrative errors
Very few administrative errors were revealed by the Audit. A total of only three errors contributed to Categories 1 and 2 in Table 1, and in two of those cases there were apparent errors in the survey results also (the administrative errors related to errors in transcription of information from the sett record sheets and also between the sheets and the field maps). The vast majority of any mapping errors appeared to have been made in the field.
These comments notwithstanding, problems have been perceived with regard to the transcribing of field data and the conglomerate maps produced. The maps produced appeared to be unclear and unwieldy, the different variables were difficult to extract and analyse, and the methods by which the data were processed and mapped appeared likely to introduce errors. Certainly, the process of selecting sample clusters of social groups (as described above) was made extremely difficult and time-consuming as a result of the way in which the data were collated.
2.3 Recommendations
It is important to record here that the details of this part of the Audit are not intended either as implicit or explicit criticisms of the individual surveyors in question, simply a statement of the factual outcomes of the re-survey. In addition, no comparison of the relative skill and expertise of the surveyors is intended; staff involved in the re-survey were already provided with sett details, and their objectives were somewhat different to those of the MAFF surveyors. However, these comments notwithstanding, the recommended improvements identified in this section are by no means working towards an impossibly high standard. It is important that as many as possible of the setts (and crucially all of the larger setts) are found and classified correctly, in order that all of the other elements of the trial which depend on these data can be pursued in an effective and reliable manner.
Finding large setts is obviously of crucial importance. Correctly classifying main setts is also a key factor, since the tessellation procedure and ascribing sett 'ownership' is fundamentally dependent upon this, as will be many related elements of the Trial relating to badger density, etc.. However, since any follow-up surveys or actions will tend to focus on setts which have already been identified, then finding as many of the smaller setts as possible, and mapping all the setts accurately, is also important. Whilst errors relating to sett size and activity would probably be resolved by follow-up surveys (in at least some of the Treatment Areas), these data are still likely to be used in some aspect of the Trial analyses.
2.3.1 Revised approach to the procedures
Fundamentally, surveying the Treatment Areas needs to be an iterative process. At present, the errors appear likely to be caused in part by the fragmented approach which, for a variety of reasons, has evolved. The piecemeal 'signing-up' of landowners, and the strict adherence to farm boundaries, is counter-productive. Ideally, the area to be surveyed should be dependent upon the distribution of setts, field signs and time available. On the basis of discussions towards the end of the audit period, it appears that some advances are already being made in this direction.
It is therefore recommended that the importance and practice of sterilisation between farms, and the strict adherence to land-ownership boundaries, be subject to a critical, pragmatic review (particularly since surveying should not necessitate visiting (at least with the same footwear) the most critical areas (livestock sheds, farmyards etc.)). In addition, the practical implications of not commencing surveys in a particular area until blocks (e.g. tetrads) of land can be surveyed as integral units should be carefully reviewed.
A continuing review of the pattern of survey data that are being compiled is also very important. Reviews of this information, not on the basis of repeated checking for completeness of land-ownership maps but with regard to larger areas, should both be on-going and triggered by certain stages (e.g. the first attempts at tessellation). It would be helpful, therefore, to begin tessellating the territory boundaries at an earlier stage. Elements of this approach already appear in the SOPs, but there was little evidence of the effect of this on the information subject to this Audit. Consultations with the supervisory staff suggested that the on-going review of data was improving in subsequent triplets, but the impression gained was that this was still focusing on smaller areas rather than a strategic overview, and that considerable progress could still be made in developing the survey of each Treatment Area as a truly iterative process.
2.3.2 Revised approach to working practices
It is recommended that a more detailed sett classification system be used, ideally similar to the standard system set out in Appendix I. This should promote a more informed and sophisticated assessment of badger social and spatial organisation in the Treatment Areas. It would also be sensible to use the system which has formed the basis of the last two National Badger Surveys.
It is also recommended that a staged, precautionary approach be encouraged with regard to the classification of larger setts, so that surveyors are only required to identify main setts on the basis of their initial survey results when such a classification is obvious. For those whose status is equivocal, this should be dependent upon survey results from neighbouring areas, and subject to re-inspection by more experienced staff if necessary.
It would be appropriate to reinforce encouragement for a team approach to surveying difficult habitats. This is identified in the SOP, but there was little evidence of it having taken place in the areas covered by the Audit. Certainly, earmarking an area for a return survey with additional personnel (possibly including minor clearance with hand tools to facilitate access in the most extreme cases) would be preferable to the real chance of missing a large sett.
Where significant doubt remains concerning the status of a larger sett, the use of a time-efficient technique, such as limited bait-marking, should be considered to resolve the situation and further improve the reliability of the data and the techniques (and other elements of the Trial) which rely upon them.
2.3.3 Training
It may be helpful to consider additional training of survey staff, focusing on main sett classification. It should be possible to design some appropriately targeted training involving, in particular, the correct identification of main setts (especially small main setts) in a variety of habitat types, perhaps including dealing with the specific instances of mis-classification etc. recorded during the Audit. It is important for the Audit to recognise, however, that MAFF's surveys were carried out under significant time pressure, and by auditing the earliest stages we have been dealing with some of the less experienced staff. Training is likely to have improved during the Audit period.
2.3.4 Administrative practices
Although few errors were identified with regard to this aspect of the survey procedures, some improvement may be possible. At present, there appears to be a significant amount of checking of field survey maps (by staff who probably have not seen that particular parcel of land), but no checking of the final maps by the field staff who have carried out the survey. Ideally, the field staff should be responsible for data transposition. Given that this is unlikely to be feasible, they should at least have a role in checking the composite field maps. There should also be a routine administrative check of data once entered on the CLA database.
2.3.5 Revision of survey data
A sound protocol is needed for the revision of survey data and the treatment of apparent errors found during the pre-culling re-surveys. In addition, some procedure is needed to deal with the problems of up-dating the tessellations and related information to cope with natural changes and changes induced by successive reactive BROs.
2.3.6 Further audit
Finally, depending upon the implementation of any of the recommendations outlined above, a further audit of the effectiveness of any revised procedures would be appropriate.
Page last modified:
12 August 2003
Page published: 5 February 2003