First Report of Statistical Auditor
I write to report under the terms of my appointment on (1) the initial statistical calculations establishing the scale of trials, and (2) the policy on release of data from the trials.
1. Design and scale of the trials
At my request, the Independent Scientific Group's statisticians, Sir
David Cox and Dr Christl Donnelly, set out their approach to the trials
in the paper 'Notes on statistical aspects
of the badger culling trial' 
(978 KB). I subsequently held a meeting with the
statisticians and the ISG's
Chairman, Prof. John Bourne, to discuss this paper in detail.
I am happy to endorse all the main points set out in that paper. The following paragraphs cover these main points in turn.
1.1 Initial power calculations for the trials
The 'initial power calculations' should be seen as purely indicative,
determining the design of the trials. As Cox and Donnelly say 'The
precision achieved in the trial will be determined by the data obtained,
totally independently of the correctness of the power calculations
'.
1.2 Design of the trials
There are two basic ingredients in the design of the trials, the number of areas studied and the length of time for which they are studied.
Cox and Donnelly explain why, for conclusions to be robust, the number of study areas included in the trial (10 'triplets', each consisting of three areas of approximately 100 square kilometres) could not be reduced, and for reasons of diminishing returns could not be significantly increased (the Krebs Report, §5.6, indicates that the trial area may include approximately two-thirds of all repeat or contiguous breakdowns).
The duration initially set for the trials, asking for a total of 50 triplet-years, is based on simple calculations given in the Krebs Report. As Cox and Donnelly say, these calculations were based on a simple assumption of Poisson error, which is likely to be somewhat optimistic, and a cautious assessment of future breakdown rates, which is probably pessimistic (from the point of view of duration required).
It is important to note that if these initial calculations do prove over-optimistic, it will in no way invalidate the trials, it will simply mean that their precision is less than expected. If the resulting precision is thought too low, the trials could be continued until the desired precision is achieved (see §1.5 below).
Recommendation 1
The ISG should give a more refined estimate of the expected duration and precision of the trials once sufficient data are available, perhaps after their first interim analysis.
1.3 Methodology for analysis
The paper sets out a sound basis for the methodology to be employed for analysis once sufficient data are available.
I see the most important part of my future role as monitoring the ISG's statistical work once such analysis starts.
1.4 Hypothesis testing or estimation
Given that eventual policy will be determined by an economic evaluation (taking 'economic' in a broad sense, to allow for social, ecological and ethical aspects), the emphasis on estimation rather than hypothesis testing seems sensible.
1.5 Stopping rules
Given uncertainty concerning the precision of estimates, it seems sensible not to determine a stopping rule, i.e. a specification of when to conclude the trial, simply in terms of the number of triplet-years. [As mentioned above, the initial power calculation suggested 10 triplets x 5 years, = 50 triplet-years.]
Care, as Cox and Donnelly recognise, is needed if the period of the trial is not fixed in advance. In particular, a stopping rule of the form 'stop at the first time for which a result is obtained, that would have been significant at a certain probability level p if that time had been predetermined' is unsound. It does not genuinely have significance level p, because multiple opportunities for 'success' are being allowed.
A simple legitimate suggestion, as described by Cox and Donnelly (referring to Anscombe, 1953), is to set the precision required in estimation of breakdown rates (or of the difference between them), stopping as soon as this level of precision has been achieved. The level of precision required could be set such that a certain percentage difference in rates, x say, would correspond to rejection of the hypothesis of equality in a significance test. The Krebs Report, and initial power calculation, used a value of x = 20%; I have not seen a justification of the use of this particular value; clearly it would be best if whatever value is used can be justified as reasonable in relation to the economic evaluation of policy options proposed (§7.7 of the ISG's Second Report).
I therefore support Cox and Donnelly's conclusion that the ISG should discuss what level of precision is suitable, and would go a little further, as follows.
Recommendation 2
For the reasons outlined in §1.5, the ISG should clarify the circumstances in which the trials might either be stopped early or prolonged beyond their originally projected duration. Again, the best time for this might be following their first interim analysis; they might also want to take into account any economic analyses available of policy options.
Data release
I have also discussed the ISG's policy on data release with the statisticians and the ISG's Chairman, and am happy to endorse their policy as set out in §8.3.5 of their Second Report. This balances a commitment to release data where possible with the need, during the trials, to keep confidential a narrow band of data whose release could prejudice the integrity of the trials.
Denis Mollison, 13th November 2000
Page last modified:
5 June 2007
Page published: 5 February 2003