Chapter 2 - Issues: IRA methodology
2.1
This chapter reviews some matters to do with IRA
methodology which were raised in evidence:
-
what the time horizon for assessment should be;
-
how to acknowledge sub-threshold risks on a
number of pests;
-
use of probability distributions in IRAs; and
-
assessment of consequences.
What should the time horizon for assessment be?
2.2
The probability of importing a disease or pest depends
on the volume of imports being considered. Since a longer period of time
implies a greater volume of imports, the assessed probability will depend on
the period of time of interest.[21]
2.3
The draft IRA report chose an assessment period of 12
months, in line with BA’s Import Risk Analysis Guidelines. It justified this saying:
In these Guidelines, a period of 12 months was chosen because it
allowed for the estimation of seasonal effects, but did not require long-range
predictions regarding trading practices, plant or commodity production factors
or pest biology.[22]
2.4
In fact the IRA guidelines contain no clear explanation
of this policy. The only references are:
The OIE Code suggests
that 1 year be adopted as period of time by which to evaluate the effect of a
projected volume of trade....
Biosecurity Australia
has designated 1 year as to be the standard period for which the effect of
trade volume is estimated.[23]
2.5
The referenced OIE Code, at least in its current
version, does not appear to make any relevant comment.[24]
2.6
Mr Peasley
argued that a one year horizon for risk assessment is too short:
An accepted time frame for planning farm business operations is
at least 5 years and I believe the risk should be calculated over this time
frame, not one year. This risk would then be 5 times that of one year and
include the increased volume of bananas. I don’t believe the impact or
consequences for the Australian banana industry, and the environment at a
local, district, state and national level can be realistically assessed over a
12 month period.[25]
2.7
The ABGC argued that both a one year and a ten year
period should be considered, as the ten year period ‘provides a clearer
indication of the risk of a pest over a reasonable longer term period.’[26]
2.8
The ABGC argued that ‘based on the IRA team’s own
scientific conclusions.. even if the recommended area of low pest prevalence
regime is imposed...there is a 99% chance that Moko will enter, establish and
spread in Australia
within 10 years after the importation of Philippine bananas commencing’.[27]
2.9
Similar concerns arose at a hearing of the Committee’s
recent inquiry into a pigmeat IRA. BA commented:
Ms Harwood—Essentially,
we have to define the appropriate level of protection—ALOP—in a way which allows
us to measure against it, and that is that the quarantine risk is reduced to a
very low level in a year of imports...
CHAIR—But if it rises
dramatically over a period of years does that say that there is a certain
inevitability about us getting this if we keep going?...
Ms Harwood—It is a
statistical fact in that it is a probability multiplied through time. But the
fact is that the actual import policies are extremely conservative. They are
adjusted through time in the light of emerging information and conditions and
developments overseas. It is not a static situation, so it does not make sense
to multiply it out into—
CHAIR—What is your
version of the 10-year risk in percentage terms?
Ms Harwood—As I said, we use a reference
point for ALOP of a very low risk of the pest or disease entering Australia
within a year or during a year.
CHAIR—But what is
your 10-year prediction?
Ms Harwood—We are not
giving a 10-year prediction. The discipline that we are working with is to measure
against a declared reference point for ALOP.[28]
2.10
BA commented that it is reviewing the IRA methodology,
with implication that this will be one of the matters considered.[29]
Comment
2.11
There seems to be no clear justification for limiting
to a one year time horizon in assessing probabilities associated with volume of
trade. Reference to ‘seasonal effects’ justifies a period of at least one year. It does not suggest
that the period should be limited to no
more than one year.
2.12
As for the difficulty of ‘long-range predictions
regarding trading practices, plant or commodity production factors or pest
biology’ - these predictions may be desirable, but they are not necessary. Amending
a one year probability to give a five or ten year probability based on current information is a matter
of simple mathematics which does not require any further prediction. It only
requires acknowledging the possibility of inaccuracy if future probabilities
are in fact different from the present ones used for the calculation. In the Committee’s
view the information is still worth having.
2.13
In the Committee’s
view limiting assessments to a one year time horizon takes an unduly short term
view. Knowing the risk over a 10 or 20 year time frame is obviously a matter of
great concern to affected industries.
2.14
This does not involve changing Australia’s
chosen ALOP. It is simply a matter of giving stakeholders more information. It
may happen that ‘very low risk over one year’ means ‘low risk over 10 years’ or
‘moderate risk over 20 years’. The statements are interchangeable: they would
be different ways of describing a single situation.[30]
2.15
The Committee recommends that the risk assessment
methodology should provide for assessing risk considered over ten years as well
as one year.
Recommendation 1
2.16
The risk assessment methodology should provide for
assessing risk considered over ten years as well as one year.
How to acknowledge sub-threshold risks on a number of pests
2.17
Mr Peasley
argued that the IRA methodology does not adequately acknowledge a situation
where there might be just-below-threshold risk on a large number of pests.
Intuitively, this creates a greater overall risk than just-below-threshold risk
on one pest; but the methodology cannot reflect this.
I do not believe the issue of multiple pests risk has been
addressed adequately. I realise the risks of each pest are independent events,
however, my intuition says that when there are more than one pest just outside Australia’s
ALOP that the overall risk is greater. The risk estimation matrix allows for a
one pest assessment.[31]
2.18
In this Committee’s
inquiry into an IRA for New Zealand apples, Apple and Pear Australia Ltd
pointed out that some countries have risk assessment methodologies that allow
for this:
The USA
for example, uses a system of allocating points for each risk point (see
Attachment 3). This means that risk
factors accumulate from one issue to the next until a final score is achieved. This is an interesting contrast from the BA
multiplication model that can allow high risks to be substantially discounted
by low risks.[32]
2.19
BA commented that it is reviewing the IRA methodology,
with implication that this will be one of the matters considered.[33]
Comment
2.20
It is certainly true that if events are independent
(for example, ‘entry, establishment and spread of moko’; ‘entry, establishment
and spread of freckle’ etc), the more possible events are in question, the
greater is the probability that at least one of them will happen.
2.21
The Committee agrees
with Mr Peasley.
There is no reason in logic why risk assessments should not allow for this. The
event of concern to Australia
is not ‘entry, establishment and spread of moko...freckle... mealybugs....etc’, each
considered in isolation. The event of concern is rather ‘entry, establishment
and spread of at least one of the above.’ The probability of that is the sum of
the component probabilities.
2.22
In the Committee’s
view Australia
should be able to reserve the right to say, ‘In respect of an import proposal,
there are so many pests of concern that the cumulative risk is regarded as
unacceptable.’
2.23
The Committee recommends that BA should investigate
changing the risk assessment methodology to allow for the fact that the total
risk is greater, the more pests there are of concern.
Recommendation 2
2.24
BA should investigate changing the risk assessment
methodology to allow for the fact that the total risk is greater, the more
pests there are of concern.
Use of probability distributions
2.25
There was some discussion in evidence about the use of
probability distributions in risk assessment. There was debate over whether
risk assessments should ‘report the 95th percentile’ or the 50th percentile.
General explanation of probability
distributions
2.26
An unwanted event (for example, ‘the pest enters,
establishes and spreads’) will often be the result of a number of preceding steps
each of which has its own probability (‘the pest is in the harvested fruit... the
pest survives transport and storage’ etc.). The probabilities of the component
events are multiplied together to give the probability of the outcome event.
2.27
Making a probability distribution is a way of
estimating the probability of the final outcome when the probability of each step
is not known exactly, but is thought to lie within a certain range.
2.28
For each step a probability is chosen randomly within
whatever range is regarded as reasonable. These numbers are multiplied together
to give a notional total probability (probability that all the steps will
happen). This is repeated 1000-2000 times, the result being different each time
because the input numbers are different each time. The results are spread over
a range from the smallest possible total probability (from multiplying the
smallest values of the component probabilities) to the largest total
probability (from multiplying the largest values of the component
probabilities).[34]
2.29
According to the IRA guidelines, ‘This distribution should be interpreted by ‘fitting’
it to the most appropriate semi-quantitative category. The approach to fitting
that has been adopted by Biosecurity Australia is to compare the fifth, 50th (or median) and 95th percentiles of the output distribution with
the probability intervals [shown in Figure 1 in chapter 1 of this report].’[35]
2.30
The guidelines do not explain very clearly what ‘compare’
means and how conclusions are to be reached and by what rules. At one point the
suggestion seems to be that the probability distribution should be labelled as ‘very
low...low...’ etc according to which label covers the probability where the median
value (the 50th percentile) falls, on the basis that more than half the results
fall within this range.[36] At another
point the instruction is: ‘The distribution should be ‘fitted’ visually and by
virtue of the distribution statistics to the most appropriate semi-quantitative
interval.’[37]
2.31
The 50th percentile will always show a lower
probability than the 95th. This may or may not affect the assessed risk. It
depends on the details of whether reporting the 50th instead of the 95th puts
the probability in a different probability category, and whether ‘combining’
this with the consequences according to the risk estimation matrix changes the
risk category.
Use of probability distributions in
the banana IRA
2.32
The IRA applied the guidelines on use of probability
distributions as follows.
2.33
The first draft IRA report (June 2002) said that it ‘reported
the 95th percentile’. It explained this as:
Distributions obtained from simulations of this model were
fitted retrospectively to the most appropriate probability range. Where the
distribution spans more than a single range, a conservative (95th) percentile
was used to determine that which should be reported.[38]
2.34
The revised draft (February 2004) used the 50th
percentile:
Distributions obtained from simulations of this model were fitted
retrospectively to the most appropriate probability range using the median
value (or 50th percentile). The 50th percentile was chosen as it provides the
most robust measure of central tendency for skewed (unsymmetrical)
distributions.[39]
2.35
The ABGC argued that this change was made without
adequate explanation, and is effectively more risk tolerant. The ABGC noted that
the 50th percentile was also used in recent reports on pigmeat and New
Zealand apples, and was concerned that this
was a systemic change.[40]
2.36
At hearings on pigmeat on 8 March 2004, and the
following Budget Estimates hearing on 24 May, BA argued that if the biological
assumptions involved are already very conservative, adding an extra level of
conservatism by reporting the 95th percentile ‘would give an unrealistic
projection outside the bounds of biological reality’:
Dr Banks—I think that once we started to
use these guidelines in a series of actual import risk analyses, we realised
that it was more appropriate in each case to use the 50th rather than the 95th....
Ms Harwood—In
situations where all the inputs to the model are judged on a very conservative
basis, if all the steps in the import pathway as you are inserting the judgment
of likelihood and the entry into the probabilistic model are already very
conservative, as in the case of those IRAs, it is more appropriate to use the
50th percentile as a genuine reflection of risk than to use the 95th, which
would give an unrealistic projection outside the bounds of biological reality....
Senator BOSWELL—...Who
made the decision to go from 95 to 50?
Ms Harwood—... it was the panel working
together that chose to use the 50th.[41]
2.37
BA later advised that both the June 2002 draft and the
February 2004 draft did in fact report the 50th percentile. The reference to
the 95th percentile in the June 2002 draft was a mistake which arose from
carrying over text from a previous version. BA noted that ‘...this and other
aspects of Biosecurity Australia’s
methodology are under constant review and evaluation, and minor changes are
often necessary to reflect ongoing best-practice in simulation and other
aspects of risk analysis.’[42]
Comment
2.38
The Committee notes
the concerns of stakeholders that the procedure recommended in the IRA
guidelines (to report the 95th percentile) seems to have been changed as a
matter of policy without adequate explanation. It is unclear to the Committee
whether this is indeed a change of policy, or whether the choice is left to the
discretion of individual IRA panels and might depend on the circumstances of
the case.
2.39
The Committee recommends that the IRA guidelines
should state a clearer policy on use of probability distributions, and should
explain it better to allay the concerns of stakeholders.
Recommendation 3
2.40
The IRA guidelines should state a clearer policy on use
of probability distributions, and should explain it better to allay the
concerns of stakeholders.
Concerns about assessment of consequences
2.41
As described in chapter 1, the risk assessment
methodology involves ‘combining’ an estimate of the probability of an unwanted
event with an estimate of the seriousness of the consequences if it occurs. According
to the IRA guidelines each type of consequence is estimated at the local,
district, regional and national level, on a scale of highly significant/
significant/ minor/ unlikely to be discernible. The results for the various
types of consequences are combined using rules in the guidelines to give an
overall rating for consequences on a scale of extreme/ high/ moderate/ low/
very low/ negligible.[43]
2.42
The Australian Banana Growers Council had concerns that
this procedure is not sufficiently rigorous:
The calculation of risk requires, first of all, that we know the
annual likelihood of outbreak, establishment and spread and, second, that we
know the consequences. We have a unit of measure for the likelihood: it is
expressed as a probability... The problem with consequence is that it is just a
term. We do not have a scale for that. So when we multiply those two together
it is not possible to use the same terms or the same scale... The correct way to
look at risk is expected loss.[44]
2.43
In a submission to this Committee’s
inquiry into an IRA on New Zealand
apples, Apple and Pear Australia Ltd had similar concerns:
The ISPM draft Pest Risk Analysis for Quarantine Pests (IPPC,
September 1999) recommends that in assessing economic consequences ‘wherever
appropriate, quantitative data that will provide monetary values should be
obtained.’ It appears that no attempt has been
made by Biosecurity Australia
to quantify the economic consequences.
Where these are likely to be profound it is inappropriate to use a
qualitative scale. The estimation of
consequences without an underlying and clearly understood monetary basis to
confirm its rigour misleads the overall analysis.[45]
2.44
The Queensland Department of Primary Industries and
Fisheries was concerned about the possible effects of import on banana-reliant
economies in North Queensland:
It is likely that these shires would experience considerable
difficulties if adjustment as a result of the loss of the banana industry were
to occur.
2.45
The Department ‘strongly advocates that the
Commonwealth Government must address the socio-economic impact of changing
import restrictions...’
While it is acknowledged that it is important not to limit
trade, the DPI&F believes that the Commonwealth must ensure that any
inequitable burden placed on local communities dependent on the commodity in
question is properly addressed.[46]
2.46
BA commented that it is reviewing the IRA methodology,
with implication that assessment of consequences will be one of the matters
considered.[47]
Comment
2.47
The Committee is
sympathetic to the industry’s concerns about assessment of consequences. The Committee
accepts the point made in the IRA guidelines that some consequences (such as
change in social amenity) are harder to measure than others (such as change in
commercial production).[48] However it
appears that even where consequences should be measurable, the IRA has made no
particular effort to do so. For example, the impact of Moko on production is
discussed qualitatively in some detail.[49]
In the Committee’s view it should not be
impossible to put some figures on this, taking into account points such as
those made by Mr Peasley
about the difficulty of controlling Moko in North Queensland’s
highly mechanised farms (see paragraph 3.20).