Chapter 4
Monitoring programs
4.1
As noted in Chapter 2, the Yara Pilbara Technical Ammonium Nitrate
Production Facility (TANPF) was approved with a number of conditions under the Environment
Protection and Biodiversity Conservation Act 1999 (EPBC Act). These
conditions included the requirement for an air monitoring program, and a
spectral mineralogy monitoring program to be implemented.
4.2
This chapter explores evidence received from submitters detailing
concerns that the monitoring programs have been inadequate and that the reports
produced as a result have been inaccurate. Criticisms include experimental
design flaws, and the inappropriate application of scientific evidence.
Independent monitoring
4.3
Independent monitoring of colour change and spectral mineralogy of the
Burrup rock art has been undertaken by CSIRO since 2004. CSIRO has prepared
annual reports that compare the results of each year's monitoring program with
results since the program's inception in 2004. Dr Helen Cleugh, Director, CSIRO
Climate Science Centre explained to the committee that:
CSIRO was selected to undertake three projects to monitor the
heritage rock art sites on the Burrup Peninsula after responding to the WA
government tenders. The parameters or design of each of the three projects,
including the scale and scope, were set by the WA government at the outset, in
2004. The three projects were to monitor air pollution and dust deposition
rates, to measure colour change and mineral spectroscopy, and to undertake
accelerated ageing tests.[1]
4.4
CSIRO also conducted a series of air quality monitoring studies in
2004–2005 and 2007–2008 to assess the likelihood that air pollution from the
Burrup industrial area would affect the rock art. In addition, between 2004 and
2006, CSIRO conducted a series of accelerated erosion tests using fumigation
chambers to assess the impact of different pollutant scenarios, and to evaluate
the role that dust may have in rock surface modification.[2]
4.5
The Burrup Rock Art Technical Working Group (BRATWG), which ceased on 30
June 2016, was responsible for reviewing the data collected from the annual
monitoring program (and other studies) and made recommendations to the Western
Australian Minister for the Environment and the Western Australian Department
of Environment Regulation (DER) prior to the data being published on the BRATWG
website.[3]
4.6
The committee received evidence from Professor John Black detailing
concerns with the accuracy and adequacy of the monitoring work undertaken by
CSIRO. In particular, concerns were raised that the three key CSIRO reports
used by government and industry to 'justify' the establishment of the TANPF and
to set its emissions limits 'are flawed in terms of scientific methods,
analyses and/or interpretations'. In addition, 'there are serious concerns
about the appropriateness of instruments [and]
methods used to measure colour and mineralogy changes at Burrup rock art sites'.[4]
4.7
In an article exploring the 'inadequacies' of research undertaken by
CSIRO, Professor Black and co-authors, stated that:
The large number of inadequacies identified in the reports
indicates the authors failed to follow the scientific method, including
undertaking a thorough review of the literature in relation to the nature of
the rock surfaces to be measured or the suitability of the instruments used to
make measurements. The authors also appear to have failed to design the
experiments, particularly in relation to the variance in measurements, factors
associated with experimental procedures, the external environment that would
influence the measured values and the number of replicates needed to prove a
specified percentage...[5]
Air quality monitoring studies
4.8
Dr Ken Mulvaney, an archaeologist and heritage expert, commented on the
limitations of the air quality monitoring studies conducted by CSIRO in
2004–2005 and 2007–2008. He noted that the studies were conducted before the
Woodside Pluto LNG plant went into production and at that time when the Yara
Pilbara liquid ammonia plant was not in full production.[6] Dr Mulvaney told the committee that:
When those studies were done there were intermittent
activities occurring, but the two main companies at the time were the Karratha
gas plant and Rio Tinto's Hammersley Iron port facilities. They were the two
main places. That study stopped in 2008. But during that time what was known as
the Burrup fertiliser plant, which Yara now controls, was constructed. However,
for various reasons, it was not in full production. In fact, for one time it
was out for a good while, and that was before the Pluto gas plant development
had gone ahead.[7]
4.9
Dr Mulvaney concluded that 'it is unlikely that there is an accurate
capture of the total pollution load from existing industrial activities'.[8]
4.10
Dr Mulvaney expressed concern that as a result of these studies, the
state government and BRAMMC had promoted that 'the industrialised areas on the
Burrup Peninsula have considerably lower concentrations of air pollutants than
cities in Australia'. Dr Mulvaney noted that:
Considering that at the time
of these studies, the Karratha Gas Plant at Withnell Bay and the shipping of
iron ore through King Bay, were the only resource industries in operation. Such
levels of pollutants being on par with a two-four million population city;
surely would raise alarm not complacency over rock art preservation. It may
have been an independent committee, however with public statements like these,
it raises concern as to whom within the committee may have had sway; the State
Development Department perhaps. [9]
2007 Fumigation studies
4.11
Both Dr Mulvaney and Professor John Black were critical of the CSIRO
report, Field studies of rock art appearance. Final Report: Fumigation and
Dust Deposition. Progress Report: Colour Change and Spectral Mineralogy,
published in March 2007.[10] Criticisms included poor experiment design such as an inadequate selection of rock
samples and inadequate replication for statistical analysis.
4.12
Dr Mulvaney and Professor Black provided evidence on the implications
for the outcomes of the studies of inadequate selection of rock samples. Dr
Mulvaney firstly explained that the petroglyphs occur on a range of rock types
on the Burrup Peninsula, and that they were produced using a variety of
methods. Dr Mulvaney told the committee that:
...the art is produced wherever there is a surface expression
of rock. There are a number of major geologies; basalt is one, there are
volcanics as well out on the outer islands, in addition to the gabbro and
granophyre. In each, the images are produced differently. One of the other
features of the rock art of this place that makes it different to anywhere else
is that the petroglyphs are produced in a wide range of techniques, and that
partly reflects the rock. So some are hammered or pecked into the rock, some
are abraded and others are just lightly scratched. Some are just, literally,
bruising the rock surface. They are all reflective, in part, of the physical
properties of the rock they have been produced on, but there are also clearly
cultural aspects at play in the production of the art.[11]
4.13
Dr Mulvaney noted that the fumigation experiments were 'conducted on
samples from a single gabbro rock with only a thin weathering rind' rather than
on 'a range of lithologies known to have rock art (granophyre, dolerite and
gabbro, nor on differing surface weathering states)'.[12] Dr Mulvaney concluded that 'it is problematic to confirm from such an
inadequate study exactly what the effects of emissions are having on the rock
art or what increased loads may cause'.[13]
4.14
Similarly, Professor Black stated that the study 'measured the effects
of immersing iron ore in either dilute of concentrated organic compounds, acids
or ammonia and measuring changes in colour and mineralogy'. Professor Black
submitted that the results of this study 'have no relevance to rock art because
the measurements were made on iron ore and not Burrup rock surfaces'.[14] Professor Black told the committee that these experiments:
...looked at acid, which was concentrated acid, and other
organic compounds, concentrated or in a dilute sense, and they tested those on
iron ore. It is completely irrelevant to what we are talking about, which is a
test on the surface, the patina, of the rock art. It is completely useless for
understanding anything about the impact on rock art because it was done on iron
ore.[15]
4.15
CSIRO, in defending its use of iron ore samples, stated that it required
suitable non-invasive methods which did not damage the rock art, and which had
the approval of the Indigenous custodians of the land. As it was unable to
directly test the rocks in the protected area, iron ore was selected as an
appropriate proxy to examine discolouration as it 'contains a similar
mineralogical profile to the rock patina' which has a major composition of
hematite with minor goethite, quartz and kaolinite.[16]
4.16
Professor Black was also critical of another fumigation experiment in
this study which involved 'hourly cyclical temperature and humidity changes
during fumigation of Burrup rock samples with a combination of gases at two
concentrations, with and without dust'. Professor Black stated that the study used
'gas concentrations below those projected for the ammonium nitrate plant and
existing Burrup industry'.[17]
4.17
In addition, Professor Black was critical of the experiment design,
stating that the 'study included either no treatment replication or
insufficient replication for statistical analysis and was of no value for
drawing conclusions'.[18] Professor Black explained to the committee that CSIRO:
...used emissions that they suspected would be 10 times what
industry—the level of emissions is below what the companies are saying that it
will be, so even the concentrations were not at a high enough level. But what
was particularly non-scientific about it is that they did one set of
experiments with dust and another experiment without dust, and they measured
before and after these 30 days of going through cycles of temperature and
humidity. But, because there was no replication for the dust and there was only
one replication, and the values were quite different for the two replicates,
you cannot analyse it statistically, so there could be no statistical analysis
of it. My big criticism was that all of the claims were made without any
statistical analysis.[19]
4.18
CSIRO, in response to these concerns, submitted that the fumigation experiment
design was based on The Air Pollution Model (TAPM)[20] and CALPUFF[21] dispersion models provided by the Western Australian Government in the
experiment tender document. It noted that the tender document had stated that
'the CALPUFF models are likely to be under-estimates and TAPM models are likely
to over predict'. As a result, 'CSIRO tested the concentrations of the fumigant
gases at 10 times the peak emission levels generated by the TAPM
dispersion models.' It also noted that it 'is unaware of any information from
industry that supports Professor Black's statement "...the level of
emissions are below what the companies are saying that it will be"'.[22]
4.19
CSIRO also submitted that the dust experiments were 'performed using the
accepted scientific approach to observing spectral change by difference and
were designed with sufficient statistical power for the required analysis'.
Specifically, the experiments included one case of dust exposure on two types
of rock surfaces. In addition, 'on each of the eighteen samples of rock,
replicate measurements were made at three different points, each separate point
approximately 2mm in diameter'.[23]
4.20
CSIRO explained that the spectral comparison involved assessing the
numerical differences between individual peaks that are normalised. It noted
that 'the spectra is normalised to ensure that the differences measured are due
differences in the sample rather than variable factors such as moisture.'
Further, the spectral comparison is involved in overall spectral comparison to
identify differences in peaks. It concluded that 'a statistical analysis of
these kinds of results is not a necessary approach and spectral comparison is a
widely accepted methodology'.[24]
2008 Burrup Peninsula air pollution
study
4.21
A number of submitters expressed concern in relation to one of the
conclusions reached in CSIRO's Burrup Peninsula Air Pollution Study: Report
for 2004/2005 and 2007/2008, released in 2008. This study was designed to
assess the likelihood that air pollution from the industrial area on the Burrup
Peninsula may damage the petroglyphs found in the area and was authored by Dr
Rob Gillett.
4.22
One of the conclusions of the study relied on a 1998 global assessment
of ecosystem sensitivity to acidic deposition authored by Cinderby et al. The
conclusion stated:
The critical load concept can be used to compare with
deposition fluxes to determine if adverse effects could result to rock or
aboriginal rock art. For a fuller discussion of this see Ayers et al. (2000).
The critical load has been defined as "a quantitative estimate of an
exposure to one or more pollutants below which significant harmful effects on
specified elements of the environment do not occur according to our current
knowledge" (Nilsson and Grennfelt, 1988). In a global assessment of
ecosystem sensitivity to acidic deposition Cinderby et al. (1998) have
determined a critical load or deposition flux of 25 meq m-2 yr-1 for the most sensitive areas of the world...In fact the assessment by Cinderby et
al. (1988) lists 5 sensitivity classes consisting of 25 meq m-2 yr-1,
50 meq m-2 yr-1, 100 meq m-2 yr-1,
150 meq m-2 yr-1 , 200 meq m-2 yr-1 and >200 meq m-2 yr-1, and places the Burrup area in
the least sensitive class. This means that the critical load for the Burrup
area is at least 200 meq m-2 yr-1, and since this is
significantly more than the observed deposition fluxes at the sites they are
unlikely to cause any deleterious effects to rock or rock art on the Burrup
Peninsula.[25]
4.23
This conclusion in particular was criticised by Professor Black, and Dr
Johan Kuylenstierna, one of the authors of the Cinderby report. Dr
Kuylenstierna submitted that Dr Gillett's assertion that the critical load for
the Burrup area is at least 200 meq/m2/year is incorrect. Dr
Kuylenstierna submitted that:
...the use of the Cinderby et al 1998 global sensitivity map
and critical loads to say anything of relevance to the rock art in the Burrup
Peninsula is just plain wrong—for many reasons and should not be used in
evidence to the committee. It cannot be used by industry or governments to justify
acid load emissions of 200 meq/m2/year. Rather a careful analysis of
the rock art and its sensitivity to acidic inputs is needed.[26]
4.24
Dr Kuylenstierna explained to the committee that:
The maps which we developed were based on soil type and the
idea was that, if you have something which has lots of minerals that can
weather quickly, then the ecosystem will be safe. That is a different end point
than the weathering of rocks with rock art and therefore it is not really
relevant in this case. As I understand it, some detailed work on the impact on
rock art is required rather than referring to what is a global assessment to
give a broadbrush idea of what is sensitive to the ecosystems such as streams and
lakes can be to acid rain.[27]
4.25
Dr Kuylenstierna went on to comment that the basis of the critical load
assessment was soil type only and did not examine the characteristics of rocks.
Dr Kuylenstierna noted that in most cases, soil type reflects parent
material (i.e. rocks) however it can be significantly affected by weathering
processes and the build-up of organic matter. Dr Kuylenstierna concluded:
But the main point is that the [critical load assessment] map
does not directly reflect the rock type and therefore cannot be used to say
anything about the rocks where the rock art is carved.[28]
4.26
In addition, the sensitivity referred to in the Cinderby et al report
refers to the sensitivity of ecosystems (i.e. the vegetation or surface waters
such as lakes and streams) and does not refer to the sensitivity of rocks to
weathering. Dr Kuylenstierna explained that:
If anything the inverse is true, as more rapid weathering of
minerals in the soil leads to better buffering and less damage to ecosystems –
but the process would be more rapid weathering in these areas. Either way this
is an inappropriate use of the critical loads – the rocks in a highly buffered
region would weather faster.[29]
4.27
Dr Kuylenstierna noted that weathering processes are complex and
specific to rock types. In order to determine how the surfaces of rocks on
which art is carved would be affected by acidic inputs, Dr Kuylenstierna
submitted that it is necessary to develop an understanding of the weathering
processes of those specific rocks.[30]
4.28
Further, Dr Kuylenstierna stated that the scale of the global soil maps
used in the Cinderby et al study was 1:5 million which shows broad patterns
rather than local detail. Dr Kuylenstierna again reiterated that 'these are
soil maps and not geology maps, and so still misses the point—the method is not
based on an assessment of the geology'.[31]
4.29
Professor Black described the experiments conducted as part of the Gillett
report as sound, noting that the data was well analysed and the report well
written. However, Professor Black expressed concern that the report had
utilised the findings of the Cinderby study to conclude that the rock art of
the Burrup Peninsula would withstand the highest critical acid load on the
international scale. Of especial concern was the fact that this conclusion had
been reached without measuring the buffering capacity of Burrup rocks.[32]
4.30
Professor Black submitted that the total acid load emitted from the
TANPF should be less than 25 meq/m2/year in order to protect
the rock art of the Burrup Peninsula.[33] Professor Black stated that no measurements of critical acid load for rock
patina on the Burrup Peninsula have been made because the buffering capacity of
the rock surfaces has never been measured. Therefore, there is no empirical
evidence for critical acid load for rock surfaces on the Burrup Peninsula. As
such, an acid load of 25 meq/m2/year is based on comparisons of
critical loads for other parent rock types and ecosystems.[34]
4.31
Professor Black particularly noted that the rocks of the Burrup
Peninsula are igneous and formed under great pressure, which makes them
extremely hard, and are amongst the slowest eroding rocks in the world.
Consequently, little soil is formed where petroglyphs occurs and erosion of parent
rocks is strongly related to buffering capacity. Professor Black submitted that
the slower erosion rate of rocks on the Burrup Peninsula would create critical
loads which are less than those for granite rocks.[35]
4.32
Professor Black also stated that scientific principles and empirical
evidence shows that rock patina dissolution commences once pH falls into the
acidic range and that the acidity of rock surfaces on the Burrup Peninsula are
already in the strongly acid pH range of 4–5. As such, the total acid load
emitted from the TANPF should be as low as possible.[36]
4.33
CSIRO responded to criticisms of the air pollution studies and submitted
that measuring the acid load of the Burrup rocks was not in the scope of the
work it was contracted to carry out. Rather, it was contracted to undertake independent
air monitoring where it 'determined the total deposition of sulfur and nitrogen
from the atmosphere by measuring sulfur and nitrogen compounds in samples of
gases, particle and rainwater at several locations'. It then compared the data
it measured as part of this work to other locations including similar sites in
Malaysia and the Northern Territory.[37]
4.34
CSIRO, in defending its choice to utilise the Cinderby et al critical
load framework, and the level of 200 microequivalents, stated that it was
intended to provide context for the air monitoring data collected.[38] Dr Melita Keywood, Principal Research Scientist, CSIRO, told the committee:
As in any scientific study, when you produce information and
data it is really important that the data and the use of that data be put into
a context that the end user can understand. At the time that critical load
framework was the best that we had available for us to put the data and
information that we collected in context, and so that is what we used for that
reference.[39]
4.35
CSIRO stated that 'the critical load framework of 200 microequivalents
cannot be used as impact assessment criteria, and this was never the intention
of the comparison'. CSIRO further noted that the Gillett report was 'peer
reviewed by an independent international reviewer and [the Cinderby et al
framework] was the best comparison to use at the time'.[40] Dr Helen Cleugh, Director, CSIRO Climate Science Centre, stated:
I would also remind the committee that the project design and
the results have been published in a report that was peer reviewed and has been
published in a peer-reviewed journal paper as well, which included the set-up
of this design and this framework as well. As Dr Keywood said, it was the best available
framework at the time and we have not been advised that there was a better
approach that we could or should have used.[41]
Extreme weathering experiments –
2017
4.36
In 2017, CSIRO published Extreme weathering experiments on the Burrup
Peninsula/Murujuga weathered gabbros and granophyres authored by Erick
Ramanaidou, Gay Walton and Derek Winchester.[42]
4.37
The report was initially published in May 2017 by the WA Government. However,
following a critique provided by Dr Ian MacLeod, Dr John Black, Dr Simon Diffey,
and Dr Stephane Hoerle, the report was removed from the website of the
Department of Environmental Regulation.[43]
4.38
Subsequently, an amended report was published. This report stated that:
This is a preliminary study using novel sample preparation
methods to provide a new approach to determining the effects of solutions of
different compositions and concentrations on rock weathering. As a scoping
tool, it was very valuable in targeting future work. This study was conducted
on 110 samples and the results found here should be confirmed using a larger
dataset. It was not intended to serve as an exhaustive or definitive analysis
of the impacts of the chosen leach solutions on granophyre and gabbro rocks nor
was it intended as an indication for permissible pollution levels. The
precautionary principle should apply here and emission capable of producing pH
below 5.5 (the pH of rainwater) should be considered potentially harmful.[44]
4.39
Professor Black assessed this report as 'of little value for assessing
the effect of nitric acid, sulphuric acid, ammonia or ammonium nitrate load on
dissolution of rock surfaces or petroglyphs on Murujuga'. Professor Black was
both critical of the conclusions reached in the report and the experiment
design.[45]
Rock art monitoring 2004–2014
4.40
The committee received evidence critical of the regular independent
monitoring of colour and spectral mineralogy of the Burrup rock art which was
carried out by CSIRO from the program's inception in 2004. CSIRO prepared
annual reports that compared the results of each year's monitoring program with
the results collected in previous years.
4.41
Professor Black submitted that the authors of these reports 'claimed
there had been no change in colour of background rock or engravings over the
time of measurement without appropriate statistical analysis'.[46] Professor Black commented further that from 2004 until 2013, CSIRO 'were making
claims of no colour change by not looking at the fundamental measurement that
comes out of a spectrophotometer'.[47]
4.42
Professor Black explained that spectrophotometers measure:
...three components which we
call colour space variables. One is L, which says how light it is, with zero
the blackest black and 100 the whitest white. So it says: how light is it? The
second one is A, which is the red-green opposing colours. If it is positive, it
is red; if it is negative, it is green. The third one is B, which is the
blue-yellow opposing colours. If it is positive, it is yellow; if it is
negative; it is blue. They are the fundamental measurements. As a scientist,
you would say you should measure and statistically analyse the change of those
fundamental colours over time.[48]
4.43
However, Professor Black stated that CSIRO did not measure and
statistically analyse the change in these fundamental colours. Professor Black
submitted that instead, CSIRO measured and compared colour changes from one
year to the next rather than from original measurements. Professor Black told
the committee that:
What they did was to then take those colours and measure what
was called colour change. That is another formula that you can use to say: how
did that colour change from this point to that point? But what they did in
those early publications was to say, 'Let me compare the change of this year
with that year, and that year with the next year, and that year with the next
year, but not the first year with the last year.' Because each year goes up and
down a bit, they said, 'Well, there's no change,' but they did not ever do an
analysis from the top to the bottom.[49]
4.44
Professor Black noted that the BRATWG had provided him with a copy of
the 2013 report. An initial analysis of the report led Professor Black to state
that:
...they needed to do a thorough statistical analysis and to do
it over the whole period. And then they sent me back the one the next year, and
they had done some statistical analyses, but they still had not done the
fundamental statistical analysis of colour change across time. If that is not
done, you are not getting the fundamental description of the data and what it
means. And that is what we did when we reanalysed the data.[50]
4.45
Professor Black undertook to reanalyse the data collected by CSIRO and
stated that 'CSIRO conclusions have been shown to be wrong'. Further, that 'an
independent reviewer of the original reports and the data reanalysis report
questions seriously the integrity of the CSIRO data'.[51]
4.46
Professor Black told the committee that the reanalysis was provided to
the BRATWG through the Western Australian Government, and that CSIRO reviewed
and provided comments on the reanalysis. Professor Black stated that 'CSIRO
said that the statistical model that we had used was not the best model'. In
response Professor Black requested that CSIRO provide a model for use in the
reanalysis. Professor Black stated:
...of course with statistics you can have different models, so
we asked them to provide us with the model that they would like us to use.
Unfortunately, they never ever sent us a model that we could use. We then had
the meeting with BRATWG, at which CSIRO were present. At that meeting, there
was a question about whether all of the changes were in a similar direction,
and the committee asked us to work with CSIRO to establish a statistical model
and to prepare a paper for publication for refereeing.[52]
4.47
Professor Black noted that the reanalysis work was only conducted after
he signed a confidentiality agreement with the Western Australian Government
and that he was been prevented from sending this work to a peer-review journal.[53]
4.48
Professor Black concluded that 'the scientists involved in studies
initially accepted the errors identified, but refused to acknowledge them after
consultation within CSIRO'. Further that CSIRO 'appears to be more concerned
about its reputation than the fate of the world significant archaeological
heritage of Burrup rock art'.[54]
Independent review process
4.49
CSIRO's monitoring and analysis work has been reviewed a number of
times: first by Professor Black and co-authors, and then by Data Analysis
Australia as requested by the Western Australian Government. This section will
outline some of the key findings of this review process and the implementation
of recommendations.
Data Analysis Australia—2016 review
4.50
In 2016, Professor Black and Dr Simon Diffey conducted an analysis of
the CSIRO monitoring program. This analysis resulted in a draft paper[55] (henceforth called the Draft Paper) which suggested that significant changes
had taken place in the in the rock art of the Burrup Peninsula. This was in
contrast to the findings of the CSIRO reports. As a result, the then Western
Australian Department of Environment Regulation (DER) engaged Data Analysis
Australia (DAA) to review the statistical issues raised in the Draft Paper, utilising
the data itself, and CSIRO reports.[56]
4.51
DAA found that the statistical methods utilised by Black et al in the
Draft Paper to be 'highly appropriate (with some minor modifications) and they
represent[ed] a substantial step forward in effective monitoring of the Burrup
Peninsula rock art sites'. However, the DAA also concluded that the analysis could
not 'overcome the lack of confidence' in the data utilised and that it would
not be appropriate for the Draft Paper to be published in its form at the time
of the review.[57]
4.52
DAA noted that the Draft Paper utilised a significantly different
approach to the analysis of monitoring data than that utilised by CSIRO and
that this approach 'should be been used for some years'. DAA stated that the
approach taken in the Draft Paper 'provides the opportunity to examine longer
term trends, to understand whether there are issues affecting multiple sites
and to potentially contrast sites close to and far from the industrial
developments'. In doing so, Black et al were able to highlight a number of
inadequacies in the CSIRO reports—particularly the absence of proper statistical
analysis in earlier reports. DAA expressed regret that the Draft Paper was
affected by the problems with the data provided to the authors. [58]
4.53
DAA's review of the CSIRO reports and data also highlighted 'significant
problems of cross-calibration between instruments, inconsistent error-prone
data management, and clear errors in the data'.[59] It stated that although the twelve years of data collected by the CSIRO are a
valuable resource that should not be discarded, 'it is not appropriate for any
decisions—including whether or not changes have taken place on the Burrup
Peninsula—to be based on it in its current form'.[60]
4.54
DAA made a number of recommendations as a result of its analysis of the
Draft Paper, and CSIRO data and reports. These recommendations were as follows:
- The
historical data collected by the CSIRO should be systematically archived and
held by DER, with consistent naming conventions, both to provide a baseline
record and to facilitate comparisons with future data. The archival data format
should enable ready access to the data via standard statistical software such
as R1.
- The
CSIRO should be asked to revisit the cross calibration issues with the
BYK-Gardner (BYK) portable spectrophotometer and the Konica Minolta (KM)
spectrophotometer, both to ensure that the historical data is properly
understood and to confirm whether or not the historical BYK data is capable of
comparison with current and future measurement instruments.
- An
analysis similar to that of Black and Diffey should be conducted using verified
ASD estimates of L*, a*, b* ideally using the original ASD spectra rather than
the averaged spectra.
- The
publication of the Black and Diffey paper should ideally wait until the
problems with the BYK data are resolved or should use the ASD data.
- Future
work by the CSIRO should be based upon an agreed analysis plan certified by a
competent statistician. Since each year the CSIRO Reports have covered the full
data set since 2004, it would be appropriate for the next published Report to
incorporate this improved analysis and in doing so, make it clear that it
should replace the analyses in their previous Reports.
- Consideration
should be given to expanding the number of measured sites and in doing so,
improving the balance of the design to include more effective controls, if
feasible.
- To
maintain scientific rigour, future data collection should follow a fully
documented and detailed protocol, and ensure that departures are documented.[61]
Data Analysis Australia—2017 review
4.55
In 2017, DAA was requested to review a CSIRO draft report, Burrup
Peninsula Aboriginal Petroglyphs: Colour Change & Spectral Mineralogy 2004–2016 (CSIRO Draft Report), authored by Noel Duffy, Erick Ramanaidou, David Alexander
and Deborah Lau. The CSIRO Draft Report covered the data collection and
analysis conducted since 2004 as part of the Burrup rock art monitoring
program, with a focus on the possible effects of industrial developments. The CSIRO
Draft Report represents the latest in a number of reports developed by the same
CSIRO group that has presented earlier data from the monitoring program.[62]
4.56
DAA noted that the contract for the 2016 monitoring program required
CSIRO to address the recommendations of the 2016 DAA review. As such, the CSIRO
Draft Report aimed to address the shortcomings of earlier work highlighted by
the 2016 DAA review.[63]
4.57
The 2017 DAA review found that:
...a considerable amount of work has been done to address some
of the concerns. In particular there have been substantial improvements to the
statistical analysis of colour changes using linear mixed models and greater
care has been taken to highlight the problems associated with the BYK
spectrophotometer used in the early years of the monitoring program. There also
appears to have been action taken to better manage the data, both to make it
available for analysis and to preserve it for future years.[64]
4.58
However, DAA also concluded that 'significant work remains if the 2016
Recommendations are to be addressed'. It noted that:
- Recommendations 4, 5 and 6 had not been met;
- Recommendation 2 was not addressed;
- Recommendation 3 was partially met; and
- Recommendation 1 was largely met.[65]
4.59
DAA recommended that if this work cannot be completed for the CSIRO
Draft Report then 'it should at the very least be highlighted as work in
progress so the reader is not given to think that the Draft Report is complete
or its conclusions final'.[66]
4.60
DAA also made a number of observations in relation to the report:
-
'The use of the BYK data is highly problematic' and 'it is a
reasonable statement that little if any scientific weight can be given to it'.
Further, 'this needs to be made more prominent, and indeed the right solution
is probably to assign the BYK data to a historical note'.[67]
- The ASD spectrograph data and its derived colour measures have
been collected with reasonable consistency with one instrument since 2004,
though there are some concerns with the 2004 data. The Draft Report gives
prominence to the spectra from this instrument and to the spectral parameters,
but little attention is given to the ASD colour measurements.[68]
- However, the presentation of the ASD spectral parameters via 'numerous
small barely readable plots' is 'not particularly helpful'. Further, no
statistical analysis is conducted. As such, 'there is little purpose presenting
them as done in the Draft Report and no purpose if they are not going to be
statistically analysed'.[69]
- The application of linear mixed models to the ASD colour data
should be commended, and is a 'marked improvement' on previous reports. However,
the presentation of the models is unclear. Further, the most basic test of
whether the contrast between engravings and their backgrounds are changing at
different rates depending on whether they are situated closer industry was not
included.[70]
4.61
CSIRO's Draft Report concluded that 'the data is scarcely unequivocal
and there are reservations on the conclusions of the statistical analysis',
however DAA considered that this statement 'could be considered misleading' as
it gives the impression that the data is incapable of giving clarity 'whereas a
more thorough statistical analysis may be able to resolve the question more
completely'.[71]
4.62
DAA was also critical of the design of the monitoring program and stated
that:
It is unfortunate that, for whatever the reasons, this was
not based upon firmer statistical principles. More sites should have been
monitored, especially more control sites and the number of replicate
measurements taken at each point seems excessive (or unnecessary). Furthermore,
as there are concerns that the measurement process is damaging the engravings,
a fractional design is indicated where not all spots were measured each year.
It is not possible to fix the historically collected data but moving forward
consideration should be given to redesigning the monitoring scheme.[72]
4.63
DAA concluded that:
...we are of the opinion that while the Draft Report
demonstrates substantial efforts on the part of the CSIRO to improve the
reporting of the data collection and to present better analysis, more needs to
be done. In particular, in its current form the Draft Report is unable to
dispel what might be described as reasonable concerns about the impact of
industry on the rock art.[73]
CSIRO 2004–2016 report
4.64
As a result of the 2017 DAA Review, a number of changes were made to the
CSIRO Draft Report. The final version, the Burrup Peninsula Aboriginal
Petroglyphs: Colour Change & Spectral Mineralogy 2004–2016 (Final
Report) was released by the Western Australian Government in September 2017.
4.65
Dr John Steele, Director, Science Impact and Policy, CSIRO explained to
the committee that in effect, CSIRO had 'used the DAA commentary as a peer
review for the purposes of producing a final report'. Dr Steele also noted that
its Final Report supersedes all prior analysis done by CSIRO as part of the
rock art monitoring program.[74]
2016 DAA Review recommendations
4.66
The following section outlines the ways in which CSIRO responded to each
of the 2016 DAA Review recommendations in its Final Report.
Recommendation 1 – archiving of
data
4.67
CSIRO noted that it has fully implemented DAA's recommendation to
systematically archive the historical data. CSIRO's Final Report stated that:
All the historical data collected for the all the
spectrometers have been systematically archived and were sent to DER with
consistent naming conventions, in a data format that is easily read by standard
statistical software.[75]
Recommendation 2 –
cross-calibration issues
4.68
CSIRO submitted that the recommendation to revisit cross-calibration
issues with the BYK and KM spectrophotometers was no longer relevant in light
of its analysis of the data. As such, CSIRO's Final Report concluded that:
...as data from the BYK spectrophotometer appears unreliable
for drawing conclusions on colour change in the rock art, the cross calibration
issues with the BYK – Gardner (BYK) portable photospectrometer and the Konica
Minolta (KM) photospectrometer will not be undertaken. All the
photospectrometer data have been provided to DER for safekeeping.[76]
Recommendation 3 – analysis using
verified ASD estimates of L*,a*,b*
4.69
CSIRO submitted that the Final Report undertook an analysis using verified
ASD estimates of L*,a*,b* as recommended by DAA. CSIRO's Final Report stated:
For this report, combining the last two years of measurements
(2015 and 2016), a complete statistical analyses of all the data (each
individual measurement for the three instruments, a total of 24,000 colour
measurements from 2004 to 2016) has been undertaken.
Measurement of the annual colour changes used two
spectrophotometer techniques, the ASD and the BYK and KM. An examination of the
colour measurements as a function of time, as well as a comparison of the two
measurement techniques, has been conducted.
For both the KM and the ASD instruments, three-dimensional
L*a*b* colour space (L* - degree of lightness, a* - degree of red/green, b* -
degree of yellow/blue), identifying a tristimulus value (L*a*b*) for each
sample point have been calculated.[77]
4.70
CSIRO also explained that CSIRO's Final Report included other models
recommended by DAA. Further, the change in measurement practice for the ASD
spectrophotometer (replacing it for each measurement from 2015 onwards) was
documented and included in each of these analyses.[78]
Recommendation 4 – agreed analysis
plan for future work
4.71
CSIRO agreed that statistical analysis should be a key part of planning
for future analysis work, whether this work is conducted by CSIRO or other
organisations. It noted that statistical analysis was only one of a number of
technical issues that should be included in a plan for future analysis and that
a number of technical practicalities would need to be taken into account. CSIRO's
Final Report noted that for all future work it is recommended that:
A complete statistical
analyses is done on the full spectrum of each individual ASD spectrum (not just
the visible part i.e. L*, a* and b*).[79]
Recommendation 5 – expanding the
number of sites
4.72
CSIRO agreed that consideration should be given to expanding the number
of measurement sites and in doing so, improving the balance of the design to
include more effective controls. CSIRO's Final Report recommended that for
future work:
A study be conducted to assess how many new sites and how
many new engravings and backgrounds should be added to the current locations to
increase the quality of the monitoring in the Burrup Peninsula. In particular,
new control sites with similar rock types should be added to the current ones
(for instance Depuch Island). It should also be noted that by increasing the
number of independent measurement on each spot (in doing so improving
statistical analysis) could also have an adverse effect on the petroglyphs.
There were signs in 2015 and 2016 that instruments measurements might be
affecting the measured spots. A balance should be found between statistical
endeavour and petroglyph protection.[80]
Recommendation 6 – data collection
should follow a protocol
4.73
CSIRO noted and agreed with the DAA recommendation that in order to maintain
scientific rigour, future data collection should follow a fully documented and
detailed protocol. It submitted that such protocols 'will continue to be
important, including for future analysis work (whether to be conducted by CSIRO
or other organisations)'.[81]
4.74
CSIRO's Final Report included commentary on the protocols used by CSIRO
during collection.[82]
2017 DAA Review recommendations
4.75
The following section outlines the ways in which CSIRO responded to each
of the 2017 DAA Review recommendations in its Final Report.
Recommendation 1 – succinct
description of measurement framework
4.76
CSIRO implemented the recommendation to include a succinct description
of the measurement framework used. The Final Report included both detailed
descriptions of the measurement framework, detailed instrument information, and
a succinct description of aspects of the study relevant to the statistical
analysis.[83]
Recommendation 2 – address issue of
poor quality of BYK data
4.77
In accordance with both the recommendations in the 2016 and 2017 DAA
reviews, CSIRO directly addressed the issue of the poor quality of BYK data in
both the Executive Study and the Conclusion. Further, BYK data was not used in
the analysis of trends.[84]
Recommendation 3 – less reliance
should be placed on the ΔE measure
4.78
CSIRO noted that the report's conclusions are not based on the ΔE
measure but rather on the statistical analysis of individual colour components
(L*a*b*).[85]
Recommendation 4 – need for a
proper statistical analysis of spectral parameters
4.79
CSIRO's Final Report extended its statistical analyses in order to test
whether there have been any changes in colour over time, and whether these
changes are at different rates at sites near to or far from industry, and
whether the difference applies equally to background rock and engravings.[86]
Recommendation 5 – prominence of
findings regarding the BYK data
4.80
As noted above, CSIRO gave prominence to the findings that the BYK data
has limited if any value in both the Executive Study and the Conclusion.[87]
Recommendation 6 – comments
regarding BYK data and colour change
4.81
CSIRO explained that the recommendation that comments that the BYK data
does not indicate change should be deleted was in error. Rather, the Draft
Report had noted that the BYK data had not indicated a different rate of change
between the northern and southern sites. However, in light of the
recommendation, CSIRO added stronger caveats into the Final Report, including
those which identify the aforementioned issues with the BYK data.[88]
Recommendation 7 more information
on statistical models
4.82
CSIRO noted that further information on statistical models was included
in the Final Report in order that the models can be fully replicated by anyone
with the access to the data.[89]
Recommendation 8 – proper
documentation of measurement practices
4.83
CSIRO noted that all changes in measurement practices were fully
documented in the Final Report as recommended by DAA. Further these were also
incorporated into analyses.[90]
Recommendation 9 and 10 – formal
design and analysis plans
4.84
CSIRO noted that Recommendation 9 and 10 were recommendations for the
next period of data collection rather than the current report. These
recommendations were that a formal design document and a formal analysis
document be developed prior to the next period of data collection. CSIRO noted
these recommendations.[91]
Monitoring and conclusions
4.85
CSIRO's Final Report concluded that the monitoring undertaken of the
rock art indicated that there has been some small but statistically significant
change to the rocks in some dimensions of colour. It found that:
For both the KM and the ASD instruments, three-dimensional
L*a*b* colour space (L* - degree of lightness, a* - degree of red/green, b* -
degree of yellow/blue), identifying a tristimulus value (L*a*b*) for each
sample point have been calculated.
Data from the KM spectrophotometer shows a trend over time in
the L* measurements. The lightness (L) decreasing at a modelled average rate of
0.31 units per year (a total decrease of about 2 units on this scale is just
noticeable to the human eye). However no trend is indicated in either a*
(degree of red/green) or b* (degree of yellow/blue).
Data from the ASD spectrometer shows trends indicated in L*
(degree of lightness) and a* (degree of red/green) but not on b* (degree of
yellow/blue), though the evidence is not as strong as with the KM instrument.[92]
4.86
However, importantly, it noted that the results are not fully conclusive
but are nonetheless important and warrant further attention. Further, none of
the instruments demonstrate a difference in the rate of change between control
sites and those closer to industry. The report stated:
The results are not fully conclusive and if the measurements
do reflect real colour change, as the data suggest, then continued observations
would continue to mark out the trend more clearly; and if not, observations
will likely continue to fluctuate over time, making the randomness of the recorded
variation more apparent...Nonetheless, the indication of significant colour
change is important, and warrants closer attention. None of the instruments
demonstrates a difference in the rate of change between the northern control
sites and the southern sites closer to industry.[93]
4.87
CSIRO noted that the report does not explicitly address the reasons for
the colour changes and the possible reasons for such small changes could
include natural weathering. CSIRO stated that the report:
...does not provide a basis to confirm or to exclude an
attribution to the industrial development, other than to note that the measured
changes are not statistically significantly different at sites near to or far
from industry.[94]
Critique of report
4.88
Following the publication of Burrup Peninsula Aboriginal Petroglyphs:
Colour Change & Spectral Mineralogy 2004–2016, Professor John Black
provided the committee a critique of the report.
4.89
Professor Black argued that the report includes an 'important admission
to substantial errors in analysis and interpretation of all previous reports'.
Of particular concern was that:
...these reports have been used
by the Western Australian and Federal governments and industry to place the
ammonium nitrate production facility in the midst of the rock art and to
justify its high levels of emissions.[95]
4.90
Professor Black noted that the results indicated a colour change of
approximately 13 per cent over the past 13 years, and that this represents a
major change which should be of concern in the preservation of rock art.
Further, statistical analyses demonstrate 'significant changes in lightness of
the rocks' and both the KM and ASK instruments found significant changes in
colour despite the high variance in measurements from year to year.[96]
4.91
However, Professor Black argued that the CSIRO report attempts 'to
diminish the value of the significant findings relating to colour changes and
to changes in lightness of the rocks'.[97]
4.92
Professor Black noted that an improved design of experimental procedures
would significantly reduce the year on year variation in measurements. Further,
Professor Black maintained that the use of two sites located in the north of
the Burrup Peninsula as control sites for monitoring colour change is
inappropriate given the close proximity of ships entering and leaving the
Dampier Port.[98]
Draft Burrup Rock Art Strategy
4.93
Following the work undertaken by DAA and the release of CSIRO's Final
Report, the Western Australian Government released the Draft Burrup Rock Art
Strategy (the Draft Strategy) in early September 2017 for public comment.
4.94
The Draft Strategy outlines a long-term framework for the management and
protection of the Aboriginal rock art of the Burrup Peninsula. It acknowledges
the concerns raised by Professor Black, and the work undertaken by DAA. It
states that 'the framework in this strategy is intended to address the
limitations of the past monitoring and analysis program'.[99]
4.95
The Draft Strategy proposes that 'improved monitoring of colour contrast
and spectral mineralogy should be continued on an annual basis with review
after five years'.[100] It states that the Western Australian Government will develop a revised method
for the collection and analysis of data that incorporates the recommendations
of the DAA reviews. The revised method will be based on a number of principles
including:
- research questions will be developed in consultation with key
stakeholders;
- equipment and procedures used for monitoring will be reviewed to
ensure that they are best practice;
- the number of measurement sites will be calculated to ensure that
statistically significant conclusions can be reached from analysis of the data
collected;
-
the sampling method and analysis will be reviewed at least every
five years by experts who are independent of the key stakeholders;
- data analysis will be certified by a suitably qualified
statistician;
- statistical analysis will support the examination of long-term
trends to understand if there are issues affecting multiple sites, and to
contrast sites situated near and far from pollutant emission sources; and
-
additional control sites away from all major sources of emissions
including industry and shipping will be incorporated into the monitoring
program to the greatest extent practicable. Where possible, it should also be
possible to discern between both of these emission sources.[101]
4.96
The Draft Strategy proposes that data collection and analysis should be
undertaken by separate parties, with the statisticians undertaking the analysis
acquiring and maintaining an adequate understanding of the data collection
processes and techniques. It states that the annual monitoring program will be
based on a number of principles which detail how data should be stored,
published and reviewed.[102]
Other studies
4.97
In addition to the program for the monitoring of the rock art, the Draft
Strategy makes a number of recommendations for other studies which will assist
in protecting the Aboriginal rock art of the Burrup Peninsula.
Acid deposition
4.98
The Draft Strategy notes the evidence given by Dr Kuylenstierna at the committee's
hearing on 17 February 2017 that the Cinderby et al report is not relevant to understanding
the sensitivity the rocks of the Burrup Peninsula to acid deposition. As such,
the Draft Strategy recommends that a better understanding of: the sources of
pollutants; the current and likely future pollutant load; and the impact of
pollutants on the rock art is required.[103]
Air quality
4.99
The Draft Strategy describes current air and meteorological monitoring
on the Burrup Peninsula as 'reliable and targeted' but notes that 'improvement
would inform a detailed cumulative spatial analysis'. As such, the Draft
Strategy recommends the introduction of a long-term and coordinated monitoring
network across all industries to expand the knowledge base required to manage the
air quality in the region. It recommends that the network should measure
exposure of the rock art to air pollutants.[104]
pH
4.100
The Draft Strategy states that regular measurements of the pH of the
surface of gabbro and granophyre rocks on the Burrup Peninsula would assist in
the early detection of conditions that would impact the rock art. It recommends
the installation of monitoring stations including rainwater gauges to measure
rainfall, pH, cations and anions as well as deposition flux of nitrogen and sulfur.[105]
Microbiology
4.101
The Draft Strategy recognises the potential impact that microbial action
may have on the weathering of rock art. Noting the expansion of industry on the
Burrup Peninsula in the years since the last study was conducted, the Draft
Strategy recommends that a study to assess microbiological numbers and
composition would be valuable, particularly as the TANPF becomes operational.
The Draft Strategy also recommends that this study should be repeated from time
to time to ensure that knowledge of microorganisms present on the rocks of the
Burrup Peninsula is up-to-date.[106]
Source of pollutants
4.102
The Draft Strategy recommends that monitoring to measure levels of
pollutants at particular sites should be conducted to enable a determination of
the source of the pollution, and link any changes in the condition of the rock
art to critical loads for pollutants and their source (industry, shipping, or
other). The Draft Strategy also recommends that other causes of change to rock
art such as guano should also be investigated.[107]
Burrup Rock Art Stakeholder
Reference Group
4.103
The Draft Strategy includes the terms of reference for a newly
established consultative committee called the Burrup Rock Art Stakeholders
Reference Group (BRASTRG).[108] This group will assist in overseeing the design and implementation of the
strategy, and includes representatives from the Murujuga Aboriginal
Corporation, state agencies, local government, industry and the community.[109]
4.104
The terms of reference note that the role of the BRASTRG is to consult,
inform and educate other stakeholders on matters referred for input or comment
by the Western Australian Department of Water and Environmental Regulation. The
BRASTRG will also contribute constructively to the monitoring and management of
the rock art.[110]
Navigation: Previous Page | Contents | Next Page