Chapter 4


Chapter 4

Monitoring programs

4.1        As noted in Chapter 2, the Yara Pilbara Technical Ammonium Nitrate Production Facility (TANPF) was approved with a number of conditions under the Environment Protection and Biodiversity Conservation Act 1999 (EPBC Act). These conditions included the requirement for an air monitoring program, and a spectral mineralogy monitoring program to be implemented.

4.2        This chapter explores evidence received from submitters detailing concerns that the monitoring programs have been inadequate and that the reports produced as a result have been inaccurate. Criticisms include experimental design flaws, and the inappropriate application of scientific evidence.

Independent monitoring

4.3        Independent monitoring of colour change and spectral mineralogy of the Burrup rock art has been undertaken by CSIRO since 2004. CSIRO has prepared annual reports that compare the results of each year's monitoring program with results since the program's inception in 2004. Dr Helen Cleugh, Director, CSIRO Climate Science Centre explained to the committee that:

CSIRO was selected to undertake three projects to monitor the heritage rock art sites on the Burrup Peninsula after responding to the WA government tenders. The parameters or design of each of the three projects, including the scale and scope, were set by the WA government at the outset, in 2004. The three projects were to monitor air pollution and dust deposition rates, to measure colour change and mineral spectroscopy, and to undertake accelerated ageing tests.[1]

4.4        CSIRO also conducted a series of air quality monitoring studies in 2004–2005 and 2007–2008 to assess the likelihood that air pollution from the Burrup industrial area would affect the rock art. In addition, between 2004 and 2006, CSIRO conducted a series of accelerated erosion tests using fumigation chambers to assess the impact of different pollutant scenarios, and to evaluate the role that dust may have in rock surface modification.[2]

4.5        The Burrup Rock Art Technical Working Group (BRATWG), which ceased on 30 June 2016, was responsible for reviewing the data collected from the annual monitoring program (and other studies) and made recommendations to the Western Australian Minister for the Environment and the Western Australian Department of Environment Regulation (DER) prior to the data being published on the BRATWG website.[3]

4.6        The committee received evidence from Professor John Black detailing concerns with the accuracy and adequacy of the monitoring work undertaken by CSIRO. In particular, concerns were raised that the three key CSIRO reports used by government and industry to 'justify' the establishment of the TANPF and to set its emissions limits 'are flawed in terms of scientific methods, analyses and/or interpretations'. In addition, 'there are serious concerns about the appropriateness of instruments [and] methods used to measure colour and mineralogy changes at Burrup rock art sites'.[4]

4.7        In an article exploring the 'inadequacies' of research undertaken by CSIRO, Professor Black and co-authors, stated that:

The large number of inadequacies identified in the reports indicates the authors failed to follow the scientific method, including undertaking a thorough review of the literature in relation to the nature of the rock surfaces to be measured or the suitability of the instruments used to make measurements. The authors also appear to have failed to design the experiments, particularly in relation to the variance in measurements, factors associated with experimental procedures, the external environment that would influence the measured values and the number of replicates needed to prove a specified percentage...[5]

Air quality monitoring studies

4.8        Dr Ken Mulvaney, an archaeologist and heritage expert, commented on the limitations of the air quality monitoring studies conducted by CSIRO in 2004–2005 and 2007–2008. He noted that the studies were conducted before the Woodside Pluto LNG plant went into production and at that time when the Yara Pilbara liquid ammonia plant was not in full production.[6] Dr Mulvaney told the committee that:

When those studies were done there were intermittent activities occurring, but the two main companies at the time were the Karratha gas plant and Rio Tinto's Hammersley Iron port facilities. They were the two main places. That study stopped in 2008. But during that time what was known as the Burrup fertiliser plant, which Yara now controls, was constructed. However, for various reasons, it was not in full production. In fact, for one time it was out for a good while, and that was before the Pluto gas plant development had gone ahead.[7]

4.9        Dr Mulvaney concluded that 'it is unlikely that there is an accurate capture of the total pollution load from existing industrial activities'.[8]

4.10      Dr Mulvaney expressed concern that as a result of these studies, the state government and BRAMMC had promoted that 'the industrialised areas on the Burrup Peninsula have considerably lower concentrations of air pollutants than cities in Australia'. Dr Mulvaney noted that:

Considering that at the time of these studies, the Karratha Gas Plant at Withnell Bay and the shipping of iron ore through King Bay, were the only resource industries in operation. Such levels of pollutants being on par with a two-four million population city; surely would raise alarm not complacency over rock art preservation. It may have been an independent committee, however with public statements like these, it raises concern as to whom within the committee may have had sway; the State Development Department perhaps. [9]

2007 Fumigation studies

4.11      Both Dr Mulvaney and Professor John Black were critical of the CSIRO report, Field studies of rock art appearance. Final Report: Fumigation and Dust Deposition. Progress Report: Colour Change and Spectral Mineralogy, published in March 2007.[10] Criticisms included poor experiment design such as an inadequate selection of rock samples and inadequate replication for statistical analysis.

4.12      Dr Mulvaney and Professor Black provided evidence on the implications for the outcomes of the studies of inadequate selection of rock samples. Dr Mulvaney firstly explained that the petroglyphs occur on a range of rock types on the Burrup Peninsula, and that they were produced using a variety of methods. Dr Mulvaney told the committee that:

...the art is produced wherever there is a surface expression of rock. There are a number of major geologies; basalt is one, there are volcanics as well out on the outer islands, in addition to the gabbro and granophyre. In each, the images are produced differently. One of the other features of the rock art of this place that makes it different to anywhere else is that the petroglyphs are produced in a wide range of techniques, and that partly reflects the rock. So some are hammered or pecked into the rock, some are abraded and others are just lightly scratched. Some are just, literally, bruising the rock surface. They are all reflective, in part, of the physical properties of the rock they have been produced on, but there are also clearly cultural aspects at play in the production of the art.[11]

4.13      Dr Mulvaney noted that the fumigation experiments were 'conducted on samples from a single gabbro rock with only a thin weathering rind' rather than on 'a range of lithologies known to have rock art (granophyre, dolerite and gabbro, nor on differing surface weathering states)'.[12] Dr Mulvaney concluded that 'it is problematic to confirm from such an inadequate study exactly what the effects of emissions are having on the rock art or what increased loads may cause'.[13]

4.14      Similarly, Professor Black stated that the study 'measured the effects of immersing iron ore in either dilute of concentrated organic compounds, acids or ammonia and measuring changes in colour and mineralogy'. Professor Black submitted that the results of this study 'have no relevance to rock art because the measurements were made on iron ore and not Burrup rock surfaces'.[14] Professor Black told the committee that these experiments:

...looked at acid, which was concentrated acid, and other organic compounds, concentrated or in a dilute sense, and they tested those on iron ore. It is completely irrelevant to what we are talking about, which is a test on the surface, the patina, of the rock art. It is completely useless for understanding anything about the impact on rock art because it was done on iron ore.[15]

4.15      CSIRO, in defending its use of iron ore samples, stated that it required suitable non-invasive methods which did not damage the rock art, and which had the approval of the Indigenous custodians of the land. As it was unable to directly test the rocks in the protected area, iron ore was selected as an appropriate proxy to examine discolouration as it 'contains a similar mineralogical profile to the rock patina' which has a major composition of hematite with minor goethite, quartz and kaolinite.[16]

4.16      Professor Black was also critical of another fumigation experiment in this study which involved 'hourly cyclical temperature and humidity changes during fumigation of Burrup rock samples with a combination of gases at two concentrations, with and without dust'. Professor Black stated that the study used 'gas concentrations below those projected for the ammonium nitrate plant and existing Burrup industry'.[17]

4.17      In addition, Professor Black was critical of the experiment design, stating that the 'study included either no treatment replication or insufficient replication for statistical analysis and was of no value for drawing conclusions'.[18] Professor Black explained to the committee that CSIRO:

...used emissions that they suspected would be 10 times what industry—the level of emissions is below what the companies are saying that it will be, so even the concentrations were not at a high enough level. But what was particularly non-scientific about it is that they did one set of experiments with dust and another experiment without dust, and they measured before and after these 30 days of going through cycles of temperature and humidity. But, because there was no replication for the dust and there was only one replication, and the values were quite different for the two replicates, you cannot analyse it statistically, so there could be no statistical analysis of it. My big criticism was that all of the claims were made without any statistical analysis.[19]

4.18      CSIRO, in response to these concerns, submitted that the fumigation experiment design was based on The Air Pollution Model (TAPM)[20] and CALPUFF[21] dispersion models provided by the Western Australian Government in the experiment tender document. It noted that the tender document had stated that 'the CALPUFF models are likely to be under-estimates and TAPM models are likely to over predict'. As a result, 'CSIRO tested the concentrations of the fumigant gases at 10 times the peak emission levels generated by the TAPM dispersion models.' It also noted that it 'is unaware of any information from industry that supports Professor Black's statement "...the level of emissions are below what the companies are saying that it will be"'.[22]

4.19      CSIRO also submitted that the dust experiments were 'performed using the accepted scientific approach to observing spectral change by difference and were designed with sufficient statistical power for the required analysis'. Specifically, the experiments included one case of dust exposure on two types of rock surfaces. In addition, 'on each of the eighteen samples of rock, replicate measurements were made at three different points, each separate point approximately 2mm in diameter'.[23]

4.20      CSIRO explained that the spectral comparison involved assessing the numerical differences between individual peaks that are normalised. It noted that 'the spectra is normalised to ensure that the differences measured are due differences in the sample rather than variable factors such as moisture.' Further, the spectral comparison is involved in overall spectral comparison to identify differences in peaks. It concluded that 'a statistical analysis of these kinds of results is not a necessary approach and spectral comparison is a widely accepted methodology'.[24]

2008 Burrup Peninsula air pollution study

4.21      A number of submitters expressed concern in relation to one of the conclusions reached in CSIRO's Burrup Peninsula Air Pollution Study: Report for 2004/2005 and 2007/2008, released in 2008. This study was designed to assess the likelihood that air pollution from the industrial area on the Burrup Peninsula may damage the petroglyphs found in the area and was authored by Dr Rob Gillett.

4.22      One of the conclusions of the study relied on a 1998 global assessment of ecosystem sensitivity to acidic deposition authored by Cinderby et al. The conclusion stated:

The critical load concept can be used to compare with deposition fluxes to determine if adverse effects could result to rock or aboriginal rock art. For a fuller discussion of this see Ayers et al. (2000). The critical load has been defined as "a quantitative estimate of an exposure to one or more pollutants below which significant harmful effects on specified elements of the environment do not occur according to our current knowledge" (Nilsson and Grennfelt, 1988). In a global assessment of ecosystem sensitivity to acidic deposition Cinderby et al. (1998) have determined a critical load or deposition flux of 25 meq m-2 yr-1 for the most sensitive areas of the world...In fact the assessment by Cinderby et al. (1988) lists 5 sensitivity classes consisting of 25 meq m-2 yr-1, 50 meq m-2 yr-1, 100 meq m-2 yr-1, 150 meq m-2 yr-1 , 200 meq m-2 yr-1 and >200 meq m-2 yr-1, and places the Burrup area in the least sensitive class. This means that the critical load for the Burrup area is at least 200 meq m-2 yr-1, and since this is significantly more than the observed deposition fluxes at the sites they are unlikely to cause any deleterious effects to rock or rock art on the Burrup Peninsula.[25]

4.23      This conclusion in particular was criticised by Professor Black, and Dr Johan Kuylenstierna, one of the authors of the Cinderby report. Dr Kuylenstierna submitted that Dr Gillett's assertion that the critical load for the Burrup area is at least 200 meq/m2/year is incorrect. Dr Kuylenstierna submitted that:

...the use of the Cinderby et al 1998 global sensitivity map and critical loads to say anything of relevance to the rock art in the Burrup Peninsula is just plain wrong—for many reasons and should not be used in evidence to the committee. It cannot be used by industry or governments to justify acid load emissions of 200 meq/m2/year. Rather a careful analysis of the rock art and its sensitivity to acidic inputs is needed.[26]

4.24      Dr Kuylenstierna explained to the committee that:

The maps which we developed were based on soil type and the idea was that, if you have something which has lots of minerals that can weather quickly, then the ecosystem will be safe. That is a different end point than the weathering of rocks with rock art and therefore it is not really relevant in this case. As I understand it, some detailed work on the impact on rock art is required rather than referring to what is a global assessment to give a broadbrush idea of what is sensitive to the ecosystems such as streams and lakes can be to acid rain.[27]

4.25      Dr Kuylenstierna went on to comment that the basis of the critical load assessment was soil type only and did not examine the characteristics of rocks. Dr Kuylenstierna noted that in most cases, soil type reflects parent material (i.e. rocks) however it can be significantly affected by weathering processes and the build-up of organic matter. Dr Kuylenstierna concluded:

But the main point is that the [critical load assessment] map does not directly reflect the rock type and therefore cannot be used to say anything about the rocks where the rock art is carved.[28]

4.26      In addition, the sensitivity referred to in the Cinderby et al report refers to the sensitivity of ecosystems (i.e. the vegetation or surface waters such as lakes and streams) and does not refer to the sensitivity of rocks to weathering. Dr Kuylenstierna explained that:

If anything the inverse is true, as more rapid weathering of minerals in the soil leads to better buffering and less damage to ecosystems – but the process would be more rapid weathering in these areas. Either way this is an inappropriate use of the critical loads – the rocks in a highly buffered region would weather faster.[29]

4.27      Dr Kuylenstierna noted that weathering processes are complex and specific to rock types. In order to determine how the surfaces of rocks on which art is carved would be affected by acidic inputs, Dr Kuylenstierna submitted that it is necessary to develop an understanding of the weathering processes of those specific rocks.[30]

4.28      Further, Dr Kuylenstierna stated that the scale of the global soil maps used in the Cinderby et al study was 1:5 million which shows broad patterns rather than local detail. Dr Kuylenstierna again reiterated that 'these are soil maps and not geology maps, and so still misses the point—the method is not based on an assessment of the geology'.[31]

4.29      Professor Black described the experiments conducted as part of the Gillett report as sound, noting that the data was well analysed and the report well written. However, Professor Black expressed concern that the report had utilised the findings of the Cinderby study to conclude that the rock art of the Burrup Peninsula would withstand the highest critical acid load on the international scale. Of especial concern was the fact that this conclusion had been reached without measuring the buffering capacity of Burrup rocks.[32]

4.30      Professor Black submitted that the total acid load emitted from the TANPF should be less than 25 meq/m2/year in order to protect the rock art of the Burrup Peninsula.[33] Professor Black stated that no measurements of critical acid load for rock patina on the Burrup Peninsula have been made because the buffering capacity of the rock surfaces has never been measured. Therefore, there is no empirical evidence for critical acid load for rock surfaces on the Burrup Peninsula. As such, an acid load of 25 meq/m2/year is based on comparisons of critical loads for other parent rock types and ecosystems.[34]

4.31      Professor Black particularly noted that the rocks of the Burrup Peninsula are igneous and formed under great pressure, which makes them extremely hard, and are amongst the slowest eroding rocks in the world. Consequently, little soil is formed where petroglyphs occurs and erosion of parent rocks is strongly related to buffering capacity. Professor Black submitted that the slower erosion rate of rocks on the Burrup Peninsula would create critical loads which are less than those for granite rocks.[35]

4.32      Professor Black also stated that scientific principles and empirical evidence shows that rock patina dissolution commences once pH falls into the acidic range and that the acidity of rock surfaces on the Burrup Peninsula are already in the strongly acid pH range of 4–5. As such, the total acid load emitted from the TANPF should be as low as possible.[36]

4.33      CSIRO responded to criticisms of the air pollution studies and submitted that measuring the acid load of the Burrup rocks was not in the scope of the work it was contracted to carry out. Rather, it was contracted to undertake independent air monitoring where it 'determined the total deposition of sulfur and nitrogen from the atmosphere by measuring sulfur and nitrogen compounds in samples of gases, particle and rainwater at several locations'. It then compared the data it measured as part of this work to other locations including similar sites in Malaysia and the Northern Territory.[37]

4.34      CSIRO, in defending its choice to utilise the Cinderby et al critical load framework, and the level of 200 microequivalents, stated that it was intended to provide context for the air monitoring data collected.[38] Dr Melita Keywood, Principal Research Scientist, CSIRO, told the committee:

As in any scientific study, when you produce information and data it is really important that the data and the use of that data be put into a context that the end user can understand. At the time that critical load framework was the best that we had available for us to put the data and information that we collected in context, and so that is what we used for that reference.[39]

4.35      CSIRO stated that 'the critical load framework of 200 microequivalents cannot be used as impact assessment criteria, and this was never the intention of the comparison'. CSIRO further noted that the Gillett report was 'peer reviewed by an independent international reviewer and [the Cinderby et al framework] was the best comparison to use at the time'.[40] Dr Helen Cleugh, Director, CSIRO Climate Science Centre, stated:

I would also remind the committee that the project design and the results have been published in a report that was peer reviewed and has been published in a peer-reviewed journal paper as well, which included the set-up of this design and this framework as well. As Dr Keywood said, it was the best available framework at the time and we have not been advised that there was a better approach that we could or should have used.[41]

Extreme weathering experiments – 2017

4.36      In 2017, CSIRO published Extreme weathering experiments on the Burrup Peninsula/Murujuga weathered gabbros and granophyres authored by Erick Ramanaidou, Gay Walton and Derek Winchester.[42]

4.37      The report was initially published in May 2017 by the WA Government. However, following a critique provided by Dr Ian MacLeod, Dr John Black, Dr Simon Diffey, and Dr Stephane Hoerle, the report was removed from the website of the Department of Environmental Regulation.[43]

4.38      Subsequently, an amended report was published. This report stated that:

This is a preliminary study using novel sample preparation methods to provide a new approach to determining the effects of solutions of different compositions and concentrations on rock weathering. As a scoping tool, it was very valuable in targeting future work. This study was conducted on 110 samples and the results found here should be confirmed using a larger dataset. It was not intended to serve as an exhaustive or definitive analysis of the impacts of the chosen leach solutions on granophyre and gabbro rocks nor was it intended as an indication for permissible pollution levels. The precautionary principle should apply here and emission capable of producing pH below 5.5 (the pH of rainwater) should be considered potentially harmful.[44]

4.39      Professor Black assessed this report as 'of little value for assessing the effect of nitric acid, sulphuric acid, ammonia or ammonium nitrate load on dissolution of rock surfaces or petroglyphs on Murujuga'. Professor Black was both critical of the conclusions reached in the report and the experiment design.[45]

Rock art monitoring 2004–2014

4.40      The committee received evidence critical of the regular independent monitoring of colour and spectral mineralogy of the Burrup rock art which was carried out by CSIRO from the program's inception in 2004. CSIRO prepared annual reports that compared the results of each year's monitoring program with the results collected in previous years.

4.41      Professor Black submitted that the authors of these reports 'claimed there had been no change in colour of background rock or engravings over the time of measurement without appropriate statistical analysis'.[46] Professor Black commented further that from 2004 until 2013, CSIRO 'were making claims of no colour change by not looking at the fundamental measurement that comes out of a spectrophotometer'.[47]

4.42      Professor Black explained that spectrophotometers measure:

...three components which we call colour space variables. One is L, which says how light it is, with zero the blackest black and 100 the whitest white. So it says: how light is it? The second one is A, which is the red-green opposing colours. If it is positive, it is red; if it is negative, it is green. The third one is B, which is the blue-yellow opposing colours. If it is positive, it is yellow; if it is negative; it is blue. They are the fundamental measurements. As a scientist, you would say you should measure and statistically analyse the change of those fundamental colours over time.[48]

4.43      However, Professor Black stated that CSIRO did not measure and statistically analyse the change in these fundamental colours. Professor Black submitted that instead, CSIRO measured and compared colour changes from one year to the next rather than from original measurements. Professor Black told the committee that:

What they did was to then take those colours and measure what was called colour change. That is another formula that you can use to say: how did that colour change from this point to that point? But what they did in those early publications was to say, 'Let me compare the change of this year with that year, and that year with the next year, and that year with the next year, but not the first year with the last year.' Because each year goes up and down a bit, they said, 'Well, there's no change,' but they did not ever do an analysis from the top to the bottom.[49]

4.44      Professor Black noted that the BRATWG had provided him with a copy of the 2013 report. An initial analysis of the report led Professor Black to state that:

...they needed to do a thorough statistical analysis and to do it over the whole period. And then they sent me back the one the next year, and they had done some statistical analyses, but they still had not done the fundamental statistical analysis of colour change across time. If that is not done, you are not getting the fundamental description of the data and what it means. And that is what we did when we reanalysed the data.[50]

4.45      Professor Black undertook to reanalyse the data collected by CSIRO and stated that 'CSIRO conclusions have been shown to be wrong'. Further, that 'an independent reviewer of the original reports and the data reanalysis report questions seriously the integrity of the CSIRO data'.[51]

4.46      Professor Black told the committee that the reanalysis was provided to the BRATWG through the Western Australian Government, and that CSIRO reviewed and provided comments on the reanalysis. Professor Black stated that 'CSIRO said that the statistical model that we had used was not the best model'. In response Professor Black requested that CSIRO provide a model for use in the reanalysis. Professor Black stated:

...of course with statistics you can have different models, so we asked them to provide us with the model that they would like us to use. Unfortunately, they never ever sent us a model that we could use. We then had the meeting with BRATWG, at which CSIRO were present. At that meeting, there was a question about whether all of the changes were in a similar direction, and the committee asked us to work with CSIRO to establish a statistical model and to prepare a paper for publication for refereeing.[52]

4.47      Professor Black noted that the reanalysis work was only conducted after he signed a confidentiality agreement with the Western Australian Government and that he was been prevented from sending this work to a peer-review journal.[53]

4.48      Professor Black concluded that 'the scientists involved in studies initially accepted the errors identified, but refused to acknowledge them after consultation within CSIRO'. Further that CSIRO 'appears to be more concerned about its reputation than the fate of the world significant archaeological heritage of Burrup rock art'.[54]

Independent review process

4.49      CSIRO's monitoring and analysis work has been reviewed a number of times: first by Professor Black and co-authors, and then by Data Analysis Australia as requested by the Western Australian Government. This section will outline some of the key findings of this review process and the implementation of recommendations.

Data Analysis Australia—2016 review

4.50      In 2016, Professor Black and Dr Simon Diffey conducted an analysis of the CSIRO monitoring program. This analysis resulted in a draft paper[55] (henceforth called the Draft Paper) which suggested that significant changes had taken place in the in the rock art of the Burrup Peninsula. This was in contrast to the findings of the CSIRO reports. As a result, the then Western Australian Department of Environment Regulation (DER) engaged Data Analysis Australia (DAA) to review the statistical issues raised in the Draft Paper, utilising the data itself, and CSIRO reports.[56]

4.51      DAA found that the statistical methods utilised by Black et al in the Draft Paper to be 'highly appropriate (with some minor modifications) and they represent[ed] a substantial step forward in effective monitoring of the Burrup Peninsula rock art sites'. However, the DAA also concluded that the analysis could not 'overcome the lack of confidence' in the data utilised and that it would not be appropriate for the Draft Paper to be published in its form at the time of the review.[57]

4.52      DAA noted that the Draft Paper utilised a significantly different approach to the analysis of monitoring data than that utilised by CSIRO and that this approach 'should be been used for some years'. DAA stated that the approach taken in the Draft Paper 'provides the opportunity to examine longer term trends, to understand whether there are issues affecting multiple sites and to potentially contrast sites close to and far from the industrial developments'. In doing so, Black et al were able to highlight a number of inadequacies in the CSIRO reports—particularly the absence of proper statistical analysis in earlier reports. DAA expressed regret that the Draft Paper was affected by the problems with the data provided to the authors. [58]

4.53      DAA's review of the CSIRO reports and data also highlighted 'significant problems of cross-calibration between instruments, inconsistent error-prone data management, and clear errors in the data'.[59] It stated that although the twelve years of data collected by the CSIRO are a valuable resource that should not be discarded, 'it is not appropriate for any decisions—including whether or not changes have taken place on the Burrup Peninsula—to be based on it in its current form'.[60]

4.54      DAA made a number of recommendations as a result of its analysis of the Draft Paper, and CSIRO data and reports. These recommendations were as follows:

  1. The historical data collected by the CSIRO should be systematically archived and held by DER, with consistent naming conventions, both to provide a baseline record and to facilitate comparisons with future data. The archival data format should enable ready access to the data via standard statistical software such as R1.
  2. The CSIRO should be asked to revisit the cross calibration issues with the BYK-Gardner (BYK) portable spectrophotometer and the Konica Minolta (KM) spectrophotometer, both to ensure that the historical data is properly understood and to confirm whether or not the historical BYK data is capable of comparison with current and future measurement instruments.
  3. An analysis similar to that of Black and Diffey should be conducted using verified ASD estimates of L*, a*, b* ideally using the original ASD spectra rather than the averaged spectra.
  4. The publication of the Black and Diffey paper should ideally wait until the problems with the BYK data are resolved or should use the ASD data.
  5. Future work by the CSIRO should be based upon an agreed analysis plan certified by a competent statistician. Since each year the CSIRO Reports have covered the full data set since 2004, it would be appropriate for the next published Report to incorporate this improved analysis and in doing so, make it clear that it should replace the analyses in their previous Reports.
  6. Consideration should be given to expanding the number of measured sites and in doing so, improving the balance of the design to include more effective controls, if feasible.
  7. To maintain scientific rigour, future data collection should follow a fully documented and detailed protocol, and ensure that departures are documented.[61]

Data Analysis Australia—2017 review

4.55      In 2017, DAA was requested to review a CSIRO draft report, Burrup Peninsula Aboriginal Petroglyphs: Colour Change & Spectral Mineralogy 2004–2016 (CSIRO Draft Report), authored by Noel Duffy, Erick Ramanaidou, David Alexander and Deborah Lau. The CSIRO Draft Report covered the data collection and analysis conducted since 2004 as part of the Burrup rock art monitoring program, with a focus on the possible effects of industrial developments. The CSIRO Draft Report represents the latest in a number of reports developed by the same CSIRO group that has presented earlier data from the monitoring program.[62]

4.56      DAA noted that the contract for the 2016 monitoring program required CSIRO to address the recommendations of the 2016 DAA review. As such, the CSIRO Draft Report aimed to address the shortcomings of earlier work highlighted by the 2016 DAA review.[63]

4.57      The 2017 DAA review found that:

...a considerable amount of work has been done to address some of the concerns. In particular there have been substantial improvements to the statistical analysis of colour changes using linear mixed models and greater care has been taken to highlight the problems associated with the BYK spectrophotometer used in the early years of the monitoring program. There also appears to have been action taken to better manage the data, both to make it available for analysis and to preserve it for future years.[64]

4.58      However, DAA also concluded that 'significant work remains if the 2016 Recommendations are to be addressed'. It noted that:

  • Recommendations 4, 5 and 6 had not been met;
  • Recommendation 2 was not addressed;
  • Recommendation 3 was partially met; and
  • Recommendation 1 was largely met.[65]

4.59      DAA recommended that if this work cannot be completed for the CSIRO Draft Report then 'it should at the very least be highlighted as work in progress so the reader is not given to think that the Draft Report is complete or its conclusions final'.[66]

4.60      DAA also made a number of observations in relation to the report:

  • 'The use of the BYK data is highly problematic' and 'it is a reasonable statement that little if any scientific weight can be given to it'. Further, 'this needs to be made more prominent, and indeed the right solution is probably to assign the BYK data to a historical note'.[67]
  • The ASD spectrograph data and its derived colour measures have been collected with reasonable consistency with one instrument since 2004, though there are some concerns with the 2004 data. The Draft Report gives prominence to the spectra from this instrument and to the spectral parameters, but little attention is given to the ASD colour measurements.[68]
  • However, the presentation of the ASD spectral parameters via 'numerous small barely readable plots' is 'not particularly helpful'. Further, no statistical analysis is conducted. As such, 'there is little purpose presenting them as done in the Draft Report and no purpose if they are not going to be statistically analysed'.[69]
  • The application of linear mixed models to the ASD colour data should be commended, and is a 'marked improvement' on previous reports. However, the presentation of the models is unclear. Further, the most basic test of whether the contrast between engravings and their backgrounds are changing at different rates depending on whether they are situated closer industry was not included.[70]

4.61      CSIRO's Draft Report concluded that 'the data is scarcely unequivocal and there are reservations on the conclusions of the statistical analysis', however DAA considered that this statement 'could be considered misleading' as it gives the impression that the data is incapable of giving clarity 'whereas a more thorough statistical analysis may be able to resolve the question more completely'.[71]

4.62      DAA was also critical of the design of the monitoring program and stated that:

It is unfortunate that, for whatever the reasons, this was not based upon firmer statistical principles. More sites should have been monitored, especially more control sites and the number of replicate measurements taken at each point seems excessive (or unnecessary). Furthermore, as there are concerns that the measurement process is damaging the engravings, a fractional design is indicated where not all spots were measured each year. It is not possible to fix the historically collected data but moving forward consideration should be given to redesigning the monitoring scheme.[72]

4.63      DAA concluded that:

...we are of the opinion that while the Draft Report demonstrates substantial efforts on the part of the CSIRO to improve the reporting of the data collection and to present better analysis, more needs to be done. In particular, in its current form the Draft Report is unable to dispel what might be described as reasonable concerns about the impact of industry on the rock art.[73]

CSIRO 2004–2016 report

4.64      As a result of the 2017 DAA Review, a number of changes were made to the CSIRO Draft Report. The final version, the Burrup Peninsula Aboriginal Petroglyphs: Colour Change & Spectral Mineralogy 2004–2016 (Final Report) was released by the Western Australian Government in September 2017.

4.65      Dr John Steele, Director, Science Impact and Policy, CSIRO explained to the committee that in effect, CSIRO had 'used the DAA commentary as a peer review for the purposes of producing a final report'. Dr Steele also noted that its Final Report supersedes all prior analysis done by CSIRO as part of the rock art monitoring program.[74]

2016 DAA Review recommendations

4.66      The following section outlines the ways in which CSIRO responded to each of the 2016 DAA Review recommendations in its Final Report.

Recommendation 1 – archiving of data

4.67      CSIRO noted that it has fully implemented DAA's recommendation to systematically archive the historical data. CSIRO's Final Report stated that:

All the historical data collected for the all the spectrometers have been systematically archived and were sent to DER with consistent naming conventions, in a data format that is easily read by standard statistical software.[75]

Recommendation 2 – cross-calibration issues

4.68      CSIRO submitted that the recommendation to revisit cross-calibration issues with the BYK and KM spectrophotometers was no longer relevant in light of its analysis of the data. As such, CSIRO's Final Report concluded that:

...as data from the BYK spectrophotometer appears unreliable for drawing conclusions on colour change in the rock art, the cross calibration issues with the BYK – Gardner (BYK) portable photospectrometer and the Konica Minolta (KM) photospectrometer will not be undertaken. All the photospectrometer data have been provided to DER for safekeeping.[76]

Recommendation 3 – analysis using verified ASD estimates of L*,a*,b*

4.69      CSIRO submitted that the Final Report undertook an analysis using verified ASD estimates of L*,a*,b* as recommended by DAA. CSIRO's Final Report stated:

For this report, combining the last two years of measurements (2015 and 2016), a complete statistical analyses of all the data (each individual measurement for the three instruments, a total of 24,000 colour measurements from 2004 to 2016) has been undertaken.

Measurement of the annual colour changes used two spectrophotometer techniques, the ASD and the BYK and KM. An examination of the colour measurements as a function of time, as well as a comparison of the two measurement techniques, has been conducted.

For both the KM and the ASD instruments, three-dimensional L*a*b* colour space (L* - degree of lightness, a* - degree of red/green, b* - degree of yellow/blue), identifying a tristimulus value (L*a*b*) for each sample point have been calculated.[77]

4.70      CSIRO also explained that CSIRO's Final Report included other models recommended by DAA. Further, the change in measurement practice for the ASD spectrophotometer (replacing it for each measurement from 2015 onwards) was documented and included in each of these analyses.[78]

Recommendation 4 – agreed analysis plan for future work

4.71      CSIRO agreed that statistical analysis should be a key part of planning for future analysis work, whether this work is conducted by CSIRO or other organisations. It noted that statistical analysis was only one of a number of technical issues that should be included in a plan for future analysis and that a number of technical practicalities would need to be taken into account. CSIRO's Final Report noted that for all future work it is recommended that:

A complete statistical analyses is done on the full spectrum of each individual ASD spectrum (not just the visible part i.e. L*, a* and b*).[79]

Recommendation 5 – expanding the number of sites

4.72      CSIRO agreed that consideration should be given to expanding the number of measurement sites and in doing so, improving the balance of the design to include more effective controls. CSIRO's Final Report recommended that for future work:

A study be conducted to assess how many new sites and how many new engravings and backgrounds should be added to the current locations to increase the quality of the monitoring in the Burrup Peninsula. In particular, new control sites with similar rock types should be added to the current ones (for instance Depuch Island). It should also be noted that by increasing the number of independent measurement on each spot (in doing so improving statistical analysis) could also have an adverse effect on the petroglyphs. There were signs in 2015 and 2016 that instruments measurements might be affecting the measured spots. A balance should be found between statistical endeavour and petroglyph protection.[80]

Recommendation 6 – data collection should follow a protocol

4.73      CSIRO noted and agreed with the DAA recommendation that in order to maintain scientific rigour, future data collection should follow a fully documented and detailed protocol. It submitted that such protocols 'will continue to be important, including for future analysis work (whether to be conducted by CSIRO or other organisations)'.[81]

4.74      CSIRO's Final Report included commentary on the protocols used by CSIRO during collection.[82]

2017 DAA Review recommendations

4.75      The following section outlines the ways in which CSIRO responded to each of the 2017 DAA Review recommendations in its Final Report.

Recommendation 1 – succinct description of measurement framework

4.76      CSIRO implemented the recommendation to include a succinct description of the measurement framework used. The Final Report included both detailed descriptions of the measurement framework, detailed instrument information, and a succinct description of aspects of the study relevant to the statistical analysis.[83]

Recommendation 2 – address issue of poor quality of BYK data

4.77      In accordance with both the recommendations in the 2016 and 2017 DAA reviews, CSIRO directly addressed the issue of the poor quality of BYK data in both the Executive Study and the Conclusion. Further, BYK data was not used in the analysis of trends.[84]

Recommendation 3 – less reliance should be placed on the ΔE measure

4.78      CSIRO noted that the report's conclusions are not based on the ΔE measure but rather on the statistical analysis of individual colour components (L*a*b*).[85]

Recommendation 4 – need for a proper statistical analysis of spectral parameters

4.79      CSIRO's Final Report extended its statistical analyses in order to test whether there have been any changes in colour over time, and whether these changes are at different rates at sites near to or far from industry, and whether the difference applies equally to background rock and engravings.[86]

Recommendation 5 – prominence of findings regarding the BYK data

4.80      As noted above, CSIRO gave prominence to the findings that the BYK data has limited if any value in both the Executive Study and the Conclusion.[87]

Recommendation 6 – comments regarding BYK data and colour change

4.81      CSIRO explained that the recommendation that comments that the BYK data does not indicate change should be deleted was in error. Rather, the Draft Report had noted that the BYK data had not indicated a different rate of change between the northern and southern sites. However, in light of the recommendation, CSIRO added stronger caveats into the Final Report, including those which identify the aforementioned issues with the BYK data.[88]

Recommendation 7 ­ more information on statistical models

4.82      CSIRO noted that further information on statistical models was included in the Final Report in order that the models can be fully replicated by anyone with the access to the data.[89]

Recommendation 8 – proper documentation of measurement practices

4.83      CSIRO noted that all changes in measurement practices were fully documented in the Final Report as recommended by DAA. Further these were also incorporated into analyses.[90]

Recommendation 9 and 10 – formal design and analysis plans

4.84      CSIRO noted that Recommendation 9 and 10 were recommendations for the next period of data collection rather than the current report. These recommendations were that a formal design document and a formal analysis document be developed prior to the next period of data collection. CSIRO noted these recommendations.[91]

Monitoring and conclusions

4.85      CSIRO's Final Report concluded that the monitoring undertaken of the rock art indicated that there has been some small but statistically significant change to the rocks in some dimensions of colour. It found that:

For both the KM and the ASD instruments, three-dimensional L*a*b* colour space (L* - degree of lightness, a* - degree of red/green, b* - degree of yellow/blue), identifying a tristimulus value (L*a*b*) for each sample point have been calculated.

Data from the KM spectrophotometer shows a trend over time in the L* measurements. The lightness (L) decreasing at a modelled average rate of 0.31 units per year (a total decrease of about 2 units on this scale is just noticeable to the human eye). However no trend is indicated in either a* (degree of red/green) or b* (degree of yellow/blue).

Data from the ASD spectrometer shows trends indicated in L* (degree of lightness) and a* (degree of red/green) but not on b* (degree of yellow/blue), though the evidence is not as strong as with the KM instrument.[92]

4.86      However, importantly, it noted that the results are not fully conclusive but are nonetheless important and warrant further attention. Further, none of the instruments demonstrate a difference in the rate of change between control sites and those closer to industry. The report stated:

The results are not fully conclusive and if the measurements do reflect real colour change, as the data suggest, then continued observations would continue to mark out the trend more clearly; and if not, observations will likely continue to fluctuate over time, making the randomness of the recorded variation more apparent...Nonetheless, the indication of significant colour change is important, and warrants closer attention. None of the instruments demonstrates a difference in the rate of change between the northern control sites and the southern sites closer to industry.[93]

4.87      CSIRO noted that the report does not explicitly address the reasons for the colour changes and the possible reasons for such small changes could include natural weathering. CSIRO stated that the report:

...does not provide a basis to confirm or to exclude an attribution to the industrial development, other than to note that the measured changes are not statistically significantly different at sites near to or far from industry.[94]

Critique of report

4.88      Following the publication of Burrup Peninsula Aboriginal Petroglyphs: Colour Change & Spectral Mineralogy 2004–2016, Professor John Black provided the committee a critique of the report.

4.89      Professor Black argued that the report includes an 'important admission to substantial errors in analysis and interpretation of all previous reports'. Of particular concern was that:

...these reports have been used by the Western Australian and Federal governments and industry to place the ammonium nitrate production facility in the midst of the rock art and to justify its high levels of emissions.[95]

4.90      Professor Black noted that the results indicated a colour change of approximately 13 per cent over the past 13 years, and that this represents a major change which should be of concern in the preservation of rock art. Further, statistical analyses demonstrate 'significant changes in lightness of the rocks' and both the KM and ASK instruments found significant changes in colour despite the high variance in measurements from year to year.[96]

4.91      However, Professor Black argued that the CSIRO report attempts 'to diminish the value of the significant findings relating to colour changes and to changes in lightness of the rocks'.[97]

4.92      Professor Black noted that an improved design of experimental procedures would significantly reduce the year on year variation in measurements. Further, Professor Black maintained that the use of two sites located in the north of the Burrup Peninsula as control sites for monitoring colour change is inappropriate given the close proximity of ships entering and leaving the Dampier Port.[98]

Draft Burrup Rock Art Strategy

4.93      Following the work undertaken by DAA and the release of CSIRO's Final Report, the Western Australian Government released the Draft Burrup Rock Art Strategy (the Draft Strategy) in early September 2017 for public comment.

4.94      The Draft Strategy outlines a long-term framework for the management and protection of the Aboriginal rock art of the Burrup Peninsula. It acknowledges the concerns raised by Professor Black, and the work undertaken by DAA. It states that 'the framework in this strategy is intended to address the limitations of the past monitoring and analysis program'.[99]

4.95      The Draft Strategy proposes that 'improved monitoring of colour contrast and spectral mineralogy should be continued on an annual basis with review after five years'.[100] It states that the Western Australian Government will develop a revised method for the collection and analysis of data that incorporates the recommendations of the DAA reviews. The revised method will be based on a number of principles including:

  • research questions will be developed in consultation with key stakeholders;
  • equipment and procedures used for monitoring will be reviewed to ensure that they are best practice;
  • the number of measurement sites will be calculated to ensure that statistically significant conclusions can be reached from analysis of the data collected;
  • the sampling method and analysis will be reviewed at least every five years by experts who are independent of the key stakeholders;
  • data analysis will be certified by a suitably qualified statistician;
  • statistical analysis will support the examination of long-term trends to understand if there are issues affecting multiple sites, and to contrast sites situated near and far from pollutant emission sources; and
  • additional control sites away from all major sources of emissions including industry and shipping will be incorporated into the monitoring program to the greatest extent practicable. Where possible, it should also be possible to discern between both of these emission sources.[101]

4.96      The Draft Strategy proposes that data collection and analysis should be undertaken by separate parties, with the statisticians undertaking the analysis acquiring and maintaining an adequate understanding of the data collection processes and techniques. It states that the annual monitoring program will be based on a number of principles which detail how data should be stored, published and reviewed.[102]

Other studies

4.97      In addition to the program for the monitoring of the rock art, the Draft Strategy makes a number of recommendations for other studies which will assist in protecting the Aboriginal rock art of the Burrup Peninsula.

Acid deposition

4.98      The Draft Strategy notes the evidence given by Dr Kuylenstierna at the committee's hearing on 17 February 2017 that the Cinderby et al report is not relevant to understanding the sensitivity the rocks of the Burrup Peninsula to acid deposition. As such, the Draft Strategy recommends that a better understanding of: the sources of pollutants; the current and likely future pollutant load; and the impact of pollutants on the rock art is required.[103]

Air quality

4.99      The Draft Strategy describes current air and meteorological monitoring on the Burrup Peninsula as 'reliable and targeted' but notes that 'improvement would inform a detailed cumulative spatial analysis'. As such, the Draft Strategy recommends the introduction of a long-term and coordinated monitoring network across all industries to expand the knowledge base required to manage the air quality in the region. It recommends that the network should measure exposure of the rock art to air pollutants.[104]

pH

4.100         The Draft Strategy states that regular measurements of the pH of the surface of gabbro and granophyre rocks on the Burrup Peninsula would assist in the early detection of conditions that would impact the rock art. It recommends the installation of monitoring stations including rainwater gauges to measure rainfall, pH, cations and anions as well as deposition flux of nitrogen and sulfur.[105]

Microbiology

4.101         The Draft Strategy recognises the potential impact that microbial action may have on the weathering of rock art. Noting the expansion of industry on the Burrup Peninsula in the years since the last study was conducted, the Draft Strategy recommends that a study to assess microbiological numbers and composition would be valuable, particularly as the TANPF becomes operational. The Draft Strategy also recommends that this study should be repeated from time to time to ensure that knowledge of microorganisms present on the rocks of the Burrup Peninsula is up-to-date.[106]

Source of pollutants

4.102         The Draft Strategy recommends that monitoring to measure levels of pollutants at particular sites should be conducted to enable a determination of the source of the pollution, and link any changes in the condition of the rock art to critical loads for pollutants and their source (industry, shipping, or other). The Draft Strategy also recommends that other causes of change to rock art such as guano should also be investigated.[107]

Burrup Rock Art Stakeholder Reference Group

4.103         The Draft Strategy includes the terms of reference for a newly established consultative committee called the Burrup Rock Art Stakeholders Reference Group (BRASTRG).[108] This group will assist in overseeing the design and implementation of the strategy, and includes representatives from the Murujuga Aboriginal Corporation, state agencies, local government, industry and the community.[109]

4.104         The terms of reference note that the role of the BRASTRG is to consult, inform and educate other stakeholders on matters referred for input or comment by the Western Australian Department of Water and Environmental Regulation. The BRASTRG will also contribute constructively to the monitoring and management of the rock art.[110]

Navigation: Previous Page | Contents | Next Page