2018-19 Major Projects Report
Entities examined: Department of Defence
Introduction
2.1
The Major Projects Report (MPR) is compiled by the Department of Defence (Defence) and the Australian National Audit Office (ANAO), comprising a selection of large-scale projects managed by Defence, known as Major Projects. The MPR reports on the performance of selected major defence equipment acquisition projects in accordance with the Joint Committee of Public Accounts and Audit 2018-19 Major Projects Report Guidelines (the Guidelines) as endorsed by the Joint Committee of Public Accounts and Audit (JCPAA, the Committee). The summary project data of all projects covered by the Guidelines is prepared by Defence and reviewed by the ANAO. The Auditor-General is required to provide an Independent Assurance Report to ensure that the information provided by Defence is correct and in accordance with the Guidelines.
Objective of the Major Projects Report
2.2
It has been the long-standing view of the Committee that the MPR’s objective is to improve the accountability and transparency of Defence acquisitions for the benefit of Parliament and other stakeholders.
2.3
Defence has interpreted this objective as being to provide ‘transparency on the progress of Defence’s largest and [most] complex acquisition projects [and] inform parliament and the public on Defence capability and related expenditure’.
2.4
Comparatively, the objective of the MPR is stated by the ANAO to be to ‘provide the Auditor-General’s independent assurance over the status of the selected Major Projects’. Assurance reviews are conducted in accordance with the standards set under section 24 of the Auditor-General Act 1997 (Cth) (the Auditor-General Act), with the Auditor-General providing assurance over the status of the Major Projects by examining the Statement by the Secretary of Defence and the PDSSs.
2.5
The Committee notes the functional objective of the ANAO in providing independent assurance of Major Projects, although emphasises the overarching objective of the MPR is to improve the accountability and transparency of Defence acquisitions for the benefit of Parliament and the public.
Background of the Major Projects Report
2.6
The MPR was established in order to provide information on an annual basis in relation to the performance of major projects in the Defence portfolio, including details on cost, scheduling and technical aspects of each project under scrutiny. The measures of cost, scheduling and technical attributes will be explained and explored in detail later in this Chapter.
2.7
Prior to the introduction of the MPR, the Parliament and key defence stakeholders had expressed concern that defence projects remained ‘largely invisible’ to the Parliament and to the public, requiring a significant improvement in ‘both the quantity and quality of information available to interested parties and to the public generally’.
2.8
The MPR was first recommended by the Senate Foreign Affairs, Defence and Trade References Committee in 2003, after conducting an inquiry examining Defence materiel acquisition and management. It was further recommended that the MPR be modelled on the substantive report produced by the UK Comptroller and Auditor-General on the progress of major defence projects for the British House of Commons.
2.9
The JCPAA agreed in December 2006 that the ANAO should receive additional funding to complete the MPR, and published a report in August 2008 which included an outline of measures that should be included in the report. The MPR contains this information, in addition to: Project Data Summary Sheets (PDSSs), which provide detailed information about individual projects; commentary by Defence; and a limited assurance review by the Auditor-General.
2.10
The ANAO provides the JCPAA with an MPR Work Plan annually prior to beginning work on the MPR. This Work Plan informs the Guidelines which are agreed to by the JCPAA, and direct the remit of the ANAO’s investigations. The Guidelines incorporate learnings from prior MPR findings and matters which require further examination. The Guidelines are provided at Part 4 of the 2018-19 MPR.
2.11
The 2018-19 MPR is the twelfth report prepared by Defence and the ANAO. The MPR is automatically referred to the JCPAA in accordance with its statutory obligations to examine all Auditor-General reports tabled in each House of Parliament. This is the tenth occasion on which the JCPAA has reported on the MPR.
2.12
The MPR has been classified as a priority assurance review under the definition of subsection 19A(5) of the Auditor-General Act. This provides the ANAO with full evidentiary-sourcing powers available to them under this Act. The review of PDSSs is performed in accordance with auditing standards under section 24 of the Auditor-General Act.
Conduct of the audit
2.13
The 2018-19 MPR reports on 26 projects which qualify under the Guidelines. In September 2019, the JCPAA endorsed project selection for the 2018-19 MPR, including the entry of five new projects. The ANAO advised that it proposed to consider the exit of seven projects from the MPR that were expected to reach Final Operating Capability (FOC) by the end of 2019, or to confirm that only low risk deliverables were remaining. Six of these projects are now considered suitable to exit.
2.14
In addition to the formal independent assurance process, the ANAO examined issues arising in the PDSSs such as ‘cost, schedule, progress towards delivery of required capability, project maturity, and risks and issues’. In relation to its assessment of the individual PDSSs, the ANAO analysed additional factors such as: governance and oversight matters, including for financial, risk and project status management; input by industry contractors; and the Statement by the Secretary of Defence, including significant events occurring post 30 June 2019.
2.15
The MPR was conducted in accordance with the ANAO Auditing Standards. Certain forecast information is excluded from the scope of the ANAO’s review, such as: capability delivery; risk issues; and forecast dates. These exclusions are due to a ‘lack of Defence systems from which to provide complete and accurate evidence’. As a result, the Independent Assurance Report cannot provide assurance over these issues. The ANAO is additionally limited in its findings by the MPR process not allowing for the same level of assurance or investigation as is conducted in the ANAO’s performance and financial statement audits.
2.16
The costs of producing the MPR are shared between the ANAO and Defence, costing the agencies approximately $2.2 million and $2.4 million respectively.
Overview of findings
2.17
The Acting Secretary of Defence stated in the Statement by the Secretary of Defence that, in her opinion, the 26 PDSSs ‘comply in all material respects with the Guidelines and reflect the status of projects as at 30 June 2019’. The Secretary also provides information regarding significant events that have occurred after 30 June 2019, which potentially impact the projects and PDSSs. Defence did not report any projects removed from the MPR which still had outstanding caveats.
2.18
While Defence have not defined the term ‘caveat’, the ANAO has observed that it has typically been used to represent exceptions to the achievement of significant milestones declared by Defence.
2.19
In the Independent Assurance Report, the Auditor-General stated that:
[N]othing has come to my attention that causes me to believe that the information provided in the 26 [PDSSs] and the Statement by the Secretary of Defence, excluding the forecast information, has not been prepared in all material respects in accordance with … the Guidelines.
Observations by the ANAO
2.20
The Auditor-General observed a number of issues that were examined during the course of the MPR, including: point-in-time information in relation to staffing and costs per project; occasions of compliance and risk management issues; guidance materials being out of date regarding the Project Maturity framework; and Major Projects reporting that significant milestones have been met with caveats.
2.21
The ANAO did not make any recommendations to Defence on the basis of these observations. The Auditor-General explained that any recommendations that ANAO make as part of the MPR:
[A]re less about the individual projects and more about how they could improve the accuracy of their reporting. What we’ve tended to do there is make commentary about things like risk and other matters… There tend to be no recommendations in the report itself, but during the process of the audit there’s always engagement between the auditors and the auditees about how things could be improved in their internal processes to get more accurate, effective reporting done.
Committee comment
2.22
In the interests of transparency and accountability, the Committee is of the view that any recommendations or considerations made by the ANAO to Defence in the course of preparing the MPR should be contained with the MPR. Reporting of any recommendations or considerations will further the MPR’s objective of improving accountability and transparency of Defence acquisitions for the benefit of Parliament and the public.
2.23
In its report on the 2007-08 MPR, the JCPAA stated that the Committee’s review ‘reflects the commitment made by the Committee to continuously monitor Defence’s acquisition processes and outcomes and to provide input and guidance where necessary’. The ANAO’s commentary and observations are intended to assist the Committee to fulfil this role. The JCPAA has reviewed the MPR and made recommendations as the primary user of the report.
2.24
The 2007-08 MPR was the inaugural report on Defence’s major acquisitions. Recommendation 1 of that report was that all future MPRs should contain a section clearly outlining the lessons learned on MPR projects which are ‘systemic and interrelated in nature’.
2.25
While this recommendation was made in the context of an additional section to the PDSSs (which is now Section 7), the Committee considers that a similar section should be included in Part 1 (ANAO Review and Analysis) of the MPR, containing any of the recommendations or key lessons learnt by the ANAO during the course of the audit. This section would improve procedural fairness in allowing Defence’s response to the ANAO’s recommendations to be incorporated into the MPR. This section would also provide a succinct and clear summary of the headline lessons to be taken from the MPR, similar to the ‘Key messages from this audit for all Australian Government entities’ section reported in performance audits.
2.26
The Committee recommends that the ANAO include a section in the MPR that clearly outlines any recommendations and/or key lessons learnt during the preparation of the MPR, which are systemic and interrelated in nature. This section is to build on the current summary of observations made in the course of the ANAO’s review. The section should contain lessons that can be incorporated into future policy and practice across the Department of Defence and other Australian Government entities. In the situation where there are no recommendations or key lessons, a short statement should be provided by the ANAO explaining the reasons for not making any recommendations or identifying any key lessons.
Outline of 2018-19 Major Projects
2.27
In the 2018-19 reporting period, 26 projects met the Guidelines for examination by the ANAO. A list of the projects and their approved budgets (including the total approved budget for all MPR projects) is provided at Figure 2.1. The total approved budget for the Major Projects for 2018-19 was approximately $64.1 billion, which equates to ’49 per cent of the total budget of active major and minor capital equipment projects of $132 billion’.
2.28
Four projects were added for the first time in the 2018-19 MPR:
SEA 1180 Phase 1 Offshore Patrol Vessel;
SEA 1439 Phase 5B2 Collins Class Communications and Electronic Warfare Improvement Program;
LAND 53 Phase 1BR Night Fighting Equipment Replacement; and
SEA 1448 Phase 4B ANZAC Air Search Radar Replacement.
2.29
The suitability for exit of the six reported projects was based on whether each project had either met the FOC requirements, would meet these requirements by December 2019 or had submitted a risk assessment confirming that FOC will be achieved shortly.
Projects of Interest and Projects of Concern
Projects of Interest
2.30
The Projects of Interest regime is targeted at projects with increased risks, such as risks regarding scheduling, costings, scope, capability and strategic value. Defence conducts monitoring processes, such as examining risk levels and consultation with stakeholders before adding to the Projects of Interest list. Projects listed are subject to additional reporting requirements and require that strategies be identified to improve the project’s management and return to the planned schedule.
2.31
Six MPR projects were considered Projects of Interest:
Joint Strike Fighter (JSF), due to risks relating to costing and certain deliverables;
LHD Ships, owing to risks including operational testing, certain requirements not yet being delivered, technical problems and a support system still under development;
Hawkei, as a result of issues such as capability and scheduling risks, the design process, and production problems due to the engine manufacturer going into voluntary administration;
Repl Replenishment Ships (Maritime Operational Support Capability), due to an amendment to delivery strategy which caused delays, and problems with the Integrated Logistics Support system;
Civil Military Air Traffic Management System, arising from schedule risks and contractual negotiations still being on foot; and
UHF SATCOM, owing to issues relating to contractual negotiations, technological progress and certification by the US Government.
Projects of Concern
2.32
The Projects of Concern regime is designed to focus the attention of Defence senior officials and ministers on particular projects which are experiencing problems in delivery. Projects listed on the Projects of Concern are subject to stronger oversight and management, and are also required to provide more extensive reporting to the Australian Government.
2.33
The ANAO reported that ‘[a]s at 30 June 2019, one MPR project, MRH90 Helicopters, was a continuing Project of Concern’. The Multi-Role Helicopter (AIR 9000 Phase 2, 4 and 6) (MRH) project has been considered a Project of Concern since November 2011 as a result of issues relating to technical issues managed by contractors which delayed progress towards major milestones. Defence are actively engaging with industry and scrutinising performance against product delivery through a number of forums as remedial action for these issues. However, despite being a Project of Concern, Defence also stated that the MRH90 Helicopters were deployed during the 2020 summer bushfires and performed well ‘under extremely adverse weather flying conditions’. Defence said that the reason the program remains a Project of Concern is due to the considerable amount of support needed to achieve the required flying rates.
2.34
A second Project of Concern, the Deployable Defence Air Traffic Management and Control System (AIR 5431, Phase 1), is not covered by the Guidelines and is thus not included for examination in the MPR.
2.35
Defence noted that Project of Concern classification provides ‘senior-level direction, guides and undertaking from industry as well as our own organisation to make sure we’re collectively focused in a partnership to deliver full capability’.
Box 2.1: Case Study: Multi-Role Helicopter (AIR 9000 Phase 2, 4 and 6)
The Multi-Role Helicopter (AIR 9000 Phase 2, 4 and 6) (MRH) Program aims to ‘rationalise the number of helicopter types’ in the Australian Defence Force (ADF). The MRH is an Australianised MOTS, 47 of which will be provided to the ADF and the Royal Australian Navy (RAN) through three phases. 12 MRH will make an additional squadron (Phase 2) for the Australian Army (Army), 28 MRH will replace the Army’s Black Hawk fleet (Phase 4), and six MRH will replace RAN Sea King Helicopters (Phase 6). The 47th MRH will be used for training purposes.
The MRH Program has experienced technical and reliability issues, requiring the Commonwealth to suspend the acceptance of aircraft from Airbus Australia Pacific (Airbus AP) in November 2010 and negotiate a remediation plan to address these issues. Acceptance of aircraft resumed in November 2011 after the Minster for Defence declared the MRH a ‘Project of Concern’ due to scheduling, technical deficiencies and Airbus AP’s performance.
Although all 47 MRH have been delivered and were accepted into service in July 2017, Final Operation Capability (FOC) will not be reached until December 2021 after the MRH are integrated into both the Army and RAN and are in full operational service. Final Materiel Release (FMR) is expected to be completed by June 2021. Both the Commonwealth and industry have provided in-service work to mitigate and rectify the capability shortfalls and implement contracted standards.
The MRH Program spent $104.8 million of the $133.7 million budget for 2018-19 reporting period, and has a current approved total budget of $3.771 billion. Defence states that ‘there is sufficient budget, including contingency remaining for the project to complete against the agreed scope’. Despite the underspend in the 2018-19 financial year, however, a contingency was applied ‘primarily for the treatment of various supportability and performance risks’.
Major risks and issues affecting the project include: aircraft system immaturity impacting the delivery schedule; technical and design issues; and a potential lack of suitably skilled expertise in industry to meet deadlines in a timely manner.
Defence held a Projects of Concern Summit on 3 December 2019, where the MRH Program was discussed. Further, as of 30 June 2019, testing of activities that are part of the Projects of Concern remediation plan had also been partially conducted by Defence.
2.36
Defence’s management of underperforming projects has been the subject of a recent ANAO audit. In conducting the audit, the ANAO sought to identify ‘whether Defence’s Projects of Concern regime is effective in managing the recovery of underperforming projects’. It concluded that:
While the Projects of Concern regime is an appropriate mechanism for escalating troubled projects to the attention of senior managers and ministers, Defence is not able to demonstrate the effectiveness of its regime in managing the recovery of underperforming projects.
2.37
In that report, the ANAO highlighted concerns such as: transparency issues, reduced adherence to formal criteria, reduced emphasis on established processes, and lack of consistency in Defence’s utilisation of the Projects of Concern regime.
2.38
That audit made two recommendations:
that ‘Defence introduce, as part of its formal policy and procedures, a consistent approach to managing entry to, and exit from, its Projects of Interest and Projects of Concern lists. This should reflect Defence’s risk appetite and be made consistent with the new Capability Acquisition and Sustainment Group Risk Model and other, Defence-wide, frameworks for managing risk. To aid transparency, the policy and the list should be made public; and
that Defence review the Projects of Concern process.
2.39
Defence agreed to these recommendations. In its response to the audit, however, Defence stated that it did not consider that the ANAO had ‘appropriately consider[ed] the evolving nature of the Projects of Concern regime, its role within the larger project management toolkit, and elevation of priority for attention by the Minister/Government of the day’. Defence further stated:
Defence does not agree with the ANAO’s statements inferring that it has avoided adding to the Projects of Concern list in the interest of trying to maintain a positive relationship with industry, nor has this resulted in a less robust governance arrangement. The reduction in the number of projects on the Project of Concern list is linked to the changing nature of the Capital Equipment Program as well as the close out of legacy projects.
2.40
The ANAO advised that Defence had taken a number of steps to address Recommendation 1 in the ANAO’s report, including developing a new Project Risk Management System which would enable uniformity for projects entering and exiting the Projects of Interest and Projects of Concern lists. Defence further advised that, in relation to Recommendation 2 of the ANAO’s report, it would review Projects of Concern to determine the cause of project delivery issues, and release its findings to inform future projects.
2.41
Further, in ANAO Report 31 (2018-19), Defence’s Management of its Projects of Concern, the Auditor-General recommended that:
Defence introduce, as part of its formal policy and procedures, a consistent approach to managing entry to, and exit from, its Projects of Interest and Projects of Concern lists. This should reflect Defence’s risk appetite and be made consistent with the new Capability Acquisition and Sustainment Group Risk Model and other, Defence-wide, frameworks for managing risk. To aid transparency, the policy and the list should be made public.
2.42
On 17 April 2020, Defence advised the Senate Standing Committee of Foreign Affairs, Defence and Trade that completion of its responses to this recommendation is scheduled for June 2020. The Committee did not receive evidence regarding these issues during the public hearings undertaken as part of this inquiry.
Schedule slippage
2.43
The ANAO reports schedule performance to identify how projects are tracking against key time metrics. These metrics include: time elapsed and project maturity; schedule slippage and acquisition type by approval date; in-year schedule performance; and schedule performance by year of entry to MPR.
2.44
The Project Management Institute provided the Committee with a general overview of the fundamentals of planning, tracking and managing projects:
In project management, the fundamental tools haven’t changed too much over the years. Having an effective Gantt chart or a project schedule that is very detailed and very accurate is extremely important, but it’s only as good as the people that use it.
2.45
Schedule slippage was 110 months lower across all MPR projects than reported in 2017-18, stated by the ANAO to be as a result of: removing completed projects with significant schedule slippage from the MPR, additional slippage during the reporting period, and the addition of new projects with schedule slippage.
2.46
A number of MPR projects were found to be contributing to schedule recovery, such as the JSF, the AWD Ships and the Growler. In total, these projects had recovered 35 months in scheduling. The ANAO, however, did not include this effect in its analysis ‘to portray the complete amount of slippage experienced by the Major Projects’.
2.47
In explaining the reasoning for schedule slippage, the ANAO stated that a full analysis of why a project was behind schedule ‘require[s] a deep analysis of project technical elements and a realistic assessment of the capacity of the private sector to deliver in the expected timeframe’.
2.48
Factors contributing to these projects’ schedule slippage that the ANAO identified included:
Whether the project is a developmental project or an Australianised Military-Off-The-Shelf (MOTS) acquisition;
Platform availability for certain projects, such as the Maritime Comms and Collins R&S projects which were impacted by docking schedule changes for other projects;
Delays to operational test and evaluation tasks; and
Issues relating to developmental content.
2.49
Defence explained that these projects experienced schedule slippage for a range of reasons, which were grouped into three categories: projects with unplanned real cost or scope variation, projects with subsequent government approvals, and projects experiencing slippage for other reasons.
2.50
More broadly, the ANAO explained that its longitudinal analysis indicated that ‘while the reasons for schedule slippage vary, it primarily reflects the underestimation of both the scope of and complexity of work, particularly for Australianised MOTS and developmental projects’. The ANAO found that older MPR projects tend to be more highly represented in schedule slippage, particularly where the projects in question are already late.
2.51
The ANAO stated that while the developmental MPR projects Hawkei and ANZAC Air Search Rader Repl do not demonstrate schedule slippage to their FOC dates yet, these projects had ‘experienced slippage to design reviews, test programs, or material release milestones’. Other recent projects, however, such as the Pacific Patrol Boat Repl and Night Fighting Equip Repl, had experienced minor slippage. The ANAO stated that:
This indicates that although developmental projects currently in the MPR are not reporting significantly more slippage to FOC than MOTS projects, developmental projects still appear to carry a higher level of technical risk.
2.52
Defence believed that the reporting of ‘total schedule slippage’ in the MPR could be improved to provide greater transparency in the information they provide to the Parliament and the public. Defence said that:
We fully support that the public should have full transparency on all schedules and on the cost issues. We were just seeking to make sure that it’s reflected in a way that is accurate. In the summation of the schedule, those projects are run in parallel; they’re not end-to-end projects. So the reporting, in the way it was described, reflected that.
2.53
The ANAO responded by stating:
Defence's position that, to get that analysis out of the information provided, you actually need to read it all and think about it. It's not something that jumps out obviously to you. People just taking the number and talking about the number isn't what I think it's there for. Like everything, you can take almost all information out of context and use it for a different purpose, but that is not the purpose that it's there for. I think it's more looking at the system as a whole rather than looking at what happened in one year or in one project for those aggregates.
2.54
To improve the insights gained by the Parliament and the public from the MPR, Defence suggested that:
A more practical measure may be to look at what schedule variance occurred between the last report and current report and why this occurred. A more informative data point could also be to look at whether any variance could be raced to identified risks in the previous report, to test the efficacy of identified mitigations of those risks.
Risk management
2.55
Major risks and issues are currently not captured within the formal scope of the Independent Assurance Report. The Auditor-General advised that this is due to Defence’s risk reporting systems not being in a state that supported ‘reliable auditing’. Matters that consist of a ‘material inconsistency’ contained in this information identified by the ANAO are incorporated into the scope of the MPR. Despite its exclusion from the formal scope of the Independent Assurance Report, the ANAO stated that risk management is one of the key aspects examined in the MPR.
2.56
The Auditor-General explained that risk is a key part of any project management, with both the ANAO and Defence having an objective of reporting risk at a better level and to a standard that would allow the ANAO to provide limited assurance on risks associated with projects in the MPR.
2.57
The PMI explained the importance of identifying risks to the project and strategies for mitigating them:
Good project management also involves risk assessment and contingency planning – that is, determining the level of risk of something going wrong… It’s key, even before the project has started, to really run a very developed risk management workshop with all of the stakeholders involved and identify these key risks.
2.58
However, a number of distinctions in the oversight of ‘mega-projects’ were also outlined, including regarding the management of risk. This included unique complexities related to cost and schedule risk.
2.59
The Committee acknowledges the importance of identifying and managing risk in projects. It is important that Defence allocate sufficient resources to address deficiencies previously identified in their handling and treatment of risk.
2.60
The ANAO details that Defence currently has a number of policies and differing implementation methods and forms of documentation across a broad group of work areas, such as group-levels, sub-group levels and project-levels. The MPR focusses on project-level risk management practices in order to provide assurance in relation to the PDSSs.
2.61
In May 2017, the Deputy Secretary of the Capability Acquisition and Sustainment Group (CASG) issued a directive to create a new risk management model to use within the broader Defence risk management framework. At the time of the MPR’s reporting, Defence had delivered two parts of the reform, while the third had been delayed for delivery until February 2020. The ANAO ‘observed that some projects chose not to review risk and issues management procedures until [the third] stage has been completed’. The ANAO stated that while it would maintain observance over the third reform part’s implementation, that it ‘will not be able to consider including risks and issues in the scope of the MPR until the reform is sufficiently progressed’.
2.62
The ANAO identified a number of concerns relating to risk management in project offices, including:
Inconsistent compliance with corporate policies and guidance material, which the ANAO stated was evident in 10 out of 26 projects in relation to currency validation of the Risk Management Plan in accordance with the Defence Project Risk Management Manual version 2.4;
Managing sustainment and acquisition risks together rather than separately, as identified in the JSF project; and
Issues in documenting, recording, and reporting risks and issues, including: varying quality in risk management logs and documentation, inadequate frequency in reviewing risk and issue logs, and inconsistency in issues being reported in risk software.
Definition of risk
2.63
The MPR Guidelines provide a high-level description of the risk framework used by Defence in examining project risks and issues. Defence explained that the approach taken to reporting risk has a multidimensional focus, taking into account scheduling, capability and cost risks.
2.64
When referring to high or extreme risk, Defence noted that defence acquisition is:
Exceptionally complex and it carries a high inherent level of risk because we’re trying to do it fast to provide the highest level of capability … [W]hat we’re seeking to do is put in a risk management framework and articulate to government and the public the raw risk, the initial risk, what the treatments are and then what the residual risk is.
Use of spreadsheets
2.65
The ANAO raised concerns regarding Defence’s use of spreadsheets as a form of risk reporting and administration. The ANAO explained its concerns regarding the use of spreadsheets in the administration of risk and issue management:
Spreadsheets lack formalised change/version control and reporting, thereby increasing the risk of error. This can make spreadsheets unreliable corporate data handling tools as accidental or deliberate changes can be made to formulae and data, without there being a record of when, by whom, and what changes were made. As a result, a significant amount of quality assurance is necessary to obtain confidence that spreadsheets are complete and accurate at 30 June, which is not an efficient approach.
2.66
The ANAO found that 15 project offices use spreadsheets as their main risk management tool, while others use a variety of other technological tools. Defence was reported to have advised the ANAO that ‘a risk management system will not be mandated until the outcomes of the CASG risk reform are known’. Table 2.1 contains information in regards to the primary method of risk reporting utilised by different projects:
Table 2.1: Risk management tools utilised by project offices
|
|
|
|
Microsoft Excel spreadsheets
|
15
|
|
Predict! Software
|
5
|
|
Microsoft Excel spreadsheets and Predict! Software (concurrent use)
|
1
|
|
Bespoke SharePoint-based tool
|
2
|
|
Microsoft Word
|
1
|
|
Project Performance Review
|
1
|
|
None used (due to delivery of all primary project elements)
|
1
|
Source: Australian National Audit Office, Report No. 19 (2019-20) 2018-19 Major Projects Report, p. 31.
2.67
This issue has been previously raised by the ANAO, and was examined by the JCPAA in a prior MPR review. The JCPAA agreed with both Defence and the ANAO the importance of data management in a risk management system, but agreed with the ANAO’s view that ‘a risk register requires appropriate version controls to reduce the risk of error’. The JCPAA recommended that Defence:
plans and reports a methodology to the Committee which shows how acquisition projects can transition from the use of spreadsheet risk registers to tools with better version control measures.
2.68
Defence agreed to the Recommendation, and advised in response that CASG was seeking to implement the Risk Reform Program, which would address the issues raised by the ANAO and the JCPAA, and was due to be implemented by the end of 2019. The Risk Reform Program included the development of a revised methodology for managing project risk, which Defence has stated will require a cultural and behavioural change to the way risk is currently managed in CASG. Defence further explained that while it had records management policies in place which could ‘partially’ resolve certain concerns regarding version control, that transitioning to the Risk Reform Program could take multiple annual cycles for all projects to be compliant.
2.69
The ANAO observed during site visits that project-specific guidance in relation to the new methodology had not been provided to Project Offices. Defence further advised that it was intending to rollout a standardised computing system tool to manage risks across all projects, which was expected to be completed by November 2019. The ANAO reported that it would examine this tool in the 2019-20 MPR.
2.70
Defence advised that Predict! Risk Analyser software was being utilised by 108 major and minor Defence projects, and a rollout was being prepared for the remaining projects. The rollout, which would include related products and training, was expected to be initiated by June 2020.
Projects achieving significant milestones with caveats
2.71
In the 2018-19 MPR, Defence reported a number of projects had achieved significant milestones with caveats, including:
The P-8A Poseidon, which achieved the Operational Capability 2 milestone in February 2019, but two caveats were declared regarding spares deficiencies and pilot simulator training; and
The Growler, which achieved the IOC milestone in February 2019, but one caveat was declared in relation to in-country aircrew training being delayed due to issues with the Mobile Threat Training Emitter System.
2.72
The ANAO also reported that two further instances of projects achieving milestones with caveats were identified after June 2019:
FOC status being granted for the LHD Ships with seven ‘notable deficiencies’ in November 2019; and
FMR achievement being reported for the Additional MRTT project with the inclusion of an ‘accepted deficiency’ in October 2019.
2.73
The ANAO expressed concern in the MPR that Defence was reporting projects meeting milestones with caveats at higher rates than in previous MPR reporting periods. The ANAO observed that Defence ‘has not defined the terms ‘caveat’ or ‘deficiency’ to the declaration of significant milestones in its internal policies and procedures’.
2.74
The ANAO stated that it would continue to monitor Defence’s use of declarations with caveats or deficiencies in future reports, including oversight of projects removed from the MPR with outstanding caveats which require continued reporting by Defence in the MPR by the Secretary of Defence.
Cost performance
2.75
The ANAO’s project performance analysis explores the cost performance for each project, which includes the percentage of budget expended, changes in budget since Second Pass Approval, in-year changes to budget, and in-year expenditure.
2.76
In the PDSSs, there is some variation regarding the basis for which the costings have been reported by Defence. For instance, the project budget is reported on an out-turned basis, whereas the value of major contracts for each project are reported at signature and at 30 June 2019, the latter of which was treated as the constant price.
2.77
During the public hearing on 20 May, Defence explained the key differences between constant and out-turned prices. Constant prices are based on:
[T]he buying power of the dollar at the time the contract is signed. That looks at the buying power of our Australian dollar and the exposure those different projects have to foreign currencies. In Defence, those foreign currencies are typically US dollars, euros or UK pounds. The variations, the mix between those, will be taken into account in that constant-dollar setting.
2.78
On the other hand, Defence explained out-turned prices as:
[W]hat we plan to spend in each year across the life of the project and then we apply a factor for inflation – that’s a weighted defence industry factor for inflation that sits just a tad under three per cent – and we apply the Treasury approved forecast for exchange rates. That varies year to year, depending on the currency mix that we’re planning on in each of those years. As you compound that out over many years, the number for an outturn figure will typically be larger than the number for the constant figure.
Committee comment
Projects of Interest and Projects of Concern
2.79
The Committee found that the ANAO and Defence were able to effectively contextualise the meaning of Projects of Interest and Projects of Concern during the public hearings, clarifying aspects of the two categories which were not apparent in the MPR.
2.80
As an example, a Project of Concern was described in the MPR as one which was identified as having:
technical, cost or schedule challenges that benefit from additional support from senior executives and Ministers. Projects are removed from the list through project remediation or project contract cancellation with the approval of the Ministers.
2.81
Defence confirmed during the public hearings that a Project of Concern could be restated as a ‘Project requiring additional oversight and management’. This rephrasing assisted Committee members in understanding the nature of the categorisation, which may not have been apparent to the Parliament and the public without further context.
2.82
The Committee considers that greater contextualisation of definitions and key terminology would be beneficial to the primary users of the MPR, including the Committee, by improving the accountability and transparency of Defence acquisitions.
2.83
On the Projects of Concern, the Committee observed that there were ongoing concerns in relation to the MRH90 Helicopters project. The Committee is of the view that Defence would benefit from a review being undertaken on their helicopter acquisitions, especially as Navy and Army acquire new helicopters as part of their fleet. This could be conducted by an independent auditor, the Auditor-General, or by a similar appropriately qualified entity.
2.84
The Committee recommends that Defence commission a performance review or independent external audit of the entire helicopter acquisition program in advance of upcoming helicopter acquisitions by Navy and Army.
Risk management
2.85
The Committee acknowledges the multifaceted nature of risk management reporting and that the risk is not currently contained within the scope of the MPR. However, the Committee believes the current description of major project risk in the Guidelines is circular in nature and does not clarify the threshold for which a risk is defined as high or extreme. Given the integral nature of these concepts in project management reporting, the Committee considers that a clearer explanation of risk would assist in achieving the objectives of the MPR, which is to provide transparency of Defence acquisitions for the benefit of Parliament and the public.
2.86
The Committee was pleased to note the steps taken by Defence to standardise its risk management software approach and notes that this will address the ANAO’s concerns about Defence’s systems being able to support ‘reliable auditing’.
Cost performance
2.87
The Committee is of the view that the regular interchanging between the two cost measurements may lead to costings being unintentionally misinterpreted by the Parliament and the public reading the MPR, with the lack of the clarity reducing the accessibility of the MPR.
2.88
While acknowledging the column titled ‘Definition/Description’ in the requirements section of the Guidelines, the Committee expresses the view that a stand-alone ‘Definitions’ section should be contained within the Guidelines to explain frequently used terms, including (but not limited to) constant costs, out-turned costs, risk, Projects of Interest, Projects of Concern, Initial Materiel Release, Initial Operational Capability, Final Materiel Release and Final Operation Capability, for the benefit of the Parliament and the public.
2.89
The Committee also considers that it would be consistent with the ANAO’s purpose of supporting accountability and transparency that contextualised definitions of words or phrases which have a specific or technical meaning that the Parliament or the public would not ordinarily know the meaning of be included in any future audit reports. This will improve the readability of audit reports and allow for a common understanding of terminology which the reader may be unfamiliar with.
2.90
The Committee recommends that the ANAO, in conjunction where appropriate with the Department of Defence, considers ways to improve the clarity of the MPR, with an emphasis on making the report more understandable to readers who may not have technical knowledge of defence terminology. This could include the following:
A “Definitions” section in the MPR, with contextual descriptions of terms that may have specific technical meaning that is unique to the Department of Defence, such as constant costs, out-turned costs, risk, Projects of Interest, Projects of Concern, Initial Materiel Release, Initial Operational Capability, Final Materiel Release and Final Operation Capability, to improve the readability and accessibility of the MPR for the Parliament and the public. It is suggested that the ANAO consult with the Department of Defence to agree consistent definitions to be used in preparing the MPR;
A description of ‘total schedule slippage’ to provide clarity to the Parliament and public as to the concurrent nature of defence acquisition and the meaning of this term;
A section explaining the technical definitions of the use of the term ‘risk’ in the context of the MPR, including a brief description of the nature of high or extreme risks requiring disclosure. The explanation should have scope to allow the Department of Defence to discuss the risks of individual projects within the Project Data Summary Sheets.
A contextual definition of ‘caveat’ or ‘deficiency’ in future Major Projects Reports, in the context of projects being described as having achieved significant milestones with caveats.
2.91
The Committee recommends that the Department of Defence provide an update on the progress of the implementation of the Risk Reform Program across the Major Projects, including information on risk management practices and resolution of the issues that render data unable to support ‘reliable auditing’.
Schedule slippage
2.92
The Committee considers that, while it may be appropriate to publish the total schedule slippage for the projects reported in the MPR, it would be beneficial for the ANAO to provide an explanation as to how the figure should be interpreted, taking into consideration that projects are undertaken concurrently and not end-to-end.
2.93
Further, the Committee believes Defence’s suggestion, regarding how analysis of project scheduling could produce more meaningful insights, should be adopted. If incorporated, this analysis would provide a better indication to Parliament and the public of specific challenges faced by Major Projects over the course of the previous financial year.
2.94
The Committee recommends that the ANAO insert a subsection in Part 1 of future Major Projects Reports detailing the schedule variation for projects over the previous financial year, with explanation of why projects have experienced delays or gains in scheduling between the previous report and the current report.
Acquisition governance
2.95
The ANAO examined a number of Defence’s acquisition governance processes during the course of its examination of Major Projects, including:
The Defence Independent Assurance Review process, which are intended to provide Senior Executive officers ‘with assurance that projects and products will deliver approved objectives and are prepared to progress to the next stage of activity’;
The Projects of Concern process;
The Defence Quarterly Performance Report (QPR), which is designed to provide information relating to the delivery of major capability to the Australian Defence Force to government stakeholders and Defence, including the Ministers for Defence and Defence Industry;
Project Directives and Material Acquisition Agreements, designed to identify matters such as: terms of government approval, project scope, timeframes, allocated resources, and significant issues or risks; and
Business systems, such as the Monthly Reporting System and the Project Performance Review (PPR) platform.
2.96
In relation to the QPR, the ANAO conducted a review in July 2019 of the Defence QPR process which recommended a series of changes be made to improve the process. The ANAO found during the course of the MPR review that:
Defence’s June 2019 QPR reported on both improved and deteriorated performance for both acquisition and sustainment products since the previous QPR. This reflects a change in trend reporting consistent with the agreed ANAO recommendation.
2.97
Separately, the ANAO reported that it was not clear whether Project Directives had been signed prior to the corresponding Materiel Acquisition Agreements for all four new projects entering into the MPR in 2018-19. The ANAO expressed concern regarding whether Program Directives were signed for these programs, as they are ‘a requirement of the Interim [Capacity Life Cycle] Manual, and their production increases the likelihood of complying with government decisions’.
2.98
Inconsistencies were also identified by the ANAO in relation to the PPR platform when compared to the PDSSs. The PPR platform is an Information Communications Technology platform designed to ‘draw project performance data from Defence systems [and] is intended for use by project managers to inform discussions with Project Directors and Branch Heads’. Defence began a staged release of the PPR platform in January 2018, and was reported to be in Phase 2 of its Release, with 60 projects utilising the platform.
2.99
The ANAO reported that six MPR projects were confirmed by Defence to be using the PPR platform, which contained details which were ‘largely consistent with the projects’ PDSSs’. It did find a number of inconsistencies between the two documents, however, in relation to:
the number of High- or Extreme-rated project risks; and
Financial management
2.100
Project financial assurance statements were originally introduced in the 2011-12 MPR, and have been incorporated into the scope of the Auditor-General’s Independent Assurance Report since 2014-15. Contingency statements have also been included in the MPR since the 2013-14 report, and ‘describe the use of contingency funding to mitigate project risks’. The ANAO states that this reporting framework is intended to provide ‘greater transparency over projects’ financial status’.
2.101
The MPR reported that the total budget for all 26 MPR projects was $64.1 billion at 30 June 2019. This represents an increase of $24.4 billion, or 38 per cent, to the approved budget for the Major Projects since Second Pass Approval.
2.102
The ANAO further stated that between 30 June 2018 and 30 June 2019, the total approved budgets for all MPR projects increased by $1225.1 million, representing an increase of 2.1 per cent on the total approved budget over the financial year. This was explained to be due to exchange rate variations and net real increases.
2.103
An underspend was reported for actual expenditure for in-year forecasting, with a total of $4831.4 million expended in the financial year ending on 30 June 2019, as opposed to the initial Portfolio Budget Statements (PBS) forecasted expenditure of $5809.4 million. The ANAO stated that the underspend was attributable to factors such as: deferred planned expenditure (AWD Ships and P-8A Poseidon), delayed invoice payments (AWD Ships, P-8A Poseidon, and Hawkei); early payment of planned 2018-19 expenditure (P-8A Poseidon); and ongoing issues regarding design, reliability and delivery of parts which have delayed payment of milestones (Hawkei).
2.104
In compiling the Independent Assurance Report for the 2018-19 MPR, the ANAO identified a number of other issues in relation to the financial management of MPR projects in Defence, including:
Project-specific risks regarding financial management;
Contingency statements and funds;
Planned changes to the method of financial reporting in the MPR; and
Project personnel numbers and costs.
2.105
Details in relation to Project Offices’ ongoing costs, training, replacement capability, and other matters, are not contained in the MPR due to its focus on approved capital budget for acquisition.
2.106
Six projects reported major variances in approved budgets when compared to initial Second Pass Approval, including:
JSF, increasing its budget by $13.8 billion due to additional aircraft, exchange rate variation, and indexation;
AWD Ships, increasing its budget by $1.9 billion due to Real Cost increase and price indexation, but offset by decreases in exchange rate variation and transfers to facilities projects;
P-8A Poseidon, increasing its budget by $1.8 billion due to additional aircraft and increases in exchange rate variation;
MRH90 Helicopters, increasing its budget by $2.8 billion due to additional aircraft and price indexation, but offset by scope transfers for facilities and decreases in the exchange rate variation;
Growler, increasing its budget by $0.8 billion due to exchange rate variation and the cost of the Mobile Threat Training Emitter System and weapons, but offset by transfers to facilities projects and returning $0.2 billion to the Defence budget of surplus funds; and
Overlander Medium/Heavy, increasing its budget by $0.8 billion due to ‘project supplementation’ and exchange rate variations.
2.107
Defence explained that 58 per cent of the increase in approved budgets was the result of the additional acquisition of aircraft for the JSF and MHR90 Helicopter programs, with no cost variations being experienced other than exchange rate variation and price indexation.
2.108
For projects that experienced a real cost decrease, Defence stated that any dollars saved on a project are put into other projects that are yet to be funded or have project requirements.
Joint Strike Fighter financial management
2.109
The JSF was identified by the ANAO as exhibiting issues regarding financial management. The ANAO reported:
In September and November 2018, due to cost pressures, the Joint Strike Fighter project received government approval to transfer the project scope of $1.5 billion to other phases of the Joint Strike Fighter program (none of which have been approved by government). There was no corresponding transfer of funds out of the project budget.
2.110
The ANAO stated that the PDSS provided by Defence states that ‘there is sufficient budget, including contingency, remaining for the project to deliver the revised scope’. The ANAO observed, however, that the PDSS:
reports a risk that project capability may be affected by overall funding or programming issues arising from internal cost growth, forecasting accuracy and external budget constraints.
2.111
The JSF was also the only project to report a significant in-year overspend in the financial year ending on 30 June 2019, reporting an expenditure of $1942 million in comparison to $1821.1 million reported in the PBS. The ANAO attributed this variance to foreign exchange updates.
2.112
Remedial actions by Defence to mitigate this risk included: clarifying potential cost pressures on the project; actively identifying cost risk and engaging senior leaders in the management of those risks; and considering options for scheduling project expenditure to align with Defence’s available funds. These remedial actions focus on Defence’s engagement with and work alongside key stakeholders both domestically and abroad, as well as establishing frameworks to monitor identified and emergent risks.
Box 2.2: Case Study: New Air Combat Capability/AIR 6000
Established in 1999, the Joint Strike Fighter (JSF) project aims to replace Australia’s current air combat capabilities provided by the F/A-18A/B and F-111 fleets by acquiring and introducing 72 F-35A JSFs that will meet Australia’s air combat needs to 2030 onwards. The JSF is under contract with Lockheed Martin and the United States Government, and is being delivered via a government to government cooperative agreement. The 72 JSF will establish three operational squadrons, a training squadron, and necessary supporting and enabling elements to replace Royal Australian Air Force (RAAF) aircraft. With the JSF receiving First Pass Approval from government on November 2006 and undergoing two stages of acquisition approvals for Second Pass Approval, the project is forecasted to achieve IOC by 2020, and FOC by 2023.
The project’s complex nature has resulted in multiple issues requiring remediation, including: technical and specifications problems; integration issues; potential funding challenges; and potential delays in contracted goals. The PDSS explains that, while Defence actively engages in risk mitigation and managing technical issues as they arise, the JSF’s procurement is ultimately the responsibility of the United States Government which has the capacity to raise issues with Lockheed Martin and Partner Nations.
The JSF was originally budgeted at Second Pass Approval at $13.264 billion, which has now increased to $16.522 billion. During 2018, Defence obtained approval from the Australian Government to ‘move enabling scope to redistribute key project elements between AIR 6000 program phases’. The movement of enabling scope did not result in overall funding being increased for the JSF.
Contingency funds
2.113
According to the ANAO, contingency budgets are ‘not programmed or funded in cash terms’, but rather are provided where funding is not sufficient within a project’s programmed budget and approval is sought to access contingency funding. The MPR explains that contingency budgets are designed to ‘estimate the inherent cost, schedule and technical uncertainties of the projects’ in-scope work. If relevant, project PDSSs include statements regarding the application of contingency funds within the reporting period and indicate risks that have been mitigated as a result of the application of the contingency funds.
2.114
The ANAO reported that three projects drew on contingency funds within the 2018-19 reporting period:
While all projects reported that they could continue to operate within the total approved budget of $64.1 billion, MRH90 Helicopters, Repl Replenishment Ships, and Battle Comm. Sys (Land) 2B were required to draw upon contingency funds to complete project activities.
2.115
Due to the scope remaining, budgets and contingency allocations from two projects being transferred to the Collins R&S project in 2017-18 and 2018-19, the ANAO reported that this reallocation of funds resulted in all 26 projects in the 2018-19 MPR having a contingency allocation.
2.116
The ANAO also raised two issues in relation to projects’ contingency logs:
A continuing issue in relation to the ‘clarity of the relationship between contingency allocation and identified risks’; and
Two project offices not meeting internal requirements for keeping records of contingency log reviews.
2.117
Across the current 26 projects reported in the MPR, approximately $1.2 billion, or 2 per cent of combined project approval value, has been spent from contingency budgets since September 2000.
Committee comment
2.118
The Committee is of the view that project financial assurance statements can be complex for an audience of a non-Defence background to read and understand. The reporting of Defence’s financial management of major projects should be written in such a way that would allow the Parliament and the general public to understand where changes in costs have occurred. These statements should also explain the reasons for changes in costs, which would improve transparency with the various cost pressures that a project, such as the JSF project, might be experiencing.
2.119
The Committee is also of the view that clarity of language should also be used when reporting on measures beyond the cost of a project, such as for the project’s scope and variations to its capability.
2.120
The Committee recommends that Defence and the ANAO use clear and accessible language in future Major Projects Reports when reporting on and describing costs, scope and capability variations.
Project Maturity Framework
2.121
Project Maturity Scores are used in the MPR to indicate the ‘relative maturity of capital investment projects as they progress through the capability development and acquisition lifecycle’. These scores are the result of an assessment of seven factors, including: Schedule, Cost, Requirement, Technical Understanding, Technical Difficulty, Commercial, and Operations and Supports. The assessment of projects is conducted at defined milestones for all projects, and provides an indication of the project’s current status and whether it is progressing adequately.
2.122
The ANAO has previously raised concerns regarding the internal consistency of the policy guidance. In the 2017-18 MPR, the ANAO identified inconsistency in applying Project Maturity Scores, which it stated ‘would benefit from a review for internal consistency and the relationship to Defence’s contemporary business’. The ANAO explained that the Project Maturity Scores did not necessarily reflect the actual budget expenditure, total progress or risk status of projects.
2.123
In its review of the 2017-18 MPR, the JCPAA recommended that Defence ‘provide a written update to the Committee outlining progress in updating Project Maturity Scores’ after the tabling of the report. This Recommendation was prompted by Defence’s ‘remain[ing] behind the Committee’s expectations’ in relation to previous Recommendations, and that progress had been ‘slow and uncertain’.
2.124
In response, Defence stated that it had finalised its preliminary review of the Project Maturity Framework in order to update scoring language. Further, in consultation with the ANAO, Defence adopted a two-stage approach in re-calibrating its scoring methodology to better reflect the Capability Lifecycle. Defence stated that the new approach would provide improvements such as: allowing scoring to begin at Gate Zero; increasing the total scores from 70 to 100; and ‘increasing the granularity of scores between [Initial Operational Capability (IOC)] and FOC and in the detailed design, testing and certification phases’ due to their importance as project development points.
2.125
Defence asserted that the two-staged approach was provided to internal stakeholders and the ANAO in June 2018, but in doing so several issues were identified. In particular, Defence explained:
Concerns were raised with respect to “value for money” in terms of cost and time required to make the MRS changes, noting that the Enterprise Resource Planning Project begins to roll-out from 2020.
2.126
Defence advised that it would continue to use the current Project Maturity Score framework until the updated policy had been put in place. Other improvements made by Defence to the current policy included: strengthening existing guidance and materials; amending Project Maturity Score testing procedures; and the inclusion of Project Maturity Scores in ‘Project Dashboards’ in the CASG Quarterly Performance Reports to raise awareness and ‘validity’ within Defence.
2.127
In the 2018-19 MPR, the ANAO reported that Defence had developed a draft Project Maturity Score policy, which was retitled the ‘Project Progress Score’. Defence stated that the new policy ‘is being reviewed as part of a wider evaluation of the Program Management governance frameworks’. The new policy was advised by Defence to better reflect language under the Smart Buyer policy. The ANAO stated that it was provided the draft policy in November 2019, and identified that the new policy will be based on 10 project attributes, rather than the current seven, and a total score out of 100, rather than the current 70. According to the draft policy, the new policy is not intended to assess project risk.
Committee comment
2.128
The Committee recognises the parallel views of the ANAO and Defence regarding project maturity scores and the shared objective for an improved framework to better reflect the life cycle of project delivery. The Committee looks forward to the inclusion of a clearer metric for project life cycle in future MPRs.
Capability performance
2.129
Capability performance has been a topic of interest in the MPR since its inception. The ANAO explained in the 2018-19 MPR how capability is defined and assessed by Defence:
Defence defines capability as the power to achieve a desired operational effect in a nominated environment, within a specified time, and to sustain that effect for a designated period. An operational effect is achieved by combining the nine Fundamental Inputs to Capability – organisation, command and management, personnel, collective training, major systems, facilities and training areas, supplies, support, and industry – and undertaking designated operations.
2.130
The ANAO has previously raised concerns that Defence’s assessment of capability is based on its ‘predictions of the final capability that would be achieved on the basis of deliverables and/or activities completed’. The ANAO argues that this model requires forecasting assumptions regarding achievements and is a subjective approach.
2.131
The JCPAA recommended in its October 2017 review of the MPR that Defence review its procedures in relation to capability estimates. In March 2018, Defence stated that it would ‘conduct a schedule baseline validation activity for the [MPR] projects to drive greater consistency in schedule reporting’. Defence did caution, however, that capability reporting was a ‘very sensitive issue’ which could be difficult to discuss publicly. Defence asserted that it would continue to work towards providing unclassified information regarding capability to ‘give confidence to the [P]arliament, to the public and to those who are interested in levels of capability’.
2.132
The ANAO reported that in September 2018 it observed continued ambiguities in the ‘Material Capability Delivery Performance’ charts prepared by Defence, and that advice from Defence in November 2018 that ‘partial progress had been made on its schedule baseline validation activity’. The ANAO further observed that ‘a measurement of schedule milestones will not necessarily reflect a measurement of capability delivered’. As a consequence of this issue, the ANAO uses a measure of key material capabilities delivered, entitled Capability Delivery Progress, which it states ‘presents a current assessment of the capability delivered, which differs from Defence’s prediction of final capability’.
2.133
In 2018-19, the ANAO reported that Capability Delivery ‘lags Project Maturity for the majority of projects (19 of 26)’. The ANAO explained that Capability Delivery was likely to increase relatively to Project Maturity over time due to increased testing and reviews, which develop confidence in a project’s ability to deliver its scope and capability prior to its delivery. The ANAO, however, observed that 12 projects were shown to be behind in Capability Delivery by 20 per cent or more relatively to Project Maturity, and that 10 of these projects were lagging by 50 per cent or more.
2.134
In discussing the different approaches to measuring capability performance, Defence stated:
We will need to continue to work … on how we best articulate [capability performance] in a public forum to give confidence to the parliament, to the public and to those who are interested in levels of capability—how we can express it in an unclassified means … What we're seeking to do in working with the Audit Office is to take out any ambiguity that might be perceived from some reporting or associated reporting.
Committee comment
2.135
The Committee notes the previous recommendations made by the JCPAA on improving the reporting of capability delivery performance and encourages Defence to continue working with the ANAO in finding an improved method of reporting this metric. An enhanced capability performance measure will ultimately improve the accountability and transparency of Defence acquisitions for the benefit of the Parliament and the general public.
Proposed review of Major Projects Report
2.136
During the course of the inquiry, Defence proposed a review of the MPR process to ensure that the report remains relevant for its intended purpose and reflects the current Defence operational environment. Defence put forward a number of reasons to review the MPR, including:
ensuring that the MPR ‘is focussed on outcomes, rather than … process and compliance’; and ‘strict bureaucratic compliance, and looking for ways of streamlining the information reported’;
considering the MPR in light of the strategic context in which Defence manages projects, including information relating to Australian Industry Capability in the form of statements of work;
the current inclusion of information that Defence argues is no longer useful or has become outdated in practice, such as: Project Maturity Scores, categories of projects which risk over-simplifying the type and engagement with industry; the nature of contracting; and standardised data which does not reflect individual projects;
difficulty for Defence in both reporting in the required timeframe and for recommendations from the previous MPR to be implemented; and
practical issues in compiling the MPR, such as the cost to both the ANAO and Defence, and the burden of reporting in multiple formats on Project Managers.
2.137
Defence recommended that it and the ANAO:
conduct a tailored Smart Buyer review of the processes, focus, and content of the report. If agreed, the independent external board members would undertake a review using the proven structured workshop approach. These findings would be presented to the Committee, and would welcome suggestions and feedback from other stakeholders, potentially including them as part of the Smart Buyer process.
2.138
On this issue, the Auditor-General has commented:
If the committee were of a mind that there should be a review, I think it would be useful for the committee to determine what the terms of reference for such a review would be and what it would be seeking to get out of it. I think it's an uncomfortable position for preparers and auditors to be the ones deciding how something should be reviewed or what priorities are underpinning the review. The key to this environment is the parliament, and I think that's where we should drive the desire from.
Committee comment
2.139
The Committee has observed differences in the approach taken by Defence and the ANAO to the MPR in recent years, highlighting the need to clarify the function and format of the MPR in line with advancements in both Defence acquisition and project management practice.
Conclusion
2.140
The Committee thanks the ANAO and Defence for their work in compiling the 2018-19 MPR and their participation in the inquiry, particularly given the unusual circumstances posed by the public health restrictions in 2020.
2.141
The ANAO’s findings in its recent performance audit of the Projects of Concern regime highlighted issues regarding consistency and transparency, which are reflected in the MPR’s findings. Given the importance of these projects and the significant risks posed by their potential mismanagement, the Committee is particularly interested in monitoring improvements. The Committee acknowledges Defence’s progress in meeting the ANAO’s recommendations, and looks forward to further developments in implementing these changes.
2.142
Further issues were identified by the ANAO in the reporting of projects meeting significant milestones with caveats. The Committee observes that Defence advised the ANAO that it discouraged Independent Assurance Reviews from recommending caveats at FOC, but that it was unclear whether caveats were considered acceptable at other major milestone points. It is also uncertain what Defence means by a ‘caveat’ or ‘deficiency’, and at what point a project inability to meet the requirements constitutes an acceptable caveat. The Committee encourages further work to improve transparency and clarity regarding this process.
2.143
The Committee acknowledges Defence’s proposal to review the MPR process, and recognises the arguments provided in Defence’s submission and the Statement by the Secretary in Defence stating why the MPR process could be considered outdated. In considering this matter, however, it is important to recognise the objective and role of the MPR, which is to provide the Committee and the Parliament with accountability and transparency in how Defence procures large-scale projects and the performance of the select projects. It is critical to observe that the primary stakeholder and client of the MPR is the Parliament, as opposed to industry and other stakeholders. The MPR should remain focused on its function as an accountability tool for the purposes of parliamentary oversight, although it should be made more accessible and readable for the general public.
2.144
The Committee notes the concern amongst Committee members about the adequacy of answers provided to questions on notice. Specifically, there were three responses from Defence which were not to the satisfaction of members. Of these responses to questions on notice, two had not addressed the information specifically requested by members and the third response was only provided after the matter was pursued by the Committee through correspondence. The Committee takes this opportunity to remind all witnesses of their responsibility to provide complete answers to questions on notice.