Audit Report No. 48 2008-2009

Chapter 9 Planning and Approval of Defence Major Capital Equipment Projects

Introduction

9.1                   The management of major capital equipment projects in Defence is a complex and challenging activity. Defence’s performance in this area has been the subject of a number of reports by Parliamentary Committees (including the JCPAA), the ANAO and other government commissioned reviews. Over the years, ANAO performance audits into Defence procurement have identified significant weaknesses in project planning—including risk identification and management, as well as project costing issues—resulting in projects experiencing cost overruns, scope changes and delayed implementation.

9.2                   In December 2002 the then Government commissioned a review—the Defence Procurement Review (DPR) (also known as the Kinnaird Review)—of major capital acquisitions in Defence. The review’s report, published in August 2003, made ten major recommendations and a number of additional points for consideration.

9.3                   Recommendation No.3 of the Kinnaird Review was aimed at strengthening the then existing two-pass approval process for Defence’s major capital equipment acquisitions. The recommendation was that:

Government should mandate, and enforce via revised Cabinet rules, a rigorous two–pass system for new acquisitions with government considerations dependent on comprehensive analyses of technology, cost (prime and whole–of–life) and schedule risks subjected to external verification.[1]

9.4                   Following the then Government’s broad acceptance of the recommendations of the Kinnaird Review in September 2003, Defence commenced implementing a strengthened two–pass process to support the planning and approval of major capital equipment projects.

9.5                   Additionally, in May 2008 the Government commissioned the Defence Procurement and Sustainment Review (also known as the Mortimer Review). The Mortimer Review, provided to Government in September 2008, made 46 recommendations aimed at addressing five principal areas of concern, one of which was ‘the inefficiency of the process leading to Government approvals for new projects’.[2] The Government agreed to 42 recommendations, agreed in part to a further three recommendations and did not agree to one recommendation (that DMO be established as an Executive Agency under the Public Service Act 1999).[3]

The Audit[4]

Audit objective and scope

9.6                   The objective of the ANAO audit was to assess whether the strengthened two–pass approval process for major capital equipment projects is being implemented effectively.

9.7                   The audit’s scope included an examination of key capability development documentation prepared for a sample of 20 projects drawn from the 84 (as at 27 May 2008) projects that had received first pass, second pass or both first and second pass approval since the introduction of the strengthened two pass approval process.[5] The audit also undertook a review, against the requirements of the Cabinet Handbook, of the submissions to Government seeking first and/or second pass approval for these 20 projects.[6] The ANAO’s sample included many of the highest value projects, balanced across the various environments (land, sea, air and joint projects) and reflected the diversity of types of major capital equipment acquisitions undertaken by Defence.

Overall audit conclusion

9.8                   The following is an extract from the ANAO’s overall audit conclusion. This report will provide more detail on the ANAO’s findings where relevant to the Committee’s inquiry.

Defence has established an appropriate administrative framework for implementing the strengthened two–pass approval process, including high–level oversight, and established the Capability Development Group (CDG) to administer the system.[7] Defence has also issued a Defence Capability Development Manual (DCDM), that describes the strengthened two–pass approval process and provides ‘authoritative guidance to CDG staff in carrying out the Group’s core tasks of developing investment proposals (including options) for new Defence capabilities for consideration by Government and managing the Major Capital Equipment program’.[8]

[However]… the execution of the capability development processes for the case study projects in the ANAO’s audit sample at times differed from the authoritative guidance set out in the DCDM and the Cabinet Handbook.

…Defence put in place a sound administrative framework following the 2003 Kinnaird Review but has not applied sufficient discipline through its governance arrangements to give assurance that the key elements of the framework are consistently applied in the development of capability development proposals. This increases the risk that the benefits, particularly in terms of reduced risks, sought through the reforms flowing from the Kinnaird and Mortimer reviews may not be realised to the extent expected.

Given the importance of effective planning and scoping to the successful delivery of capability, further attention to a range of issues is required to provide government with assurance that the body of information provided to inform its decisions on major defence acquisitions meets the standards previously set, and expected, by government. These issues include the need to:

n  revise the administrative framework, particularly the DCDM, such that:

§  clear guidance is provided as to the key elements that are required to produce sound proposals to government at first and second pass; and

§  the requirement for authorisation at an appropriate level for the rationale for, and the elements of, the approach to be applied in a particular project is mandated and that such authorisation is recorded;

n  substantially improve CDG’s recordkeeping policies and performance to ensure that key material supporting first and second pass submissions is appropriately stored and accessible. Sound document management is particularly important in the context of Defence major capital equipment projects given the sometimes long time-lines associated with both the capability development phase and also the acquisition phase;

n  agree with Finance a suitable approach to allow Finance’s early and ongoing involvement in the evaluation of capability development proposal costings;

n  adequately resource, train and support CDG desk officers; and

n  ensure adherence to the requirements of the Cabinet Handbook, particularly in relation to preparation and provision of costings and the assessment and description of technical risk in submissions.

ANAO recommendations

9.9                   The ANAO made the following recommendations:

Table 6.1        ANAO recommendations, Audit Report No. 48 2008-2009

1.	The ANAO recommends that Defence agree with Finance a suitable approach to allow Finance’s early and ongoing involvement in the evaluation of capability development proposal costings.   Defence response: Agree
2.	The ANAO recommends that Defence develop, promulgate and implement a sound records management policy within CDG.   Defence response: Agree
3.	The ANAO recommends that Defence refine its methodology for assessing and describing technical risk for future acquisitions and ensure that submissions do not proceed without a clear statement of technical risk, consistent with the agreed methodology as set out in the Cabinet Handbook.   Defence response: Agree
4.	The ANAO recommends that Defence ensure that submissions to the NSC for first or second pass approval include explicit acquisition and whole-of-life cost estimates, agreed by Finance, as required by the Cabinet Handbook.   Defence response: Agree

The Committee’s review

9.10               The Committee held a public hearing on Wednesday 28 October 2009 to examine this audit report. Witnesses from the following agencies attended and gave evidence:

n  Department of Defence (Defence);

n  Defence Materiel Organisation (DMO); and

n  Australian National Audit Office (ANAO).

9.11               The Committee heard evidence on the following issues:

n  individually tailoring project requirements;

n  description of technical risk and presentation of cost estimates in capability development Cabinet submissions;

n  involvement of the Department of Finance in verifying cost estimates in Cabinet submissions;

n  records management within Defence; and

n  relationship between this performance audit and the Major Projects Report.

Individually tailoring project requirements

Audit Findings[9]

9.12               The Defence Capability Development Manual (DCDM) was released in 2005, amended in 2006, and at the time of the Committee hearing was in the process of being revised by Defence. The DCDM provides ‘authoritative guidance’ on the implementation of the strengthened two-pass approval process for Capability Development Group (CDG) staff. CDG are responsible for various aspects of the capability development process within Defence.

9.13               The ANAO’s analysis of 20 sample projects proposed since reform to the two-pass process found a ‘range of deviations from the guidance in the 2006 DCDM’. The Chief of CDG advised the ANAO that processes had evolved since the 2006 DCDM, which meant that there had not been strict adherence to the manual. The Chief of CDG also advised that the DCDM was developed by Defence to provide generic guidance to desk officers, but in practice the process is tailored for each project.

9.14               The DPR states that ‘(A) strong mandatory two-pass system should provide a precise and understandable process for the procurement of defence capabilities, which ensures that government will be presented with robust proposals’.[10] The ANAO notes that adopting a tailored, project by project approach is not inconsistent with this:

Nevertheless, where a tailored approach is to be adopted on a project by project basis, it is still important to ensure that key elements required to produce sound proposals to government at first and second pass are clearly identified and executed and that the rationale for, and the elements of, the approach to be applied in a particular project are clearly authorised at an appropriate level. It was not evident that Defence consistently applied this level of discipline in relation to the approaches taken to developing the first and/or second pass approval submissions to Government for the projects in the ANAO’s audit sample.

9.15               While not making a recommendation in this area, the ANAO considered that:

Defence should ensure that the revised DCDM clearly identifies for CDG staff the key elements that are required to produce robust proposals to Government at first and second pass, and provides guidance on the process to be followed to obtain authorisation for the particular approach to be pursued in the context of an individual project.[11]

The Mortimer Review and the Government response to the Review

9.16               The Mortimer Review argued that, to improve efficiency, the two-pass approval process should become more flexible and the number of times projects went to Government should be dependent upon project cost, maturity, complexity and risk.[12] The Government agreed with the Review’s recommendation that ‘Government approval of major Defence projects should occur through a tailored application of the two-pass process’, however in their response highlighted the strength and flexibility of the existing approach. The Government’s response also noted that the current two-pass approval process ‘already allows for combined First and Second Pass approval for less complex projects and for more than two considerations where projects are particularly complex’.[13]

Committee examination

9.17               The CEO DMO was asked for an overall appraisal of the effectiveness of the two-pass approval system. He observed the Mortimer Review findings that the process worked well but in some cases additional or fewer passes were required. A hypothetical example of a project that would only require Government consideration once would be a follow-on buy of an asset already in operation. Examples were provided of projects that have or will go for Government consideration more than twice:

n  Joint Strike Fighter will likely be considered by Government four or five times;

n  Air Warfare Destroyer went to Government seven times.[14]

9.18               The Committee asked the ANAO what is required to ensure a flexible approach to capability development is acceptable. The new Defence Capability Development Manual (DCDM) would be central to outlining requirements:

In the course of the audit, [Vice Admiral Tripovich, Chief of Capability Development Group] did communicate to us that he was in the process of developing a new Defence Capability Development Manual and that would look to give guidance to CDG staff such that they would understand what to do in the flexible design to fit the particular project environment. Our only concern was that the guidance to staff made clear to them what elements are mandatory and what elements are not mandatory—in certain circumstances it may be that none are mandatory—that the process for particular projects is authorised at a sufficiently senior level and signed off and that, when things are done at the direction of government, the advice from government is recorded, clear and available.[15]

9.19               The ANAO was not opposed to having flexibility in the process, however stressed that ‘there needs to be agreement in government on what is required and what is not required’. Defence acknowledged the need to correctly identify and record reasons for taking certain decisions. They felt, however, that the decision to take a tailored approach was captured in advice to the Minister and their subsequent agreement that a deviation was necessary.[16]

9.20               It was noted by Defence that most of the ANAO’s sample projects would have deviated in their process from the guidance in the DCDM, and the ANAO added that there was no evidence that recorded the authorisation to tailor the approach. Defence was confident that despite a lack of records, decisions to tailor approaches were properly made.[17]

9.21               When Defence noted that relevant capability development desk officers do not themselves determine when to deviate from DCDM guidance, the Committee questioned Defence regarding the ANAO finding that there was no evidence tailored approaches had been clearly authorised. Defence stated that desk officers are guided by internal committee processes which are minuted:

ANAO observed that we did not capture in that or in any other form every single decision to deviate. We had quite a robust discussion about the level of recording that should be necessary. I have taken that on board and it is obviously important that, if at any time government wants to look back and ask, ‘Why were certain things done?’, we need to be able to provide the evidence.[18]

9.22               The Committee questioned whether the DCDM was generic guidance, highlighting the manual states that it is ‘authoritative guidance’. Defence acknowledged that it was generic guidance, and noted that the revised manual would incorporate all lessons learned from the various reviews undertaken:

But it will always be the case that it will paint, if you like, the generic model, but it must always be tailored for the particular project. Otherwise we will be bound by process.[19]

9.23               Defence gave assurance to the Committee that a new DCDM will set out the approval process in cases where projects need to deviate from that listed in the manual. Defence added that the process will continue to evolve.[20]

9.24               It is appropriate to modify the process to approach Government, provided that this is initiated or agreed to by Government. Defence needs to ensure that officials involved in preparing submissions are aware of the requirements of an authorised tailored approach. Clear, formal records of deviations from standard process are needed.

 

Recommendation 15

9.25

The Committee recommends that when preparing submissions Defence develop a procedure to ensure that any divergence from the generic guidance provided in the Defence Capability Development Manual (DCDM) is authorised at an appropriate level and to record: n  any modifications to the capability development approval process contained in the Defence Capability Development Manual (DCDM) for a project; and n  reasons for diverging from the generic model.

 

Description of technical risk and presentation of cost estimates in capability development Cabinet submissions

Audit Findings[21]

9.26               The ANAO assessed whether the 20 projects in their sample met revised Cabinet Handbook requirements, which were introduced as a result of the Kinnaird Review.[22] Because it took CDG eight months to supply the ANAO with what it considered to be the required documents, the ANAO did not have time to undertake an audit of the adequacy of the information provided to support cabinet decisions, only to review whether key documents had been prepared. The ANAO’s analysis found three key problem areas: addressing technical risk, whole-of-life costings, and discussion of trade-offs. Technical risk and whole-of-life costs were key focus areas in the Kinnaird review, and were raised during the Committee’s hearing.

Description of technical risk in Cabinet submissions

Findings of the Defence Procurement Review

9.27               The 2003 Defence Procurement Review (DPR) stated that:

Standardised Technology Readiness Levels should be used to assess the technology maturity of equipment, including sub-systems, at various stages of development. Proposals lacking technology risk ratings would not proceed for government consideration…

Implementation of such a system enables non-technical readers to better understand the level of technological risk of particular proposals and therefore facilitat[e] a better assessment of their merits. We understand that [the Defence Science and Technology Organisation] would be capable of using this methodology to rate technology risks for new capabilities.[23]

Audit findings[24]

9.28               The ANAO’s analysis found that Cabinet submissions were inconsistent, and some even ‘vague’, in their assessment and description of technical risk, and used different categorisations including one (System Readiness Levels) that Defence was unable to explain. It found that only one out of 10 second pass submissions utilised the method to measure technical risk (Technology Readiness Levels) recommended by the DPR and required under the Cabinet Handbook. The ANAO recommended that Defence refine its methodology for addressing technical risk to ensure it is clear and conforms to Cabinet Handbook requirements.

Committee examination

9.29               The provision of advice on technical risk to government was viewed by the ANAO as having improved over time, however it was still a work in progress. They were also surprised at the lack of precision in some of the Cabinet submissions.[25]

9.30               Defence agreed that the process was one of continual improvement, which was why Defence had developed a better method of expressing technical risk than only TRLs:

… we have grown to learn, as has the Government, that a TRL is in itself not sufficient. The more comprehensive technical risk assessment that now forms part of the Cabinet submission…is a more effective way than just a TRL to explain the important technical risks and the issues that arise from that for government.[26]

9.31               Defence agreed with the Committee’s proposition that they were ‘ahead of the game’ in relating to addressing risk but this was not reflected in the Cabinet Handbook.[27]

9.32               Defence Science and Technology Organisation (DSTO) provided a detailed response to concerns over the adequacy of and approach to addressing technical risk in Cabinet submissions. Defence produce a document called a ‘Technical Risk Assessment’ (TRA) for each project, which ‘informs the preparation of the Cabinet submission’. The TRA:

… starts with an assessment of a Technology Readiness Level of all the key subsystems in that equipment and from there goes to look at how those systems are integrated together, the suitability of the technology for the purpose for which it is intended and the risks which may arise due to the need to develop technology by a particular time so that you can acquire the system effectively. All of that information is assembled into a document which has a standard and defined format where all the risks terms are full[y] defined and that informs the preparation of the Cabinet submission. The simple answer is we do not use TRLs as a stand-alone indicator of risk because they do not actually tell you anything about risk; they tell you about the maturity of the technologies you start from.[28]

9.33               DSTO argued that TRLs are focused on the maturity of a particular piece of technology (a sub-system) and not the technical risk of integrating the sub-systems and delivering the integrated system. Defence are refining their methodology on addressing risk ‘and we would expect to provide some recommendations as to what measures should be used to better describe risk so that the Cabinet Handbook can be updated’.[29]

9.34               In terms of the adequacy of advice, DSTO pointed out that the ANAO focused on the Cabinet submissions, and are ‘not seeing the Defence documentation which lies behind those Cabinet submissions’. DSTO highlighted two publicly available publications that outline the nature of their TRAs.[30] The Committee later examined the two public reports highlighted by DSTO.[31]  

9.35               The Technical Risk Assessment of Australian Defence Projects publication states that TRAs undertaken by Defence focus on TRLs (the maturity and feasibility of individual technologies) during the early stages of project development and the focus then shifts to using Systems Readiness Levels (SRLs), which measure technical risk associated with systems, including their integration into the one prime system, in the later stages of capability development, with SRLs becoming the primary measure at second pass approval. SRLs were seen to ‘augment’ the use of TRLs, however TRLs are still to be used ‘at each decision point in the capability development lifecycle’.[32] In some cases TRL scores could be high (low risk) but SRL scores low ‘because of integration issues, environmental issues, interoperability and possible dependence on other (as yet untested) technologies’.[33]

9.36               In addition, the technical risk assessment is supported by a categorisation of likelihood (likely, possible, unlikely) and consequences of risks being realised (minor, moderate, major), with categorisations based on the standard AS/NZS 4369:2004.[34]

9.37               The second publication, Technical Risk Assessment: a Practitioner’s Guide notes that using TRLs to identify the maturity of technologies for a project is working well, however the use of SRLs is more problematic and there is confusion about the difference between technical and technology risk and identifying technical risks.[35]

9.38               In the hearing the ANAO emphasised that they were not purporting to have greater expertise on technical risk than Defence and DSTO, nor criticise the merits of Defence’s approach; their audit of technical risk was based on the requirements in the Cabinet Handbook. They noted that:

Certainly what is being explained to us—the work that is being done at the moment—sounds very positive in further addressing the desire of Government for improvement in this area. We have no particular attachment to TRLs. It is simply that they were what [were] required at the time.[36]

9.39               Before making concluding remarks, the Committee wishes to highlight the ANAO finding that only 1 out of 10 second pass submissions used TRLs, while another used it for some options (not the recommended option) and assigned an imprecise rating of TRL 3-7. Two first pass submissions used TRLs (the Handbook only mandated TRLs for use at second pass), however one used an imprecise rating of TRL 4-8.[37] A lack of TRLs in Cabinet submissions may reflect that they are used in documentation lying behind Cabinet submissions.

9.40               Based on evidence to the Committee and Defence publications, there appears to be a systematic process as part of capability development that focuses on risk, including utilising different standardised measurements for different stages of project development, and the Defence technical experts (DSTO) having a key role in the process. The Committee believes Defence is best placed to determine how they wish to rank technical risk. However, whatever approach is utilised needs to be communicated consistently, and in a concise and understandable manner, a point expressed in the hearing by Defence:

The important thing is that, when the government makes its decision, Defence and government fully understand what we are getting ourselves into, that we have full transparency of all of the issues and that they are known and in the Cabinet submission. Government has an assurance that we have them covered.[38]

9.41               It appears that this comprehensive approach using TRAs, including utilising both TRLs and SRLs, may not be making its way into Cabinet submissions in a clear, consistent and systematic manner.

9.42               In this context, the Committee encourages Defence to seek amendments to the Cabinet Handbook to reflect their approach, ensuring that this leads to consistent and accurate description of risk. Additionally, only one score should be applied to each TRL and SRL, not imprecise multiple scores (such as 3-7, 4-8 or even 1-2).

9.43               Government must be fully informed on capability options in a way that is understandable and allows comparison across projects. It is crucial that Defence ensure that technology and technical risks are clearly stated. This will provide a good basis with which government can make a decision on capability.

 

Recommendation 16
9.44	The Committee recommends that Defence work with the Department of Prime Minister & Cabinet (PM&C) to amend the Cabinet Handbook to accurately reflect the more specific risk measurement process developed by Defence which should be included in submissions for both first and second pass assessment.

 

Recommendation 17

The Committee recommends that Defence ensure that Cabinet submissions for future major capital equipment projects provide advice that is clear and consistent, and include the following: n  Technology Readiness Level (TRL) and System Readiness Level (SRL) scores for each option; n  description of likelihood and consequence of risk based on these ratings. The Committee further recommends that Defence develop a procedure to ensure that staff involved in Technical Risk Assessments (TRAs) have a clear understanding of System Readiness Level (SRL) measurements so that the figure provided precisely reflects risk levels.

 

Presentation of cost estimates in Cabinet submissions

Findings of the Defence Procurement Review

9.45               The 2003 Defence Procurement Review (DPR) stated that:

When taking decisions on capability options at first and second pass the whole-of-life costs must be presented to and understood by government. These not only comprise the cost of the prime equipment, but also infrastructure, equipment operating costs, through-life-support, and the resources required to manage acquisition.[39]

Government response to the Mortimer Review

9.46               As part of accepting a recommendation of the Mortimer Review relating to introducing a capability into service, Defence stated:

Capability Managers must include whole of life implications and independent advice from the CEO DMO on the cost, risk and schedule implications for projects in their advice to Government.[40]

Defence White Paper 2009

9.47               Under the heading of ‘better managing Defence costs’, The White Paper stated that:

The Government has also directed that greater attention be paid in the planning process to the whole-of-life cost dimensions of capability. This will be critical in providing Government with increased levels of confidence with respect to cost, schedule and technical risk when it considers major Defence projects. This will also assist in minimising scope variations in major acquisitions, and help drive down cost pressures.[41]

9.48               These three reports highlight the importance of, and continuing focus on, whole-of-life costing.

Audit findings[42]

9.49               The ANAO’s analysis found that most Cabinet submissions (14 out of 23) did not contain estimated whole-of-life costs as required by the Cabinet Handbook. The ANAO found that Defence often used NPOC (Net Personnel Operating Cost) estimates.[43] NPOC estimates only reflect anticipated change in operating costs associated with replacing a capability (where applicable), or update the estimate in the Defence Capability Plan for new capabilities. NPOC does not provide a transparent, understandable whole-of-life cost estimate, and may be a small number (even zero) depending on the estimated cost relative to the current capability, or relative to Defence Capability Plan estimates.

9.50               The DPR and the Defence White Paper 2009 emphasise the importance of getting whole-of-life costs, as they are a significant source of cost (over two-thirds of the whole-of-life cost of a platform will be incurred after the platform is introduced into service). The ANAO recommended that Defence include explicit whole-of-life cost estimates, agreed by Finance, as required by the Cabinet Handbook.

Committee examination

9.51               NPOC was described as the extra cost (‘new money that goes on top of the current funding stream’) to go from the current capability to the new one. The annual NPOC cost was multiplied by the estimated life-of-type of the platform, and added together with the acquisition cost and other costs such as contingency to get the whole-of-life cost.[44]

9.52               When queried by the Committee how these costs differed from what the ANAO considered to be understandable whole-of-life costs, Defence believed that the key issue was consistent presentation to government. Defence stated that agreement had been reached with the Department of Prime Minister and Cabinet (PM&C) and Treasury on a format for the cost sheet that includes NPOC and whole-of-life cost.[45]

9.53               The Committee is pleased to hear that a standard format has been agreed to highlight both NPOC and whole-of-life costs. Again, like the technical risk issue discussed above, it is important that such clear and consistent information is communicated to Government in submissions. The ANAO found only nine of 23 Cabinet submissions contained comprehensive, transparent whole-of-life costs. It appears that NPOC costs were being added to other costs to give a total overall cost.

9.54               Using NPOC does not appear to reflect a true life-of-type costing: a submission with NPOC costs does not tell government how much the current capability costs to run. For ease of comparison, Defence may wish to include in Cabinet submissions both the whole-of-life cost and also mention the NPOC, which would be of interest to government as it is the additional money required for personnel and operating costs.

9.55               To support the views on whole-of-life cost presentation found in the ANAO’s recommendation, the Defence White Paper 2009 and the Government’s response to the Mortimer Review, the Committee would emphasise to Defence that when presenting whole-of-life costs to government, they are clear, consistent and include the total estimated personnel and operating costs, not only variations from current costs. The Committee urges Defence to bear this point in mind when implementing Recommendation No. 4 of the ANAO report.

Involvement of the Department of Finance in verifying cost estimates in Cabinet submissions

Findings of the Defence Procurement Review

9.56               The 2003 Defence Procurement Review (DPR) found that:

Finance agreement to Cabinet submission costings is generally sought shortly before lodgement, and does not allow sufficient time for any in-depth analysis of capability, strategic, technical, legal or commercial issues associated with the costs and risks of major capital investments or other procurements. Ideally, Defence should provide well developed business cases for the proposed investments, based on sound costings models, for validation by Finance.[46]

9.57               The review stated that Finance ‘should be involved much earlier, and on a continuous basis, throughout the two-pass approval process so that they can contribute to effective quality assurance in relation to costings and risk, and technology readiness respectively’.[47]

Government response to the Mortimer Review

9.58               The Government’s response to the Mortimer Review states that reforming the Capability Development process to ‘provide Government with more reliable information on which to base judgements and a more efficient and effective capability development process’ will be achieved by, amongst other initiatives ‘ensuring early consultation with the Department of Finance and Deregulation on options, cost estimates and project risks’.[48]

Audit findings[49]

9.59               Defence advised the ANAO during the audit that it had implemented reforms in this area in response to the Kinnaird Review’s findings. The ANAO found, however, that Defence and Finance did not have an agreed process to facilitate Finance’s involvement with capability proposals. Each Department also had different perspectives on how effectively Defence engaged with Finance. Defence disagreed with Finance’s view that it does not receive relevant information early in the process, only receives costing and risk information 30 days or less from a submission to NSC, and that this information is often limited and of questionable quality.

9.60               The ANAO stated that it is important for both Departments to develop a common understanding on an appropriate approach to engagement. The ANAO also stated that the absence of agreed procedures and guidance for Defence engagement with Finance means that:

Government may not be consistently provided with the level of independent evaluation of capability development proposal costings envisaged by the DPR. In practice, this has meant that of the 23 submissions to government reviewed by the ANAO in this audit, four went to government containing cost estimates that Finance had been unable to agree to. The Cabinet Handbook states that ‘each first and second pass submission or memorandum requires agreement with Finance on the detailed acquisition and operating costings and financial risk assessment’.

9.61               Defence agreed to the ANAO’s recommendation that ‘Defence agree with Finance a suitable approach to allow Finance’s early and ongoing involvement in the evaluation of capability development proposal costings’. In their response to the recommendation Defence stated:

Defence will work to formalise the process for engagement with DoFD [Department of Finance and Deregulation] (and PM&C and Treasury).

Committee examination

9.62               When asked about the different views between Defence and Finance on engagement around costings, Defence reiterated their perspective, expressed in the audit, that Finance was sufficiently engaged early in the process. Defence noted that after the audit report was released, they engaged at a SES band 2 level (equivalent to a Rear Admiral) with Finance, PM&C and Treasury to come up with an acceptable agreement on engagement, which had been established.[50]

9.63               The Committee sought comment from Finance in response to Defence’s comments. The Department of Finance and Deregulation confirmed that it ‘has agreed an approach with the Department of Defence to allow Finance early and ongoing involvement in the evaluation of capability development proposal costings’.[51]

9.64               Defence stated that while they always try to get agreement with Finance on costings before submissions are lodged, they are not always successful. In such cases, the Minister can decide to take the submission forward, accepting the disagreement. The ANAO accepted this, noting that in some circumstances there may be disagreement that needs to be resolved by Cabinet. The audit was following the requirements of the Cabinet Handbook, which required Finance agreement. Of the four cases where Finance did not agreed to the costings, in only one case did government decide to defer consideration because of the disagreement.[52]

9.65               The Committee is keen to ensure that costings receive adequate scrutiny by Finance, and part of this is ensuring they receive not only early engagement but also receive the Cabinet submission costings early enough to allow sufficient time for scrutiny. The Committee is pleased Finance has an agreed approach to allow sufficient scrutiny of capability development proposal costings.

Records management within Defence

Audit findings[53]

9.66               The ANAO analysed the DCDM (high-level guidance) and CDG’s Process Map (detailed guidance) to identify the key activities and documents required under the strengthened two-pass approval process for 20 sample projects. Because it took CDG eight months to supply the ANAO with what it considered to be the required documents, the ANAO did not have time to undertake an audit of the adequacy of key documentation, instead they simply reviewed whether key documents had been prepared.

9.67               The ANAO was quite critical when outlining the overall results of the analysis:

Defence was unable to demonstrate… that the procedures outlined in the DCDM and the Process Map have been consistently followed…or that alternative procedures were appropriately authorised, managed and documented.

CDG could not provide the ANAO with final versions of around half of the key project documents requested…

In most cases, the ANAO was unable to determine whether Defence’s inability to provide requested documents was a consequence of poor records management practices or because the documents had not been prepared. Defence does not have an adequate audit trail of key decisions and requirements in relation to the capability development process.

9.68               The ANAO stated that significant improvement in CDG’s record keeping was needed and recommended that ‘Defence develop, promulgate and implement a sound records management policy within CDG’.

Committee examination

9.69               In the hearing, the ANAO summarised the need for good records:

… as the National Archives say in the guidelines that they put out, you need to have the documentation so that people know that the records are genuine, that they are accurate, that they can be trusted, that they are complete, that they have not been altered, that they are secure, that they can be found when they are needed and that they relate to the relevant documents. For us, that was the heart of the CDG records management issue.[54]

9.70               Defence’s progress on implementing the ANAO’s recommendation was sought, and they provided an update to the Committee:

… since the audit came out, we have, within CDG, promulgated formal policies on our document and management system, a formalised document-naming convention to make it easier to recall things from the system and a process for signing off documents and keeping signed copies. Training has commenced across the 250 people that work in my organisation. I will have a change of some staff over Christmas and it will be firmly part of our annual training continuum thereafter. By about mid-2010, I will have done a compliance audit, an internal check, to see if people had been doing what they were told to do. As I said, the training has started to roll out. It is very difficult to try to recover documents that you cannot find now.[55]

9.71               Defence’s management action plan for responding to audit recommendations was completed by December 2009.[56] The Committee is hopeful that this action will mean that the significant problems with records access encountered by the ANAO will be ameliorated in the future.

9.72               Defence stated that a key contributor to poor records management is ‘it is very difficult to find documents on Defence’s ICT system’.[57] In this context the Committee notes that the ANAO has recently commenced an audit into Defence’s ICT systems.

The relationship between the Performance Audit and the Major Projects Report

9.73               In addition to addressing issues raised in the audit report, the question was asked about the relationship between the performance audit report and the ANAO’s assurance report Defence Materiel Organisation Major Projects Report (MPR).[58] Concerns were raised that issues significant enough to be found in the Planning and Approval of Defence Major Capital Equipment Projects audit were not raised in the MPR review.[59]

9.74               The ANAO stated that the MPR was a different report from a more in-depth performance audit, and each report focused on different aspects of a project. The performance audit was of the approval process, while the MPR review covers the post approval process, where responsibility for procuring a capability is shifted to DMO. The assurance provided in the MPR is more limited than for a performance audit.[60] Only one ‘pilot’ MPR report has been released so far, and a key outcome of an annual MPR report is to document for Parliament trends in major projects.[61]

9.75               When asked whether the MPR document is a useful and reliable document given the lower lever of assurance, the CEO DMO stated:

I think the major projects report is very useful and reliable. I think we had a discussion in this committee several years ago about what the major projects report was supposed to achieve. We were faced with either doing 30 performance audits at the higher level of approval…or giving a summary year by year. What we are doing is giving a longitudinal summary so that you can see for every year how things have changed: how the risks have changed and how the dollars have changed. It is a very different process indeed from what [Audit Report 48] was about, which was examining the two-pass process.[62]

Conclusion

9.76               Asked whether the findings of the audit sample (20 projects) could reasonably be expected to be found in other Defence projects, the ANAO agreed that the sample findings:

… would lead us to believe that there would be other projects that exhibited those characteristics and some of those may be in the current bundle of work.[63]

9.77               Defence agreed that similar issues could arise in projects from the same era, however:

I would like to think, though, that, if you picked one that was just about to go to cabinet this week, for example, you would not find those things. I would be very confident that ANAO would see a completely different document and a completely different process as a result.[64]

9.78               From the evidence provided in the hearing, there appears to have been considerable effort and action undertaken by Defence to improve their approach to capability development proposals. Major Defence procurement is a very complex area, and one that has received significant attention over time.  It also takes time to bed down new processes and determine whether reforms have been implemented successfully.

9.79               One theme arising from the Committee’s inquiry is a need for renewed effort in communicating to government consistently and clearly. Defence appear to have established a sound process internally to assess technical risk, and it is important that this work is adequately captured in submissions to Cabinet. Government also needs clear information on the whole-of-life cost of capability options, including total personnel and operating costs.

9.80               Documentation and records management are important in maintaining the integrity of the capability development process. Defence must be able to deviate from the process outlined in the DCDM if directed by government. In such cases, staff within CDG must be informed as to what is required. In other cases of deviation not at the direction of government, a record must be made that includes sufficient explanation why the process is different and containing senior authorisation.

9.81               Finally, appropriate engagement of the Department of Finance and Deregulation is critical, as it provides government with independent scrutiny of Defence costings.

9.82               The ANAO’s audit has proven to be highly valuable in examining Defence’s progress in implementing reform properly and consistently, particularly as the nature of the information that goes to government receives little other public scrutiny. Given the findings of the ANAO, Defence internal reviews and external reviews such as Mortimer, and Defence assurances that processes will be improved, there is merit in the ANAO conducting a similar review to Audit No. 48 in the future, one that will cover issues addressed in the Committee’s inquiry. Defence as well as the ANAO stated in the hearing that Defence is on a path of continuous improvement; it is appropriate that Defence’s progress is monitored.