Audit Report No. 48 2008-2009
Chapter 9 Planning and Approval of Defence Major Capital Equipment Projects
Introduction
9.1
The management of major capital equipment projects in Defence is a
complex and challenging activity. Defence’s performance in this area has been
the subject of a number of reports by Parliamentary Committees (including the JCPAA),
the ANAO and other government commissioned reviews. Over the years, ANAO
performance audits into Defence procurement have identified significant
weaknesses in project planning—including risk identification and management, as
well as project costing issues—resulting in projects experiencing cost
overruns, scope changes and delayed implementation.
9.2
In December 2002 the then Government commissioned a review—the Defence
Procurement Review (DPR) (also known as the Kinnaird Review)—of major capital
acquisitions in Defence. The review’s report, published in August 2003, made
ten major recommendations and a number of additional points for consideration.
9.3
Recommendation No.3 of the Kinnaird Review was aimed at strengthening
the then existing two-pass approval process for Defence’s major capital
equipment acquisitions. The recommendation was that:
Government should mandate, and enforce via revised Cabinet
rules, a rigorous two–pass system for new acquisitions with government
considerations dependent on comprehensive analyses of technology, cost (prime
and whole–of–life) and schedule risks subjected to external verification.[1]
9.4
Following the then Government’s broad acceptance of the recommendations
of the Kinnaird Review in September 2003, Defence commenced implementing a
strengthened two–pass process to support the planning and approval of major
capital equipment projects.
9.5
Additionally, in May 2008 the Government commissioned the Defence
Procurement and Sustainment Review (also known as the Mortimer Review). The Mortimer
Review, provided to Government in September 2008, made 46 recommendations aimed
at addressing five principal areas of concern, one of which was ‘the
inefficiency of the process leading to Government approvals for new projects’.[2]
The Government agreed to 42 recommendations, agreed in part to a further three
recommendations and did not agree to one recommendation (that DMO be
established as an Executive Agency under the Public Service Act 1999).[3]
The Audit[4]
Audit objective and scope
9.6
The objective of the ANAO audit was to assess whether the strengthened
two–pass approval process for major capital equipment projects is being
implemented effectively.
9.7
The audit’s scope included an examination of key capability development
documentation prepared for a sample of 20 projects drawn from the 84 (as at 27
May 2008) projects that had received first pass, second pass or both first and
second pass approval since the introduction of the strengthened two pass
approval process.[5] The audit also undertook a
review, against the requirements of the Cabinet Handbook, of the
submissions to Government seeking first and/or second pass approval for these
20 projects.[6] The ANAO’s sample
included many of the highest value projects, balanced across the various
environments (land, sea, air and joint projects) and reflected the diversity of
types of major capital equipment acquisitions undertaken by Defence.
Overall audit conclusion
9.8
The following is an extract from the ANAO’s overall audit conclusion.
This report will provide more detail on the ANAO’s findings where relevant to
the Committee’s inquiry.
Defence has established an appropriate administrative
framework for implementing the strengthened two–pass approval process,
including high–level oversight, and established the Capability Development Group
(CDG) to administer the system.[7] Defence has also issued a
Defence Capability Development Manual (DCDM), that describes the strengthened
two–pass approval process and provides ‘authoritative guidance to CDG staff in
carrying out the Group’s core tasks of developing investment proposals
(including options) for new Defence capabilities for consideration by
Government and managing the Major Capital Equipment program’.[8]
[However]… the execution of the capability development
processes for the case study projects in the ANAO’s audit sample at times
differed from the authoritative guidance set out in the DCDM and the Cabinet
Handbook.
…Defence put in place a sound administrative framework
following the 2003 Kinnaird Review but has not applied sufficient discipline
through its governance arrangements to give assurance that the key elements of
the framework are consistently applied in the development of capability
development proposals. This increases the risk that the benefits, particularly
in terms of reduced risks, sought through the reforms flowing from the Kinnaird
and Mortimer reviews may not be realised to the extent expected.
Given the importance of effective planning and scoping to the
successful delivery of capability, further attention to a range of issues is
required to provide government with assurance that the body of information
provided to inform its decisions on major defence acquisitions meets the
standards previously set, and expected, by government. These issues include the
need to:
n revise the
administrative framework, particularly the DCDM, such that:
§
clear guidance is provided as to the key elements that are
required to produce sound proposals to government at first and second pass; and
§
the requirement for authorisation at an appropriate level for the
rationale for, and the elements of, the approach to be applied in a particular
project is mandated and that such authorisation is recorded;
n substantially improve
CDG’s recordkeeping policies and performance to ensure that key material
supporting first and second pass submissions is appropriately stored and
accessible. Sound document management is particularly important in the context
of Defence major capital equipment projects given the sometimes long time-lines
associated with both the capability development phase and also the acquisition
phase;
n agree with Finance a
suitable approach to allow Finance’s early and ongoing involvement in the
evaluation of capability development proposal costings;
n adequately resource,
train and support CDG desk officers; and
n ensure adherence to
the requirements of the Cabinet Handbook, particularly in relation to
preparation and provision of costings and the assessment and description of
technical risk in submissions.
ANAO recommendations
9.9
The ANAO made the following recommendations:
Table 6.1 ANAO recommendations, Audit Report No. 48
2008-2009
1.
|
The ANAO recommends that Defence agree with Finance a
suitable approach to allow Finance’s early and ongoing involvement in the
evaluation of capability development proposal costings.
Defence response: Agree
|
2.
|
The ANAO recommends that Defence develop, promulgate and
implement a sound records management policy within CDG.
Defence response: Agree
|
3.
|
The ANAO recommends that Defence refine its methodology
for assessing and describing technical risk for future acquisitions and
ensure that submissions do not proceed without a clear statement of technical
risk, consistent with the agreed methodology as set out in the Cabinet
Handbook.
Defence response: Agree
|
4.
|
The ANAO recommends that Defence ensure that submissions
to the NSC for first or second pass approval include explicit acquisition and
whole-of-life cost estimates, agreed by Finance, as required by the Cabinet
Handbook.
Defence response: Agree
|
The Committee’s review
9.10
The Committee held a public hearing on Wednesday 28 October 2009 to
examine this audit report. Witnesses from the following agencies attended and
gave evidence:
n Department of Defence
(Defence);
n Defence Materiel
Organisation (DMO); and
n Australian National
Audit Office (ANAO).
9.11
The Committee heard evidence on the following issues:
n individually
tailoring project requirements;
n description of
technical risk and presentation of cost estimates in capability development
Cabinet submissions;
n involvement of the Department
of Finance in verifying cost estimates in Cabinet submissions;
n records management
within Defence; and
n relationship between
this performance audit and the Major Projects Report.
Individually tailoring project
requirements
Audit Findings[9]
9.12
The Defence Capability Development Manual (DCDM) was released in 2005,
amended in 2006, and at the time of the Committee hearing was in the process of
being revised by Defence. The DCDM provides ‘authoritative guidance’ on the
implementation of the strengthened two-pass approval process for Capability
Development Group (CDG) staff. CDG are responsible for various aspects of the
capability development process within Defence.
9.13
The ANAO’s analysis of 20 sample projects proposed since reform to the
two-pass process found a ‘range of deviations from the guidance in the 2006
DCDM’. The Chief of CDG advised the ANAO that processes had evolved since the
2006 DCDM, which meant that there had not been strict adherence to the manual.
The Chief of CDG also advised that the DCDM was developed by Defence to provide
generic guidance to desk officers, but in practice the process is tailored for
each project.
9.14
The DPR states that ‘(A) strong mandatory two-pass system should provide
a precise and understandable process for the procurement of defence
capabilities, which ensures that government will be presented with robust
proposals’.[10] The ANAO notes that
adopting a tailored, project by project approach is not inconsistent with this:
Nevertheless, where a tailored approach is to be adopted on a
project by project basis, it is still important to ensure that key elements
required to produce sound proposals to government at first and second pass are
clearly identified and executed and that the rationale for, and the elements
of, the approach to be applied in a particular project are clearly authorised
at an appropriate level. It was not evident that Defence consistently applied
this level of discipline in relation to the approaches taken to developing the
first and/or second pass approval submissions to Government for the projects in
the ANAO’s audit sample.
9.15
While not making a recommendation in this area, the ANAO considered that:
Defence should ensure that the revised DCDM clearly
identifies for CDG staff the key elements that are required to produce robust
proposals to Government at first and second pass, and provides guidance on the
process to be followed to obtain authorisation for the particular approach to
be pursued in the context of an individual project.[11]
The Mortimer Review and the
Government response to the Review
9.16
The Mortimer Review argued that, to improve efficiency, the two-pass
approval process should become more flexible and the number of times projects
went to Government should be dependent upon project cost, maturity, complexity
and risk.[12] The Government agreed
with the Review’s recommendation that ‘Government approval of major Defence
projects should occur through a tailored application of the two-pass process’, however
in their response highlighted the strength and flexibility of the existing
approach. The Government’s response also noted that the current two-pass
approval process ‘already allows for combined First and Second Pass approval
for less complex projects and for more than two considerations where projects
are particularly complex’.[13]
Committee examination
9.17
The CEO DMO was asked for an overall appraisal of the effectiveness of
the two-pass approval system. He observed the Mortimer Review findings that the
process worked well but in some cases additional or fewer passes were required.
A hypothetical example of a project that would only require Government
consideration once would be a follow-on buy of an asset already in operation. Examples
were provided of projects that have or will go for Government consideration more
than twice:
n Joint Strike Fighter
will likely be considered by Government four or five times;
n Air Warfare Destroyer
went to Government seven times.[14]
9.18
The Committee asked the ANAO what is required to ensure a flexible
approach to capability development is acceptable. The new Defence Capability
Development Manual (DCDM) would be central to outlining requirements:
In the course of the audit, [Vice Admiral Tripovich, Chief of
Capability Development Group] did communicate to us that he was in the process
of developing a new Defence Capability Development Manual and that would look
to give guidance to CDG staff such that they would understand what to do in the
flexible design to fit the particular project environment. Our only concern was
that the guidance to staff made clear to them what elements are mandatory and
what elements are not mandatory—in certain circumstances it may be that none
are mandatory—that the process for particular projects is authorised at a
sufficiently senior level and signed off and that, when things are done at the
direction of government, the advice from government is recorded, clear and
available.[15]
9.19
The ANAO was not opposed to having flexibility in the process, however
stressed that ‘there needs to be agreement in government on what is required
and what is not required’. Defence acknowledged the need to correctly identify
and record reasons for taking certain decisions. They felt, however, that the
decision to take a tailored approach was captured in advice to the Minister and
their subsequent agreement that a deviation was necessary.[16]
9.20
It was noted by Defence that most of the ANAO’s sample projects would
have deviated in their process from the guidance in the DCDM, and the ANAO
added that there was no evidence that recorded the authorisation to tailor the
approach. Defence was confident that despite a lack of records, decisions to
tailor approaches were properly made.[17]
9.21
When Defence noted that relevant capability development desk officers do
not themselves determine when to deviate from DCDM guidance, the Committee
questioned Defence regarding the ANAO finding that there was no evidence
tailored approaches had been clearly authorised. Defence stated that desk
officers are guided by internal committee processes which are minuted:
ANAO observed that we did not capture in that or in any other
form every single decision to deviate. We had quite a robust discussion about
the level of recording that should be necessary. I have taken that on board and
it is obviously important that, if at any time government wants to look back
and ask, ‘Why were certain things done?’, we need to be able to provide the
evidence.[18]
9.22
The Committee questioned whether the DCDM was generic guidance,
highlighting the manual states that it is ‘authoritative guidance’. Defence
acknowledged that it was generic guidance, and noted that the revised manual
would incorporate all lessons learned from the various reviews undertaken:
But it will always be the case that it will paint, if you
like, the generic model, but it must always be tailored for the particular
project. Otherwise we will be bound by process.[19]
9.23
Defence gave assurance to the Committee that a new DCDM will set out the
approval process in cases where projects need to deviate from that listed in
the manual. Defence added that the process will continue to evolve.[20]
9.24
It is appropriate to modify the process to approach Government, provided
that this is initiated or agreed to by Government. Defence needs to ensure that
officials involved in preparing submissions are aware of the requirements of an
authorised tailored approach. Clear, formal records of deviations from standard
process are needed.
Recommendation 15
|
9.25
|
The Committee recommends that when preparing submissions Defence
develop a procedure to ensure that any divergence from the generic guidance
provided in the Defence Capability Development Manual (DCDM) is authorised at
an appropriate level and to record:
n any
modifications to the capability development approval process contained in the
Defence Capability Development Manual (DCDM) for a project; and
n reasons
for diverging from the generic model.
|
Description of technical risk and
presentation of cost estimates in capability development Cabinet submissions
Audit Findings[21]
9.26
The ANAO assessed whether the 20 projects in their sample met revised Cabinet
Handbook requirements, which were introduced as a result of the Kinnaird
Review.[22] Because it took CDG
eight months to supply the ANAO with what it considered to be the required
documents, the ANAO did not have time to undertake an audit of the adequacy of
the information provided to support cabinet decisions, only to review whether key
documents had been prepared. The ANAO’s analysis found three key problem areas:
addressing technical risk, whole-of-life costings, and discussion of
trade-offs. Technical risk and whole-of-life costs were key focus areas in the
Kinnaird review, and were raised during the Committee’s hearing.
Description of technical risk in
Cabinet submissions
Findings of the Defence Procurement
Review
9.27
The 2003 Defence Procurement Review (DPR) stated that:
Standardised Technology Readiness Levels should be used to
assess the technology maturity of equipment, including sub-systems, at various
stages of development. Proposals lacking technology risk ratings would not proceed
for government consideration…
Implementation of such a system enables non-technical readers
to better understand the level of technological risk of particular proposals
and therefore facilitat[e] a better assessment of their merits. We understand
that [the Defence Science and Technology Organisation] would be capable of using
this methodology to rate technology risks for new capabilities.[23]
Audit findings[24]
9.28
The ANAO’s analysis found that Cabinet submissions were inconsistent,
and some even ‘vague’, in their assessment and description of technical risk,
and used different categorisations including one (System Readiness Levels) that
Defence was unable to explain. It found that only one out of 10 second pass
submissions utilised the method to measure technical risk (Technology Readiness
Levels) recommended by the DPR and required under the Cabinet Handbook.
The ANAO recommended that Defence refine its methodology for addressing
technical risk to ensure it is clear and conforms to Cabinet Handbook
requirements.
Committee examination
9.29
The provision of advice on technical risk to government was viewed by
the ANAO as having improved over time, however it was still a work in progress.
They were also surprised at the lack of precision in some of the Cabinet
submissions.[25]
9.30
Defence agreed that the process was one of continual improvement, which
was why Defence had developed a better method of expressing technical risk than
only TRLs:
… we have grown to learn, as has the Government, that a TRL
is in itself not sufficient. The more comprehensive technical risk assessment
that now forms part of the Cabinet submission…is a more effective way than just
a TRL to explain the important technical risks and the issues that arise from
that for government.[26]
9.31
Defence agreed with the Committee’s proposition that they were ‘ahead of
the game’ in relating to addressing risk but this was not reflected in the Cabinet
Handbook.[27]
9.32
Defence Science and Technology Organisation (DSTO) provided a detailed response
to concerns over the adequacy of and approach to addressing technical risk in
Cabinet submissions. Defence produce a document called a ‘Technical Risk
Assessment’ (TRA) for each project, which ‘informs the preparation of the
Cabinet submission’. The TRA:
… starts with an assessment of a Technology Readiness Level
of all the key subsystems in that equipment and from there goes to look at how
those systems are integrated together, the suitability of the technology for
the purpose for which it is intended and the risks which may arise due to the
need to develop technology by a particular time so that you can acquire the
system effectively. All of that information is assembled into a document which
has a standard and defined format where all the risks terms are full[y] defined
and that informs the preparation of the Cabinet submission. The simple answer
is we do not use TRLs as a stand-alone indicator of risk because they do not
actually tell you anything about risk; they tell you about the maturity of the
technologies you start from.[28]
9.33
DSTO argued that TRLs are focused on the maturity of a particular piece
of technology (a sub-system) and not the technical risk of integrating the
sub-systems and delivering the integrated system. Defence are refining their
methodology on addressing risk ‘and we would expect to provide some
recommendations as to what measures should be used to better describe risk so
that the Cabinet Handbook can be updated’.[29]
9.34
In terms of the adequacy of advice, DSTO pointed out that the ANAO
focused on the Cabinet submissions, and are ‘not seeing the Defence
documentation which lies behind those Cabinet submissions’. DSTO highlighted two
publicly available publications that outline the nature of their TRAs.[30]
The Committee later examined the two public reports highlighted by DSTO.[31]
9.35
The Technical Risk Assessment of Australian Defence Projects publication
states that TRAs undertaken by Defence focus on TRLs (the maturity and
feasibility of individual technologies) during the early stages of project
development and the focus then shifts to using Systems Readiness Levels (SRLs),
which measure technical risk associated with systems, including their
integration into the one prime system, in the later stages of capability
development, with SRLs becoming the primary measure at second pass approval. SRLs
were seen to ‘augment’ the use of TRLs, however TRLs are still to be used ‘at
each decision point in the capability development lifecycle’.[32]
In some cases TRL scores could be high (low risk) but SRL scores low ‘because
of integration issues, environmental issues, interoperability and possible
dependence on other (as yet untested) technologies’.[33]
9.36
In addition, the technical risk assessment is supported by a categorisation
of likelihood (likely, possible, unlikely) and consequences of risks being
realised (minor, moderate, major), with categorisations based on the standard
AS/NZS 4369:2004.[34]
9.37
The second publication, Technical Risk Assessment: a Practitioner’s
Guide notes that using TRLs to identify the maturity of technologies for a
project is working well, however the use of SRLs is more problematic and there
is confusion about the difference between technical and technology risk and
identifying technical risks.[35]
9.38
In the hearing the ANAO emphasised that they were not purporting to have
greater expertise on technical risk than Defence and DSTO, nor criticise the
merits of Defence’s approach; their audit of technical risk was based on the
requirements in the Cabinet Handbook. They noted that:
Certainly what is being explained to us—the work that is
being done at the moment—sounds very positive in further addressing the desire
of Government for improvement in this area. We have no particular attachment to
TRLs. It is simply that they were what [were] required at the time.[36]
9.39
Before making concluding remarks, the Committee wishes to highlight the ANAO
finding that only 1 out of 10 second pass submissions used TRLs, while another
used it for some options (not the recommended option) and assigned an imprecise
rating of TRL 3-7. Two first pass submissions used TRLs (the Handbook only
mandated TRLs for use at second pass), however one used an imprecise rating of
TRL 4-8.[37] A lack of TRLs in
Cabinet submissions may reflect that they are used in documentation lying
behind Cabinet submissions.
9.40
Based on evidence to the Committee and Defence publications, there
appears to be a systematic process as part of capability development that
focuses on risk, including utilising different standardised measurements for
different stages of project development, and the Defence technical experts (DSTO)
having a key role in the process. The Committee believes Defence is best placed
to determine how they wish to rank technical risk. However, whatever approach
is utilised needs to be communicated consistently, and in a concise and
understandable manner, a point expressed in the hearing by Defence:
The important thing is that, when the government makes its
decision, Defence and government fully understand what we are getting ourselves
into, that we have full transparency of all of the issues and that they are
known and in the Cabinet submission. Government has an assurance that we have
them covered.[38]
9.41
It appears that this comprehensive approach using TRAs, including
utilising both TRLs and SRLs, may not be making its way into Cabinet
submissions in a clear, consistent and systematic manner.
9.42
In this context, the Committee encourages Defence to seek amendments to
the Cabinet Handbook to reflect their approach, ensuring that this leads
to consistent and accurate description of risk. Additionally, only one score
should be applied to each TRL and SRL, not imprecise multiple scores (such as
3-7, 4-8 or even 1-2).
9.43
Government must be fully informed on capability options in a way that is
understandable and allows comparison across projects. It is crucial that Defence
ensure that technology and technical risks are clearly stated. This will
provide a good basis with which government can make a decision on capability.
Recommendation 16
|
9.44
|
The Committee recommends that Defence work with the
Department of Prime Minister & Cabinet (PM&C) to amend the Cabinet
Handbook to accurately reflect the more specific risk measurement process
developed by Defence which should be included in submissions for both first
and second pass assessment.
|
Recommendation 17
|
|
The Committee recommends that Defence ensure that Cabinet
submissions for future major capital equipment projects provide advice that
is clear and consistent, and include the following:
n Technology
Readiness Level (TRL) and System Readiness Level (SRL) scores for each
option;
n description
of likelihood and consequence of risk based on these ratings.
The Committee
further recommends that Defence develop a procedure to ensure that staff
involved in Technical Risk Assessments (TRAs) have a clear understanding of
System Readiness Level (SRL) measurements so that the figure provided
precisely reflects risk levels.
|
Presentation of cost estimates in
Cabinet submissions
Findings of the Defence Procurement
Review
9.45
The 2003 Defence Procurement Review (DPR) stated that:
When taking decisions on capability options at first and
second pass the whole-of-life costs must be presented to and understood by
government. These not only comprise the cost of the prime equipment, but also infrastructure,
equipment operating costs, through-life-support, and the resources required to
manage acquisition.[39]
Government response to the Mortimer
Review
9.46
As part of accepting a recommendation of the Mortimer Review relating to
introducing a capability into service, Defence stated:
Capability Managers must include whole of life implications
and independent advice from the CEO DMO on the cost, risk and schedule
implications for projects in their advice to Government.[40]
Defence White Paper 2009
9.47
Under the heading of ‘better managing Defence costs’, The White Paper
stated that:
The Government has also directed that greater attention be
paid in the planning process to the whole-of-life cost dimensions of
capability. This will be critical in providing Government with increased levels
of confidence with respect to cost, schedule and technical risk when it
considers major Defence projects. This will also assist in minimising scope
variations in major acquisitions, and help drive down cost pressures.[41]
9.48
These three reports highlight the importance of, and continuing focus on,
whole-of-life costing.
Audit findings[42]
9.49
The ANAO’s analysis found that most Cabinet submissions (14 out of 23)
did not contain estimated whole-of-life costs as required by the Cabinet
Handbook. The ANAO found that Defence often used NPOC (Net Personnel
Operating Cost) estimates.[43] NPOC estimates only
reflect anticipated change in operating costs associated with replacing a
capability (where applicable), or update the estimate in the Defence Capability
Plan for new capabilities. NPOC does not provide a transparent, understandable
whole-of-life cost estimate, and may be a small number (even zero) depending on
the estimated cost relative to the current capability, or relative to Defence
Capability Plan estimates.
9.50
The DPR and the Defence White Paper 2009 emphasise the importance
of getting whole-of-life costs, as they are a significant source of cost (over
two-thirds of the whole-of-life cost of a platform will be incurred after the
platform is introduced into service). The ANAO recommended that Defence include
explicit whole-of-life cost estimates, agreed by Finance, as required by the Cabinet
Handbook.
Committee examination
9.51
NPOC was described as the extra cost (‘new money that goes on top of the
current funding stream’) to go from the current capability to the new one. The
annual NPOC cost was multiplied by the estimated life-of-type of the platform,
and added together with the acquisition cost and other costs such as
contingency to get the whole-of-life cost.[44]
9.52
When queried by the Committee how these costs differed from what the
ANAO considered to be understandable whole-of-life costs, Defence believed that
the key issue was consistent presentation to government. Defence stated that
agreement had been reached with the Department of Prime Minister and Cabinet (PM&C)
and Treasury on a format for the cost sheet that includes NPOC and
whole-of-life cost.[45]
9.53
The Committee is pleased to hear that a standard format has been agreed
to highlight both NPOC and whole-of-life costs. Again, like the technical risk
issue discussed above, it is important that such clear and consistent information
is communicated to Government in submissions. The ANAO found only nine of 23
Cabinet submissions contained comprehensive, transparent whole-of-life costs.
It appears that NPOC costs were being added to other costs to give a total
overall cost.
9.54
Using NPOC does not appear to reflect a true life-of-type costing: a
submission with NPOC costs does not tell government how much the current
capability costs to run. For ease of comparison, Defence may wish to include in
Cabinet submissions both the whole-of-life cost and also mention the NPOC,
which would be of interest to government as it is the additional money required
for personnel and operating costs.
9.55
To support the views on whole-of-life cost presentation found in the
ANAO’s recommendation, the Defence White Paper 2009 and the Government’s
response to the Mortimer Review, the Committee would emphasise to Defence that
when presenting whole-of-life costs to government, they are clear, consistent
and include the total estimated personnel and operating costs, not only
variations from current costs. The Committee urges Defence to bear this point
in mind when implementing Recommendation No. 4 of the ANAO report.
Involvement of the Department of Finance in verifying cost estimates in
Cabinet submissions
Findings of the Defence Procurement
Review
9.56
The 2003 Defence Procurement Review (DPR) found that:
Finance agreement to Cabinet submission costings is generally
sought shortly before lodgement, and does not allow sufficient time for any
in-depth analysis of capability, strategic, technical, legal or commercial
issues associated with the costs and risks of major capital investments or
other procurements. Ideally, Defence should provide well developed business
cases for the proposed investments, based on sound costings models, for
validation by Finance.[46]
9.57
The review stated that Finance ‘should be involved much earlier, and on
a continuous basis, throughout the two-pass approval process so that they can
contribute to effective quality assurance in relation to costings and risk, and
technology readiness respectively’.[47]
Government response to the Mortimer
Review
9.58
The Government’s response to the Mortimer Review states that reforming the
Capability Development process to ‘provide Government with more reliable
information on which to base judgements and a more efficient and effective
capability development process’ will be achieved by, amongst other initiatives ‘ensuring
early consultation with the Department of Finance and Deregulation on options,
cost estimates and project risks’.[48]
Audit findings[49]
9.59
Defence advised the ANAO during the audit that it had implemented reforms
in this area in response to the Kinnaird Review’s findings. The ANAO found,
however, that Defence and Finance did not have an agreed process to facilitate
Finance’s involvement with capability proposals. Each Department also had
different perspectives on how effectively Defence engaged with Finance. Defence
disagreed with Finance’s view that it does not receive relevant information
early in the process, only receives costing and risk information 30 days or
less from a submission to NSC, and that this information is often limited and
of questionable quality.
9.60
The ANAO stated that it is important for both Departments to develop a
common understanding on an appropriate approach to engagement. The ANAO also
stated that the absence of agreed procedures and guidance for Defence
engagement with Finance means that:
Government may not be consistently provided with the level of
independent evaluation of capability development proposal costings envisaged by
the DPR. In practice, this has meant that of the 23 submissions to government
reviewed by the ANAO in this audit, four went to government containing cost
estimates that Finance had been unable to agree to. The Cabinet Handbook states
that ‘each first and second pass submission or memorandum requires agreement
with Finance on the detailed acquisition and operating costings and financial
risk assessment’.
9.61
Defence agreed to the ANAO’s recommendation that ‘Defence agree with
Finance a suitable approach to allow Finance’s early and ongoing involvement in
the evaluation of capability development proposal costings’. In their response
to the recommendation Defence stated:
Defence will work to formalise the process for engagement
with DoFD [Department of Finance and Deregulation] (and PM&C and Treasury).
Committee examination
9.62
When asked about the different views between Defence and Finance on
engagement around costings, Defence reiterated their perspective, expressed in
the audit, that Finance was sufficiently engaged early in the process. Defence
noted that after the audit report was released, they engaged at a SES band 2
level (equivalent to a Rear Admiral) with Finance, PM&C and Treasury to
come up with an acceptable agreement on engagement, which had been established.[50]
9.63
The Committee sought comment from Finance in response to Defence’s
comments. The Department of Finance and Deregulation confirmed that it ‘has
agreed an approach with the Department of Defence to allow Finance early and
ongoing involvement in the evaluation of capability development proposal
costings’.[51]
9.64
Defence stated that while they always try to get agreement with Finance
on costings before submissions are lodged, they are not always successful. In
such cases, the Minister can decide to take the submission forward, accepting
the disagreement. The ANAO accepted this, noting that in some circumstances
there may be disagreement that needs to be resolved by Cabinet. The audit was
following the requirements of the Cabinet Handbook, which required
Finance agreement. Of the four cases where Finance did not agreed to the
costings, in only one case did government decide to defer consideration because
of the disagreement.[52]
9.65
The Committee is keen to ensure that costings receive adequate scrutiny by
Finance, and part of this is ensuring they receive not only early engagement
but also receive the Cabinet submission costings early enough to allow sufficient
time for scrutiny. The Committee is pleased Finance has an agreed approach to
allow sufficient scrutiny of capability development proposal costings.
Records management within Defence
Audit findings[53]
9.66
The ANAO analysed the DCDM (high-level guidance) and CDG’s Process Map
(detailed guidance) to identify the key activities and documents required under
the strengthened two-pass approval process for 20 sample projects. Because it
took CDG eight months to supply the ANAO with what it considered to be the required
documents, the ANAO did not have time to undertake an audit of the adequacy of
key documentation, instead they simply reviewed whether key documents had been
prepared.
9.67
The ANAO was quite critical when outlining the overall results of the
analysis:
Defence was unable to demonstrate… that the procedures
outlined in the DCDM and the Process Map have been consistently followed…or
that alternative procedures were appropriately authorised, managed and
documented.
CDG could not provide the ANAO with final versions of around
half of the key project documents requested…
In most cases, the ANAO was unable to determine whether
Defence’s inability to provide requested documents was a consequence of poor
records management practices or because the documents had not been prepared.
Defence does not have an adequate audit trail of key decisions and requirements
in relation to the capability development process.
9.68
The ANAO stated that significant improvement in CDG’s record keeping was
needed and recommended that ‘Defence develop, promulgate and implement a sound
records management policy within CDG’.
Committee examination
9.69
In the hearing, the ANAO summarised the need for good records:
… as the National Archives say in the guidelines that they
put out, you need to have the documentation so that people know that the
records are genuine, that they are accurate, that they can be trusted, that
they are complete, that they have not been altered, that they are secure, that
they can be found when they are needed and that they relate to the relevant
documents. For us, that was the heart of the CDG records management issue.[54]
9.70
Defence’s progress on implementing the ANAO’s recommendation was sought,
and they provided an update to the Committee:
… since the audit came out, we have, within CDG, promulgated
formal policies on our document and management system, a formalised
document-naming convention to make it easier to recall things from the system
and a process for signing off documents and keeping signed copies. Training has
commenced across the 250 people that work in my organisation. I will have a
change of some staff over Christmas and it will be firmly part of our annual
training continuum thereafter. By about mid-2010, I will have done a compliance
audit, an internal check, to see if people had been doing what they were told
to do. As I said, the training has started to roll out. It is very difficult to
try to recover documents that you cannot find now.[55]
9.71
Defence’s management action plan for responding to audit recommendations
was completed by December 2009.[56] The Committee is hopeful
that this action will mean that the significant problems with records access
encountered by the ANAO will be ameliorated in the future.
9.72
Defence stated that a key contributor to poor records management is ‘it
is very difficult to find documents on Defence’s ICT system’.[57]
In this context the Committee notes that the ANAO has recently commenced an
audit into Defence’s ICT systems.
The relationship between the
Performance Audit and the Major Projects Report
9.73
In addition to addressing issues raised in the audit report, the
question was asked about the relationship between the performance audit report
and the ANAO’s assurance report Defence Materiel Organisation Major Projects
Report (MPR).[58] Concerns were raised
that issues significant enough to be found in the Planning and Approval of
Defence Major Capital Equipment Projects audit were not raised in the MPR
review.[59]
9.74
The ANAO stated that the MPR was a different report from a more in-depth
performance audit, and each report focused on different aspects of a project. The
performance audit was of the approval process, while the MPR review covers the
post approval process, where responsibility for procuring a capability is
shifted to DMO. The assurance provided in the MPR is more limited than for a
performance audit.[60] Only one ‘pilot’ MPR
report has been released so far, and a key outcome of an annual MPR report is
to document for Parliament trends in major projects.[61]
9.75
When asked whether the MPR document is a useful and reliable document
given the lower lever of assurance, the CEO DMO stated:
I think the major projects report is very useful and
reliable. I think we had a discussion in this committee several years ago about
what the major projects report was supposed to achieve. We were faced with
either doing 30 performance audits at the higher level of approval…or giving a
summary year by year. What we are doing is giving a longitudinal summary so
that you can see for every year how things have changed: how the risks have
changed and how the dollars have changed. It is a very different process indeed
from what [Audit Report 48] was about, which was examining the two-pass process.[62]
Conclusion
9.76
Asked whether the findings of the audit sample (20 projects) could
reasonably be expected to be found in other Defence projects, the ANAO agreed
that the sample findings:
… would lead us to believe that there would be other projects
that exhibited those characteristics and some of those may be in the current
bundle of work.[63]
9.77
Defence agreed that similar issues could arise in projects from the same
era, however:
I would like to think, though, that, if you picked one that
was just about to go to cabinet this week, for example, you would not find
those things. I would be very confident that ANAO would see a completely
different document and a completely different process as a result.[64]
9.78
From the evidence provided in the hearing, there appears to have been
considerable effort and action undertaken by Defence to improve their approach
to capability development proposals. Major Defence procurement is a very
complex area, and one that has received significant attention over time. It
also takes time to bed down new processes and determine whether reforms have
been implemented successfully.
9.79
One theme arising from the Committee’s inquiry is a need for renewed
effort in communicating to government consistently and clearly. Defence appear
to have established a sound process internally to assess technical risk, and it
is important that this work is adequately captured in submissions to Cabinet.
Government also needs clear information on the whole-of-life cost of capability
options, including total personnel and operating costs.
9.80
Documentation and records management are important in maintaining the
integrity of the capability development process. Defence must be able to
deviate from the process outlined in the DCDM if directed by government. In
such cases, staff within CDG must be informed as to what is required. In other
cases of deviation not at the direction of government, a record must be made
that includes sufficient explanation why the process is different and
containing senior authorisation.
9.81
Finally, appropriate engagement of the Department of Finance and
Deregulation is critical, as it provides government with independent scrutiny
of Defence costings.
9.82
The ANAO’s audit has proven to be highly valuable in examining Defence’s
progress in implementing reform properly and consistently, particularly as the
nature of the information that goes to government receives little other public
scrutiny. Given the findings of the ANAO, Defence internal reviews and external
reviews such as Mortimer, and Defence assurances that processes will be
improved, there is merit in the ANAO conducting a similar review to Audit No.
48 in the future, one that will cover issues addressed in the Committee’s
inquiry. Defence as well as the ANAO stated in the hearing that Defence is on a
path of continuous improvement; it is appropriate that Defence’s progress is
monitored.