Chapter 5

Chapter 5

Requirements stage

Overview of the requirements stage

5.1        The Defence Capability Development Handbook (DCDH) identifies two goals of the requirements stage. Firstly, first pass approval by government to allow further investigation and refinement of the option(s) that will 'satisfy the identified capability need'.[1] The other goal is that of second pass approval by government to 'acquire and implement an agreed capability that fulfils the capability requirement identified in the Defence Capability Plan' (DCP). The DCDH notes that this approval will include a 'defined acquisition budget, schedule and level of performance, and a budgeted whole-of-life cost and workforce requirement'.[2]

5.2        The Kinnaird, Mortimer and Pappas reviews identified key challenges in relation to the requirements stage. They included: the efficiency of the two-pass process for new acquisitions; balancing cost and risk including identification and mitigation of technical risks; benchmarking off-the-shelf options; accountability and responsibility for program management; skills and expertise in capability planning; strengthening the Capability Development Group; and understanding whole-of-life costs.

Process

DCP entry to first pass

5.3        Once a project commences (following entry into the DCP), a number of documents are developed to inform the final Ministerial Submission (MINSUB) or Cabinet Submission (CABSUB) provided to government for approval. The information required by government at first pass includes a business case for each capability option which must identify the:

5.4        In order to reach first pass, a substantial number of documents are prepared starting with a Project Management Plan (PMP) which outlines what is to be done, when, by whom and at what cost. It also identifies the risks and responding mitigation strategies. Thereafter, the key information required prior to first pass and key steps in the process to obtain it include consideration of the scope, operational parameters, option set for a project and how the project should be tailored to suit its needs.

5.5        Part of the pre-first pass documentation preparation process will also include scoping studies and industry engagement to establish availability of product in the marketplace, and provide an indication of whole-of-life costs and innovative options that might be available. The formal pre-first pass solicitation mechanisms include the following:

5.6        In terms of establishing technical risks, DSTO develops a Technical Risk Indicator (TRI) to determine the feasibility of the technology to provide the capability being proposed and identify any potential areas of significant risk. The TRI will also identify high technical risks associated with any options and address differing risk profiles that arise with each capability option (i.e. military-off-the-shelf, Australianised).[4] In addition, a draft Materiel Acquisition Agreement (MAA) and Acquisition Strategy, which identifies the preferred alternative for procuring and implementing each capability system beyond second pass, must be developed.

5.7        Kinnaird, Mortimer and Pappas recommended that a military-off-the-shelf (MOTS) alternative be part of any set of options put to government to ensure as Kinnaird noted that a 'benchmark is established against which the costs, military effects, and schedule of all proposals can be assessed'.[5] Conversely, all reviews found that any requirements set beyond that of off-the-shelf equipment generate what Mortimer described as 'disproportionately large increases to the cost, schedule and risk of projects'.[6] They would therefore need to be based on 'a rigorous cost-benefit analysis of the additional capability sought against the cost and risk of doing so'.[7]

5.8        In light of these recommendations, the 2009 DWP states that MOTS and commercial-off-the-shelf (COTS) solutions will be the benchmark against which a rigorous cost benefit analysis of the military effects and schedule aspects of all proposals will be undertaken.[8] The DCDH reconfirms the DWP by noting that where an off-the-shelf option exits, it will be presented to government and be the benchmark against which a rigorous cost-benefit analysis of any additional capability is sought, taking into account the cost and risk in doing so. The DCDH explains that when an off-the-shelf option is 'judged not to exist', this will be explained in the first pass submission to government.[9] In relation to an Australianised option, the DCDH states that any option proposing the Australianisation or modification of off-the-shelf equipment must detail the rational and associated costs and risks. The DCDH also makes clear that the first-time integration of a number of separate off-the-shelf systems is no longer an off-the-shelf solution and must be considered 'developmental'.[10]

5.9        Defence noted that, as indicated in the 2009 DWP, it is seeking to drive down the costs of ownership of military capability and that part of this drive will include, where appropriate, 'focus on military- and commercial-off-the-shelf equipment'.[11] In relation to Mortimer's recommendation 2.3 concerning the provision of cost-benefit analysis of any option that is not off-the-shelf, however, the minister stated in May 2011 that Defence would 'accelerate implementation' of this recommendation which is yet to be fully implemented.[12] 

Second pass

5.10      Once government has approved a capability proposal at first pass, the options agreed by government will be further refined. The key activities to achieve second pass include the development of:

Structure

Capability Development Group

5.11      CDG takes carriage of the requirements stage and is responsible for developing options for government consideration at both first and second pass. The Capability Systems Division (CS Div) Desk Officers within CDG manage this process. They are responsible for leading the Integrated Project Team and bringing together people 'from the capability manager, from DSTO, from DMO, maybe CIOG [Chief Information Officer Group] if required, and maybe then might hire contractors to provide some professional support as well'.[14] In fulfilling this role, the CS Div Desk Officers develop capability options and manage the development of the project document suite which includes the Capability Definition Document (CDD) for each option; Workforce Plan and cost estimates; Science and Technology Plan with DSTO; draft MAA with DMO; and the capability proposal from which the MINSUB/CABSUB is produced for government consideration at first and second pass.

5.12      Prior to submission at first and second pass, the CS Div Desk Officers are responsible for ensuring that the necessary documentation is considered by a number of internal Defence committees starting with the Options Review Committee (ORC). Chaired by the Chief of CDG, the ORC will provide direction on capability options that should be developed for first pass consideration.[15]

MOTS benchmarking and options

5.13      Part of the role of the CS Div Desk Officer is to ensure that a MOTS option is provided as a benchmark against which other options can be considered by government at first pass. However, evidence provided to the committee suggests that whilst the 2009 DWP requires the inclusion of a MOTS option for each project, the requirement will often translate into a hypothetical off-the-shelf option or local construction of an off-the-shelf design.[16] Other evidence highlighted the consequences for domestic industry of a MOTS purchase in terms of viability and skill base with concerns raised that MOTS:

5.14      The Rizzo Report highlighted that the benchmark must take into account the longer-term costs of COTS/MOTS acquisition whereby Defence loses engineering capacity which then carries costs in terms of project and capability failure with amphibious ships being a case in point.[21] He also noted that long-term costs in terms of actions required to rebuild the technical capability some years down the track should also be considered. Rizzo further recognised that the combination of MOTS and the Commercial Support Program (CSP) has implications for Defence personnel and technical competence as well as domestic industry. The committee is interested to establish, therefore, what elements of industry capability are in fact strategic capability that should not be lost to overseas suppliers.  This goes to the question of moving elements of industry division to CDG and having this decision made as part of first pass approval.

5.15      Other considerations in relation to MOTS include interoperability as well as compliance with Australian regulations including health and safety requirements. As overseas providers and governments have different risk tolerance to that of the Australian authorities, a purely MOTS acquisition can therefore place Defence in a position of either being non-compliant with Australian law or having to accept additional cost and modification to make the equipment compliant. Sometimes the non-compliance is inconsequential and yet because so much is driven by process rather than decisions made by informed people, significant cost and waste results from attempts to modify equipment unnecessarily. On the other hand, sometimes the true costs of required upgrades are not factored in because the process assumes that a US Army or Air Force product will automatically be suitable for ADF use.  It has been suggested to the committee that the concept of an Airworthiness Board-like review of CDG business cases will allow informed, independent corporate knowledge to be applied to the issue early on in the development cycle.  

5.16      The committee recognises that there is considerable debate around the MOTS benchmarking requirement including the suggestion that the requirement has turned into a preference for MOTS. The committee also acknowledges the wider options debate and the impact of MOTS procurements on the domestic defence industry. The committee intends to consider these debates in its main report.

Independent analysis and engagement with industry

5.17      Kinnaird and Mortimer recommended the use of resources on early analysis including funding industry studies and gathering the best 'commercial advice on acquisition options'.[22] In accordance with these recommendations, in the pre-first pass phase, the CS Div Project Officers can draw on Project Development Funds to develop the CDD, conduct technical and trade studies, market studies, costing studies and analysis.[23]

5.18      Another consideration in relation to the capability development, procurement and sustainment options is that of the capacity and sustainment of Australia's defence industry. Mr Graham Priestnall of the Australian Industry and Defence Network noted of industry that:

The sustainment of ADF capability, an area where many SMEs operate, is the greatest cost to the government in acquiring and maintaining capability, yet this draws relatively minor focus and analysis within all ongoing reviews.[24]

5.19      Despite recommendations by Kinnaird, Mortimer and Pappas, industry representatives argue that this remains an area that draws relatively minor focus and analysis during the capability development lifecycle and in ongoing reviews.[25] Moreover, the perception is that through-life support and sustainment experts in industry are not involved in the design phase of a developmental project. The committee intends to consider the evidence that sustainment of industry is not a central consideration in the context of through-life costs and capability sustainment. Additionally, the committee is interested in establishing an understanding of how such considerations are articulated and of the relative importance given to them.  

5.20      In relation to engagement with industry more broadly, Defence recently reinvigorated its environmental working groups to facilitate early informal engagement in order to establish what is available on the market and gather ideas from industry. These forums provide an opportunity for Defence and industry to discuss DCP projects during the requirements stage leading then to formal engagement on the draft tender document. CDG will invite industry comment on the draft tender document in order to ensure that only specifications that can be delivered are included in the final tender document.[26]

Defence Science and Technology Organisation

5.21      DSTO provides the technical and technology risk analyses required at first and second pass. In light of findings and recommendations by Kinnaird, Mortimer and Pappas regarding the need for greater consideration of such risks, DSTO has been mandated to 'provide external evaluation and verification of project proposals'.[27]

5.22      The Joint Decision Support Centre which is a CDG–DSTO initiative is a forum where methodologies can be applied to assist decision makers within CDG to look at particular concepts or options that might arise in a DCP project. According to Dr Ian Sare, Deputy Chief Defence Scientist, DSTO provides the staffing for the centre which is tasked by CDG CS Div Desk Officers to conduct studies to assist them in preparing the formal documentation for government consideration. In this regard, Dr Sare noted that the centre 'has been a very effective way of trying to apply objective operational analysis-type methodologies to assist decision support'.[28]

5.23      The DSTO provides detailed analyses of the technical issues in relation to the options that might be then brought forward to government for consideration.[29] In relation to this process, Dr Sare informed the committee that the role of DSTO is to provide advice and:

We will then frequently do studies and analyses to investigate the feasibility of different options that might then deliver a capability. We will utilise our best knowledge of what is happening in the broader community. We have, for example, very strong international defence science links with the US, the UK, Canada and New Zealand.[30]

5.24      As previously noted, a TRI forms part of the documentation required by government at first pass. The TRI identifies the key systems with which the proposed options will need to interact to deliver the required capability. It  can also note any developmental system or technology that needs to be developed in time to meet the proposed schedule and which could potentially provide greater capability than those options previously identified.[31] Air Marshal Harvey further explained the TRI process:

What happens is that you have a broad study to understand the field and what might be possible. You will talk in general terms about what the technical risks are and that helps to inform the options review committee in order to determine which options to pursue.[32]

5.25      Whilst the TRI provides an early indication of risk, the Technical Risk Assessment (TRA) provides a detailed assessment of technical risks and issues associated with each option in the capability proposal prior to first pass consideration. A number of internal Defence review committees will consider the DSTO risk assessments from which the Chief Defence Scientist will draft a Technical Risk Certification for inclusion in the MINSUB/CABSUB at first pass as appropriate.

5.26      Pappas referred to DSTO's involvement in pre-approval assessments of major acquisition projects but was of the view that there was 'scope for more constructive involvement'. Importantly, he noted that DSTO's involvement and assessments are 'not always paid the respect they should be'. He went on to say:

Scope and specification changes make the conduct of a Technical Risk Assessment very difficult and there does not appear to be consistent criteria that determines the degree of initial and ongoing DSTO involvement in retiring technical risk in projects.[33] 

5.27      He urged closer cooperation which would, among other things, assist project teams to understand the 'grounds for risk assessments and the potential ways to reduce/mitigate the major risks'.[34] Pappas also made a number of recommendations directed at more effective management of technical risk.

5.28      Preparation for second pass approval involves detailed assessment by DSTO of the options the government has agreed to pursue. This assessment includes the identification and execution of risk treatment and issue resolution activities that may involve industry as well as the preparation of statements of technical risk.  Documentation produced by DSTO on risk includes a second pass TRA, a Science and Technology (S&T) Plan for second pass and a final TRC for inclusion in the MINSUB/CABSUB as appropriate.[35]

Defence Materiel Organisation

5.29      The Mortimer Review noted that the DMO is responsible for delivering military equipment to the ADF according to cost, schedule and specifications agreed to by government. Mortimer argued that in order to be held accountable for such delivery, the DMO must provide independent advice to government on matters within its remit. Indeed, the review specifically recommended that the DMO be responsible for the 'equipment acquisition strategy throughout the requirement definition process'.[36] To this end, Mortimer recommended that the CEO of the DMO provide independent advice to government on cost, schedule, risk and commercial aspects of all major capital equipment acquisitions at DCP entry and at first and second pass.[37] Moreover, in order to be able to answer the government's questions on these matters, he recommended that the CEO be a permanently invited adviser to government committees considering Defence equipment acquisitions.[38]

5.30      In keeping with Mortimer's recommendations, the DMO is progressively more involved as a project moves through the requirements phase. DMO's responsibilities include direct support in developing the CDD, cost and schedule estimates, and in undertaking risk reduction studies whilst evaluating proposals and offers from industry.[39] Indeed, consistent with Mortimer's recommendation 2.7, the DMO is responsible of the acquisition strategy throughout the capability development process with the CEO of the DMO responsible for signing off on the strategy. Furthermore, in accordance with Mortimer's recommendation 2.10, the CEO of DMO provides independent advice to the Defence Minister and Cabinet on the costs, schedule and other commercial aspects of military equipment procurements in each capability development Cabinet submission.[40]

5.31      The DMO is also represented on the four internal Defence committees responsible for reviewing and endorsing options at first and second pass to ensure that a 'One Defence' view is offered to government. Such representation includes either membership or permanent invitation.[41]

5.32      Whilst the Defence committee system was established to strengthen the two-pass process and provide greater rigour and scrutiny of projects across Defence, the Black Review found that committees 'tend to function to confuse accountability, blur strategic focus and disperse decision-making capacity'.[42] The findings that committees serve to diffuse individual accountabilities will be considered in greater depth by the committee.

Capability Managers

5.33      Capability Managers are responsible for 'raising, training and sustaining force', and have an overarching role across the capability development cycle to ensure that it all comes together as a complete capability. Furthermore, as defined in the Defence Capability Development Handbook (DCDH), Capability Managers have a 'far greater role right up front on capability development and a far greater say over the development of those projects as they come into service'.[43]  Indeed, according to Chief of Air Force and Capability Manager, Air Marshal Brown, as the Capability Managers are ultimately accountable for a capability, they will follow the project from start to finish.[44]

Department of Finance and Deregulation

5.34      In response to recommendations by Kinnaird, Mortimer and Pappas concerning greater certainty of costs and a strengthened two-pass process, the Defence Capability Assessment Branch was established in 2004 within the Department of Finance and Deregulation (Finance) Budget Group. The branch is staffed by specialist cost analysts responsible for evaluating the costs and financial risks associated with Defence capability procurement proposals. In order to implement the Kinnaird recommendations, Finance agreed to a guide that defines the standards by which the Budget Group assesses the cost and risk estimates associated with major Defence acquisition proposals at first and second pass. Finance then provides advice to its minister on these costs and risks.[45]

5.35      The Finance Minister is a member of the Expenditure Review Committee and, according to Finance, is often coopted to the NSC to consider major capability proposals of $100 million in value or greater that have been brought forward by Defence for government consideration.[46] The Defence Minister and Finance Minister jointly consider proposals with a total project value between $20 million and $100 million, unless either minister refers the project to the NSC.

People

CDG CS Div Desk Officers

5.36      CDG brings together all available advice from DSTO, DMO, Capability Managers and industry in order to develop the CABSUB/MINSUB for the government to consider at first and second pass.[47] This includes information emanating from the CDG-DSTO Joint Decision Support Centre which applies 'objective operational analysis-type methodologies' to assist CDG in decision making'.[48]

5.37      In order to fulfil this role, Mortimer emphasised the importance of CDG being adequately resourced in terms of 'workforce numbers and skills to develop capability proposals and incorporate specialist advice' from DMO and the DSTO.[49]

5.38      Indeed, of five principal areas of concern identified by Mortimer in relation to the procurement process, inadequate project management resources in CDG and shortages in DMO personnel were listed amongst them.[50] Mortimer held that the accuracy of information provided to government would be improved if CDG was better and more appropriately resourced and skilled to consult and consider expert advice.[51] Evidence before the committee, moreover, suggested that CDG CS Div Desk Officers were not adequately trained and lacked appropriate supporting management structures, processes and tools to carry out their role.[52] Furthermore, they did not have experienced independent individuals with the necessary corporate knowledge and ability to capture lessons learned to lead, guide and mentor them.

5.39      Mortimer noted that core personnel in CDG were military officers on short term postings with an average of 18 months in an area where the work was complex. Mortimer argued in favour of extending the duration of postings to CDG and recommended that CDG and the DMO both deepen their expertise in cost and schedule estimation and project management as a matter of priority.[53] Pappas had similar concerns, noting that the short assignments of CDG desk officers created continuity problems in acquisition program management as multiple desk officers could be responsible for the evolution of the specifications of a single platform.[54] High turnover of staff within CDG and DMO was also raised in evidence as a problem in relation to the consequent difficulties for large primes and SMEs.[55] Other submitters noted that CDG personnel are primarily military officers who might bring military experience and expertise to the technical aspects of proposals whereas their primary role in CDG is project management and administration. The ANAO noted that this dynamic coupled with a lack of training and management support given to CDG desk officers 'particularly hampered their ability to understand complex cost and schedule estimations for the capability proposals'.[56] In relation to this matter, the committee notes Mortimer's recommendation that in order to make effective use of the technical, engineering and commercial expertise received from DMO and the DSTO, CDG may require in some areas additional personnel with specialist expertise.[57] The committee questions whether this recommendation has been fully realised.

5.40      In response to these findings and recommendations concerning skills development and support to CDG CS Div Desk Officers, Air Marshal Harvey observed that CDG managers currently consider 90 per cent of the desk officers to be sufficiently skilled to perform their assigned duties without additional support. Furthermore, a structured CDG Desk Officer skilling program has been implemented to address core capability development skilling. Air Marshal Harvey informed the committee that the program provides an annual 'induction course and then a flexible, progressive skilling program to address project and individual needs'. Air Marshal Harvey concluded:

Realisation of the full benefits of the initiatives will be protracted, due to the extended procurement life cycles, which are typically two to five years just in the approval process.[58]

5.41      Furthermore, he stressed that training is currently underway to address the skills shortages in relation to cost estimations.[59]

5.42      However, Defence also acknowledged challenges in relation to attracting and retaining qualified and skilled staff to progress projects through the capability life cycle. Air Marshal Harvey detailed the initiatives in place to address these challenges:

Several skilling and professionalisation strategies have been implemented to enhance the skills base of Defence and DMO workforces. Identified skills shortages are being addressed via education and training, targeted recruitment and employment schemes and above-the-line contractor support, when necessary. As part of our commitment to improve our performance, Defence and DMO have introduced a professional industry standards certification framework for procurement and contracting staff. This includes a continuing professional development or CPD program.[60]

5.43      In addition to training initiatives, Air Marshal Harvey informed the committee that CDG was 'looking at higher pay for specific individuals who are particularly valued by the organisation' as well as securing industry support and increasing the numbers available within CDG.[61] He explained that where baseline funding is not adequate, CDG can bid for resources through the Workforce Financial Management Committee to the CDF and Secretary of Defence.[62]

Committee evidence

5.44      While reforms and improvements have been made to strengthen the two-pass process, the committee received considerable evidence to suggest that problems and challenges remain. CS Div Desk Officers play a crucial role in drawing on and synthesising often highly technical information into key documents for government consideration and it is in relation to their role that this evidence is largely directed.

5.45      Such concerns included the following:

5.46      The committee raised these questions with CDG during two days of public hearings. It sought evidence from Defence and specifically CDG in order to establish the extent to which CDG has a demonstrated ability to draw on and process technical, engineering and commercial expertise and advice received from the DMO, DSTO and industry. Whilst the committee appreciates that efforts and improvements have been made in relation to attracting and retaining qualified personnel, skill development and training, it recognises that challenges remain which must be addressed. The committee intends, therefore, to consider these matters in greater depth.

5.47      One of the consistent themes in evidence before the committee concerned the level of engineering skill and input across the capability development and acquisition life cycle. Some submitters recognised this challenge as symptomatic of a general deskilling across Defence and consequent dilution of technical support services and engineering skills.[68] Much of the discussion regarding engineering and technical skills focused on the DMO and the Services. Even so, the committee appreciates that engineering input would be essential at the needs and requirements phase to the extent that such skills would enable CDG to have a greater appreciation for, and ability to, interpret input from industry and DSTO. This is another area for further committee consideration.

Capability Managers

5.48      One of the issues raised in evidence and taken up by the committee during the hearings with Defence was the extent to which the centralisation of resources to CDG (and the DMO in relation to acquisition and sustainment) and away from the Services has affected the ability of Service Chiefs to manage the capability process. Furthermore, evidence to the committee suggests that under the current governance model, the Capability Managers are not and cannot be accountable and that this is one of the fundamental flaws in governance that has prevented successful reform.[69]

5.49      ANAO's recent audit regarding Navy highlighted the impacts of centralisation.  ANAO found that the removal of capability personnel from Navy into CDG to manage capability proposals diminished the ability of Chief of Navy as Capability Manager to obtain guidance and assistance in relation to the requirements determination and how it fits into the process.[70] Ms Fran Holbert, Executive Director of ANAO's Performance Audit Services Group described the consequences:

It meant that the technical regulatory framework had stepped back and was not engaged with the procurement framework being operated through DMO. It is not clear how so much distance could have arisen given that there are Navy people in Navy, in DMO and in CDG. But it is the case that gaps opened up in the knowledge those different areas had about how risks were developing and what was going on with them.[71]

5.50      The committee recognises the substantial work undertaken by the ANAO in this regard and as centralisation is a key theme emanating from the evidence, the committee expects to consider it and its significance in a subsequent report. 

Defence industry

5.51       Some submissions to the inquiry argued that the NSC's annual approval rate has dropped to 10 projects rather than the expected rate of 50 projects a year.[72] The consequences for industry of such a slippage include increased cost as project teams are formed and disbanded and aggravation of the 'already severe problem of uneven workload'.[73] However, Defence rejected the suggestion that there had been slippage in the approval rate. Defence maintained that whilst a project may 'slip a month or so in terms of scheduling with the cabinet', it was not aware of any material delays with any implications for delivery of capability at all over the last few years.[74] The committee recognises a serious disconnect between Defence and industry's view of the approval rate, which is affecting industry's ability to plan and up-skill as well as the working relationship between Defence and industry. Indeed, the position of Defence is that industry must plan to ensure that it can play its part.[75] This divergence in understanding and the respective information underpinning it together with its consequences will be considered in greater depth by the committee at a later stage.

5.52      These considerations, the impact of MOTS procurement on the domestic industry in terms of viability and skills maintenance, the tension between security needs and industry needs, schedule delays resulting from slow supply from industry, and the challenges for the defence industry will also be considered in greater detail by the committee.

Navigation: Previous Page | Contents | Next Page