Chapter 2Key Issues
2.1The evolving nature of the risks facing Australia has driven a rapid expansion in the scope and complexity of the capabilities required for the nation’s ongoing security. For a long time, peace and security in the region were underpinned by the unipolar leadership of the United States of America (U.S.). Growing regional power competition now has the potential to compromise Australia’s interests and security.
2.2Concurrently, these evolving risks have reduced the time available for capability procurement and have increased supply chain vulnerability. Security can increasingly be threatened in multiple ways, as identified by the DefenceStrategic Review (DSR). These are not limited to direct military attack, with more countries now able to threaten trade and supply routes which are vital for Australia’s prosperity.
2.3The Defence Capability Assurance and Oversight Bill 2023 (the bill) attempts to respond to these risks by enacting reforms intended to ensure that Australia’s defence is supported by a new test and evaluation system.
2.4Of the eleven submissions received, most submitters supported the intent of the bill, albeit with amendments. They also identified several technical concerns with its contents and instances where greater clarity was necessary. This chapter explains these concerns and ends with the Committee's conclusion and recommendation.
Test and evaluation
2.5T&E is a structured process for obtaining objective information on both risk and quality. A witness appearing before the Committee cited the process of obtaining a driver’s licence as involving a form of T&E most people would be familiar with:
The most common form of test and evaluation, which almost all Australians are familiar with, involves the all-important journey to obtain a driver's licence. Driving instructors are independent and certified. They test to a standard to ensure a consistent standard and practice for driving on Australian roads. The instructor usually provides the coaching and instruction and has conducted the testing on a continuous assessment basis. We've come a long way from our parents teaching us what they gleaned on the road and being tested by the local bobby or the motor registrar.
2.6Similarly, in the defence context T&E aims to establish capability risk potential in the design or manufacture of military equipment before it is used:
The sound application of T&E during development is intended to identify and remediate deficiencies prior to acceptance of a product from the vendor.
2.7High standards in T&E are vital to both military capability acquisition and the systems which surround and support the deployment of this equipment:
Historically, military T&E has focussed on assessing aircraft, ships, other vehicles or weapons as self-contained entities, broadly referred to as platforms. Platforms do not provide capability in isolation, they need support and sustainment systems to be effective.
Capability assurance in the U.S. and U.K.: a different approach
2.8Dr Keith Joiner CSC, a senior lecturer in T&E with a long career in the field, submitted that U.S. and U.K. perspectives on capability assurance have created a distinct difference in approach to T&E between those countries and Australia.
2.9In the case of the U.S., its government established the independent office of the Director, Operational Test &Evaluation (DOT&E), while in the U.K. there exists a long-term partnership with industry to provide T&E.
2.10Some witnesses suggested that Australia is held back by a lack of independence in T&E processes, as well as inadequacy in setting competent standards, a lack of commitment to adequate funding and training, and a culture that is resistant to delving into and transparently reporting risk.
2.11For example, Dr Joiner stated with respect to workforce challenges:
Globally research and practices in these fields have moved dramatically since the 20th Century dogma on which the majority of Defence capability acquisition and assurance is practised. Defence largely internally promotes an aging acquisition and assurance workforce with self-consultation about competencies that do not embrace global practices, like agile requirements development, software development and testing, test design, modelling and simulation, cybersecurity, and digital engineering.
2.12The Committee sought to understand whether the perceived failure to make the required investment in T&E stemmed from a cultural problem. DrJoiner suggested that this was the case, with people in charge of projects often not wanting to know what the risks involved are, for fear of losing funding or missing a milestone. Dr Joiner cited an example:
I'll give one quick example. AIR 9000, I think, was the Navy's helicopter project. We were buying, or we had selected, a helicopter that was used to ferry people out to oil rigs. So that was marinised, used and certified to oversee a city. The Police Force were flown around in it. On that basis, they said, 'We don't need to do preview testing on that helicopter.' I started at the project director level and then went to a two-star who was a Navy commodore and flight-test qualified who was not interested in letting testers go and fly that before we signed the contract. The two-star wasn't interested. The then head of the DMO, Warren King, was not interested. They feared that we would find something that would delay signing that contract and the funding would go to somebody else.
In the end we came to a compromise with Admiral Jones in a separate group. We paid the $43,000 for 10 flying hours for three staff from AMAFTU [Aircraft Maintenance and Flight Trials Unit] to go and fly that helicopter. That taught me a lot about the cultural issues. You don't always get that sort of level of serious interference, but it is about risk management. There is something wrong in the culture of Defence when people consistently want to hit milestones for fear of losing their reputation and promotion, I guess, as opposed to what we ultimately found, which was a bunch of things that the contract negotiators were able to get into the price. If you know anything about that project, it's 26 years worth of maintenance and support out of Nowra, so every little thing that you find before you go into contract and have as part of the baseline price pays huge dividends over 26 years.
2.13Dr Joiner agued that the U.S. decision to establish its Directorate of Operation Test and Evaluation (DOT&E) also enhanced its own T&E processes:
Despite having ‘Operational’ in the title, under Title 10 Congressional Law, DOT&E licenses and approves all test and evaluation planning and appointments across the whole lifecycle, reports annually to Congress, and provides funding to cover significant test processes, education and infrastructure improvements. This independence has enabled DOT&E to help its Defence respond more quickly to cybersecurity threats and use modern, efficient modelling, simulation and test designs in their acquisition. Importantly, their influence has created three passes of Government approval compared to Australia’s two passes, whereby contracted development can only proceed to low-rate initial production until an operational assessment is complete. This extra milestone avoids over-commitment in costly production until proper testing of prototypes and the associated risk disclosure, generally with schedule and cost advantages.
2.14Furthermore, U.S. agencies responsible for the leading edge of T&E infrastructure and practices receive direct funding and policy support for use in Defence. In contrast, Dr Joiner described the areas responsible for T&E in Australia as ‘chronically underfunded’ and with ‘minimal policy and test infrastructure.’
These deficiencies are despite two decades of significant related ANAO reports, Defence’s Roadmap for Test and Evaluation, and a Senate Inquiry into Defence Procurement.
2.15Dr Joiner supported the component of the bill that would establish the DCAA:
If the Government does not legislate and create some form of CapabilityAssurance Agency with authority and independence, our military commanders, positioned as they are in a very diverse and interdependent organisation, will continue to be self-determinant, dogmatic and mostly dated, unknowing and inefficient in the way they assure their capabilities.
2.16The notion that there needs to be more independence in T&E received strong support from submitters and witnesses. As put by the International Test and Evaluation Association’s Southern Cross Chapter:
We are, however, firmly of the opinion that there should be adequate independence between people and agencies performing test and evaluation and those who are directly responsible for the cost schedule and acceptance of acquisitions. We believe this is necessary to protect against undue pressure and influence.
2.17Nova Systems Australia and New Zealand, an engineering services and technology solutions company, cited the U.K. example, where an independent oversight body has been successfully established.
2.18Nova emphasised that independence is expected in analogous fields, again drawing parallels with the independence of driving instructors who also test prospective drivers’ skills on the road:
We in society accept the driving instructor as an independent trainer, adviser and examiner. Society doesn't take issue with any conflict of interest in these roles when all the instructor's efforts are independent of the parties who might benefit. On balance, Nova would be in favour of an independent entity to raise the level of T&E, to advise on T&E matters and to sustain and steward T&E capability as a whole for the long term. We feel it will have a strong positive impact on the assurance and timely delivery of defence capability at a time when that is incredibly important to the nation and the safety and success of our warfighter. In doing so, it should also act to reduce costs.
T&E in Defence
2.19Several witnesses highlighted failings in the Defence T&E process. Representatives appearing on behalf of the International Test and Evaluation Association Southern Cross Chapter, for example, pointed to evaluation practitioners being pressured to change their findings when those findings were at odds with the expectations of supervisors or those higher up in the procurement chain:
I think it's a sad fact that every test and evaluation practitioner who has had a long career and has been involved in test programs of significance has been subjected to those types of pressures. It's a sad fact that, when the community comes together at association dinners and the like, the stories always turn around to 'when I tested something and somebody didn't like the answer, and the consequences that I experienced on a personal basis’.
2.20The committee noted that the witness cited above inferred that unfavourable testing conclusions and recommendations may at times have been hidden, or ‘interpreted differently from how they were intended to be interpreted’.
2.21Another witness, Mr Mark Schweikert, a T&E consultant with extensive professional expertise working with Defence, pointed to a lack of T&E input in the requirement-setting stage of Defence’s T&E process. MrSchweikert implemented a course designed to teach T&E so that its theory and application would be properly understood.
2.22The committee also heard in-camera evidence on a recent Defence acquisition process where transparency was not provided to decision makers that project requirements were not being met. Another example the committee noted in-camera concerned undue consideration given to the subjective advice of other parties in preference to the recommendations of T&E practitioners working within an approved T&E entity.
2.23Defence has itself acknowledged problems in the way in conducts T&E and put to the Committee that it has taken significant steps to address them. The department submitted that it has been executing a new T&E strategy since 2021 which has steadily improved T&E competency and transparency, that it has instigated ‘modest partnership’ with industry, and has deepened understanding of T&E in a manner that will serve as a foundation for further improvement.
2.24The current Defence T&E governance model is based on principles similar to those outlined in the bill: independence, competence, transparency and accountability. Defence therefore broadly agrees with the intent of the bill and agrees that the four core principles are essential to ensuring that capability decisions are suitably risk-informed by the T&E process. It was clear from evidence received by the Committee that Defence is not at all blind to the problems identified, and has been taking significant steps to address the issues along similar lines to those suggested by many witnesses and submitters.
2.25However, Defence does not support the establishment of the DCAA for the reasons outlined below.
Potential problems with the proposed DCAA
2.26Defence submission states that establishing the proposed DCAA may:
…undermine officers’ ability to assure that acquired systems and capabilities meet the integrated, multi-domain requirements and integrated force missions in accordance with the intent of DSR and Australia’s strategic circumstances.
2.27Defence submitted that, as the proposed DCAA would apply only to acquisition-related T&E, it would not contribute directly to Operational T&E (OT&E) or Force-Level OT&E (FLOT&E). This could result in a split in responsibilities between Defence and the proposed DCAA, thereby disrupting the ‘thread of assurance designed into the Defence T&E Governance Model’. The split would also, in Defence’s view, erode the focus on realising strategic capability objectives which exists through the structure of the Joint Warfare Development Branch.
2.28Defence emphasised the importance of ownership over all forms of T&E in terms of the impact on accountability:
I think it's important that Defence retains the ability to set the policy for conduct of its own T&E to enable the accountability of the Joint Force Authority…It's the part around the assessment and the auditing policy et cetera that Defence would wish to retain in terms of being able to have the Joint Force Authority and the other capability managers meet their accountabilities.
2.29Furthermore, Defence sees a risk that the proposed DCAA could have a negative impact on the prioritisation of T&E resources, the transfer of knowledge and efficient coordination with industry. This would, Defence states, ‘delay the realisation of integrated joint force capability, as directed by the DSR, at a time when we are trying to accelerate delivery of minimum viable capability.’
2.30Defence drew the Committee’s attention to the steady progress the department believes it has made in improving T&E through its recently implemented T&E Strategy, highlighting achievements relating to overarching policy, training, risk assessment and the workforce.
2.31The Committee heard that the T&E policy now requires acquisition projects to consider the use of T&E as early as possible to reduce risk. It also requires decision-makers to be informed of capability risks identified by T&E.
2.32The new strategy also covers training:
Each year, Defence trains approximately 120 members through the internal test and evaluation practitioners course, with another 10 to 15 completing more extensive external courses. Industry also train approximately 50 of their employees annually. Our full financial year 2022–23 reporting showed that, of the 466 staff employed in T&E duties, 58 per cent were fully competent and an additional 24 per cent were on track to be qualified following the completion of on-the-job training. The quarterly community of practice online events attract around 200 participants, and 961 individuals registered to attend the annual T&E conference in 2023, which was up from 666 the previous year.
2.33In terms of risk assessment, Defence is now using ‘a capability risk assessment mechanism as an independent channel to report T&E outcomes’:
It will capture a project's susceptibility to technical deficiencies, knowledge gaps to be filled, and capability risks identified through T&E, supplementing the Defence Science and Technology Group's technical readiness level assessments and also the established contestability process.
2.34Defence has also created a new ‘Joint Warfare Development Branch’ within its Force Integration Division, bringing together ‘joint concepts, joint experimentation, joint mission design and Defence test and evaluation to create a continuous thread of integrated capability assurance across the capability life cycle.’
2.35At the same time, Defence noted that its new model diverges from the bill in terms of operational test and evaluation:
The bill focuses on T&E performed in support of the acquisition process, while Defence's model also incorporates oversight of operational testing at both an individual capability level and a joint force level. The DSR recognises the criticality of integrated joint capability such as multidomain strike in deterring potential adversaries. Such integrated capabilities are made up of multiple systems working together as a system of systems. Therefore, force-level AT&Es are needed to confirm that the combinations of those capability systems operate effectively in an integrated way. As a result, Defence has introduced a Defence Force integration assurance function within the T&E model to achieve this end.
2.36Other submissions similarly raised the prospect of the DCAA in fact duplicating tasks already being conducted by Defence. Mr Mark Schweikert posited that risking duplication ‘could add time and complexity where leadership, clarity, speed and flexibility are needed most.’
Workforce implications
2.37The Committee explored the potential impact the establishment of a new agency might have on Defence in terms of what is already a stretched workforce, that is, whether the DCAA would pull expert staff, who are in scarce supply, away from Defence. Responses to this question were mixed.
2.38Defence identified the supply of qualified T&E practitioners as a potential problem arising from the bill, since the proposed DCAA would likely draw staff from the existing national T&E workforce, while Defence would rely on the same workforce to retain T&E expertise:
The current shortfall of qualified T&E practitioners, across both Defence and defence industry, was estimated under the study to be 450, growing to 1000 by 2030. The creation of the DCAA and IGDCA will impose additional demands on the already constrained T&E workforce that will need to be carefully managed during implementation.
2.39Representatives from the Southern Cross Chapter of ITEA (ITEA) were of the view that any initial challenges could be overcome, and the system would find its equilibrium:
In regard to would the new agencies take people away from their current roles, we're of the opinion that if the bill went through there would be a form of harmonisation against the existing workforce. We note that VCDF group is already pursuing an agenda of increased T&E governance, including standing up new roles in that organisation. So we would be of the opinion that if the bill went through there would be some sort of harmonisation between the agencies that are being set up at the moment by VCDF and what would be brought by the bill. In terms of whether there would be more workforce pressures, going back to our training and competency focus of our association, if the bill went through and this increased opportunities for Australians to access better quality T&E training, then perhaps the workforce would be more productive, more efficient. Whilst there would probably be an initial hump to get over whilst the new agency is bedded into their roles, if we had better access to training we'd have better trained people and perhaps our workforce would be more productive. That would be our hope.
Collaboration with industry and the risk of conflicts of interest
2.40The bill provides for a long-term agreement with an Australian industry partner capable of providing domain expertise and facilitating consistency in T&E across the capability cycle. This measure is designed to support independence, and the industry partner would perform this role through four key functions set out in the explanatory memorandum to the bill:
(1)Regulatory. The DCAA will regulate the qualifications and professional standards of the Defence or other industry workforce tasked to conduct T&E.
(2)A T&E centre of excellence. The industry partner will provide a small but highly experienced workforce with competence in T&E across operational domains (land, air, maritime, space and cyber).
(3)Training. The industry partner would be responsible for the coordination and efficacy of T&E training (e.g.: approving training providers and specific courses).
(4)T&E infrastructure. The DCAA will identify T&E infrastructure (e.g.: weapons and acoustic test ranges or electro-magnetic test facilities such as stirred mode chambers, or environmental test and virtual test facilities) required as part of procurement or through-life capability management.
2.41The explanatory memorandum to the bill states that the partnership would ‘provide depth of domain expertise and facilitate a consistent, comprehensive approach to T&E across the capability life cycle for all Defence systems and environments.’
2.42Concerns about the role of an industry partner in setting T&E standards and exercising the regulatory functions of the DCAA were commonly raised in the committee’s hearing on the bill.
2.43The Maritime Union of Australia supported the provision for a collaborative role for industry, noting that this would bring the benefit of the commercial maritime industry’s highly qualified workforce and:
…[draw] on the commercial expertise of relevant industry sectors such as the maritime sector, where there resides a wealth of expertise in ship survey, naval architecture, shipbuilding maintenance and repair, ship chartering, ship operations, ship propulsion systems as well as port operations.
2.44Defence, however, was concerned about allowing a commercial entity to function as the T&E regulator while also being a training and service provider, stating that this could create a conflict of interest and have a detrimental effect overall.
2.45Defence added that, in its view, an industry regulator might:
…undermine the accountabilities of the Department of Defence, which would lose the ability to balance regulatory compliance against capability outcomes in circumstances of heightened strategic tension as described in the Defence Strategic Review.
2.46The department emphasised the importance of the Commonwealth being able to manage risk by ‘retaining accountabilities and the levers to fulfil them.’
2.47The Committee explored this further with Defence in a public hearing, pointing out that any service provision by an industry partner would be limited in nature. Defence clarified that its concerns centre on the level of market influence the industry partner might have and the risk of creating a monopoly:
[The] concern is around the disproportionate market influence that could be afforded to an industry entity if it was appointed as a regulator. Our view is that that might cause some disincentives for wider industry entities to invest in and grow some of those sovereign T&E capabilities. Part of that would also be that the divesting of a regulatory role may risk conflict of interest and potential monopolisation, notwithstanding your comments.
2.48The Committee sought clarity on Defence’s position regarding the potential for conflict in the proposed model, that is, if a private entity is appointed to manage the sector and set standards. Defence expanded on its concerns:
An organisation appointed to set the standards and regulatory function—I don't think we would have any issues with that. It's if that same organisation has influence over the provision of regulatory services over and above that regulatory function. If it is an industry partner, and thereby being in industry it has commercial motivations, if there's a conflict between those commercial motivations and how it regulates the enterprise, that is the foundation of our concern on that. An entity setting regulatory standards in itself is sensible. I'm trying to think of an example in any other industry where a commercial entity is the regulator.
Unintended consequences
2.49The Committee explored whether the partnership with a single industry partner proposed by the bill could conceivably create a monopoly, where one company controls the T&E sector, and have an anti-competitive effect because of this reliance on a single company to perform regulatory functions in assessing qualifications and experience standards.
2.50Nova Systems recognised the risk of unintentionally allowing a monopoly, but suggested that an antidote to this could be built into the proposed legislation:
We acknowledge that there is a clear risk of a monopoly situation being created if you just approach it on the basis of an existing entity being there and in its own right being the source of all capabilities and expertise as that industry partner. To assure the independence of the industry partner, as proposed by the bill, we see that there is a need for the industry partner to in fact be a special purpose vehicle that has no other function than to fulfil the needs of the DCAA. In parallel, however, to ensure that you don't end up with a closed industry environment, a monopoly, you actual should build into the legislation the requirements for continuous market testing and refresh—that is, you assign to it the requirement legislatively for a permanent Australian industry capability gateway and refresh process to draw in expertise as is needed from other places and at other times. You cannot just have a single closed-door operation for an extended period of time. That is not what is in the interests of Australian industry. What is in the interests of Australian industry is a clear means by which commercial resources can be brought to bear. The best way to do that is through that special purpose industrial vehicle.
2.51The Committee sought views on whether appointing one company might effectively allow that company to pick who among its competitors can participate in the sector, and the disadvantage this might create for small and medium-sized enterprises.
2.52Nova Systems argued that oversight of what the proposed industry partner is doing would be of paramount importance and suggested strengthening the bill with the aim mitigating against an anti-competitive effect:
This again is where the board structure becomes absolutely fundamental to oversee and watch what that industry partner is doing. It's also where the nature of the special purpose vehicle, as structured to be that industry partner, how it is put together, its charter, its purpose and the roles that are delegated to it under the legislation become the guiding principle there. Again, this is why I highlight that an ideal addition to the legislation as it stands is to set up the mechanism by which there has to be that level of ongoing industry engagement.
Parliamentary oversight of DCAA
2.53The Parliamentary Joint Committee on Defence proposed by the bill would perform an independent oversight role.
2.54It should be noted, however, that the government has recently agreed to establish such a committee, the Joint Statutory Committee on Defence. Establishing this committee was a recommendation of the Joint Standing Committee on Foreign Affairs, Defence and Trade (JSCFADT), made in November2018; however the government’s agreement to establish the committee came in response to a different JSCFADT report. Further work to determine implementation details for this new committee will be undertaken in due course.
2.55Defence submitted that the new committee has the potential to fill the independent oversight role proposed by the bill. The Committee is not aware of any reason the new committee would not be able to perform the functions outlined in the bill. These would include, the explanatory memorandum explains, ‘specific oversight over capability acquisition, which would include Australia’s involvement in AUKUS, and the operation of the DCAA and the IGDCA’.
Committee view
2.56The Committee notes that submissions received, including from individuals with significant expertise in T&E, observed concerns with Australia’s defence capability assurance system. The Committee also acknowledges the question that exists as to whether Defence will be able to resolve the issues it faces internally, or if the system would benefit from the establishment of an external, independent agency.
2.57The Committee agrees with submitters and witnesses that capability decisions made by Defence must be appropriately informed by robust T&E processes and that those processes must be informed by the principles discussed in the EM to the bill, namely: independence, competence, transparency and accountability.
2.58The Committee was concerned to hear from witnesses of allegations where those principles of independence, competence, transparency and accountability were not upheld to the highest standards. The Committee considers such failures to not just be failures of process, but failures to adequately consider the needs of the war-fighter and the rightful demands of Australians to see those funds spent in their name spent with due consideration to best practice in capability acquisition. It also notes that this is not the first time that these concerns have been raised, whether through parliamentary processes or elsewhere. That such concerns appear to continue to be unresolved is not acceptable.
2.59The Committee is encouraged by Defence’s own acknowledgement of deficiencies with respect to T&E and the work it has done to address these deficiencies with its new T&E strategy, which specifically considers many of the themes raised by submitters and witnesses.
2.60Nonetheless, it also received evidence which indicated that elements of Defence’s own T&E strategy, such as the consideration of T&E as early as possible in capability acquisition, have not been properly observed. This would indicate that Defence still has much more work to do not just to improve its T&E processes, but also the widespread acceptance and adoption of such processes.
2.61The Committee acknowledges that most submissions were supportive of the bill, but also notes that the inquiry only received a small number of submissions, meaning that the views expressed in them may not represent views that are universally held. It also notes that perspectives among submitters and witnesses differed as to whether the bill would address their concerns about Defence’s existing T&E processes adequately. Whilst it is the case that throughout the Committee’s hearings some submitters who appeared as witnesses did, through greater explanation of the content of the bill by its author, have some of their concerns alleviated, it should be noted that significant amendments appeared to remain necessary in order to give effect to this stated intent.
2.62Whilst little of the Committee’s inquiry was concerned with the second major component of the bill, namely that to establish a new joint committee of the parliament to oversee Defence, the Committee welcomes the AlbaneseGovernment’s commitment to establish a new Joint Statutory Committee on Defence, which will fulfill many of the same functions as the committee proposed in the bill.
2.63The Committee supports elements of the bill’s intent in principle, but it retains specific concerns about the provisions in it. It is particularly concerned about the risk its passage would present by way of unintended monopolistic and anti-competitive effects through the DCAA’s selection of an industry partner to support it in its functions.
2.64The Committee is also cognisant of defence procurement reforms that signal a shift to minimum viable capability, which is about supporting developmental projects and recognising the need for rapid acquisition, principles that are at odds with the content and effects of the bill. This does not mean that the Committee disagrees with the contention that an accurate assessment of risk through appropriate T&E processes is an essential element to ensuring timely acquisition, rather that it is mindful that the time involved in establishing such a body as the DCAA would unnecessarily dampen the rapidity of acquisition in time of significant geopolitical pressure in our region.
2.65On balance, the Committee considers that the bill should not be passed. However, this does not mean that the Committee is entirely satisfied that despite the work done by Defence to improve T&E processes there is still not more work that needs to be done. It will continue to monitor this matter in the future and reserves the right to make further comment should promised systemic improvements not eventuate.
2.66The Committee recommends that the bill not be passed.
Senator Raff Ciccone
Chair
Labor Senator for Victoria