Chapter 11

Findings on tender design

11.1      This chapter outlines the committee's findings and recommendations on the design of the tender. That is, the technical requirements and tender conditions set out in the request documentation. In particular, the committee is required under its terms of reference to consider the following matters relevant to the design of the tender:

• the requirements of tenders and how effectively these will be met;^[1] and
• the methodology and adequacy of the decision process and whether the services to be supplied in the contract were determined on the basis of objective and supportable, current and likely future requirements or were structured so as to unfairly advantage a particular respondent.^[2]

11.2      During the course of the inquiry a number of matters were raised not only with respect to the design of tender documents but with the process of preparing and releasing tender documents and in the evaluation of tender responses. The committee considers the adequacy of communication and consultation, the quality of the documentation, including the TEP and the SER and the checking and verification of calculations in the SER. It looks at key issues raised before the committee pertained to:

• the decision to re-tender;
• the technical specifications, in particular the cargo volumetric capacity and preference for a single aircraft solution;
• the tender response period; and
• the tender evaluation criteria and their application.

Decision to re-tender, specifications and timeframe

11.3      It was alleged by Strategic that the first three matters did not reflect operational need, were not commercially justifiable in terms of value for money, and were designed to disadvantage the existing operator in favour of a foreign, broker‑based solution.

The evidence

11.4      Having regard to the factors cited by Defence, the committee is satisfied that there was a sound business case for re-tendering the 2008 contract. Defence identified an opportunity to achieve significant cost savings as a result of the global financial crisis which created excess capacity in the commercial air charter industry; a decline in demand for international passenger air travel, shrinking aviation industry profitability, idle aircraft and falling charter rates.[3] It also noted that re-tendering was appropriate in light of the numerous amendments to the 2008 contract which 'had significantly altered' the agreement.[4] These changes were in areas including fuel allocation, routing, block hours flown, pricing structure, the aircraft used and consequent load-splitting arrangements in relation to cargo.[5] The committee notes the findings of the Deloitte review that the decision to tender was 'based on a reasonable expectation of achieving a better value for money outcome for the Commonwealth'.[6] The AFCD review stated that the decision to re-tender was based on 'valid commercial and operational considerations'.[7]

11.5      The committee is also satisfied that the design of the RFT was supported by operational need, and was not intentionally designed to advantage or disadvantage individual respondents. It accepts Defence's line of reasoning that there was sufficient evidence of operational need to justify the increased cargo volumetric requirements in item 7.13 of the request, and the preference for a single aircraft solution.

11.6      Indeed, the balancing of these considerations was legitimately a matter of discretion for Defence as purchaser. In the absence of evidence suggesting that the decision to include these requirements was unreasonable, or was based on incorrect (or no) factual information, there would be little utility in the committee making a different assessment of how operational needs should be reflected in the tender requirements. In this respect, the committee notes the findings of the AFCD and Deloitte reviews that the tender requirements were functionally specified in the request, and that consideration was given to cargo-splitting solutions put forward by respondents. The committee also notes the evidence of Defence that the 2010 contract has achieved significant financial savings on the previous contract, and that the additional aircraft capacity has been utilised in the services performed to date.

11.7      The committee makes clear, however, its ongoing interest in this matter. It is concerned to ensure that the technical requirements identified in requests for the provision of air sustainment services—whether under the current contract or under future contracts—continue to be based on documented evidence of operational need. Accordingly, the committee requests that Defence provide it with periodic reports on the ongoing performance of the 2010 contract, including in the realisation of projected savings, the continuing need for the increased cargo volumetric requirements and the contractor's compliance with the tender requirements (see recommendation 8).

11.8      The committee accepts the evidence suggesting that the eight-week tender response period was reasonable. While the limited timeframe inevitably meant that some respondents would not be in a position to meet the aircraft certification requirements, the committee acknowledges that this factor must be weighed against competing considerations. These include, for example, operational needs and the costs associated with granting a longer response period to allow potential tenderers to obtain certification or source aircraft as necessary. Accordingly, the committee considers that the decision to require an eight-week response timeframe was reasonably open to Defence.

Areas for improvement—documentation, consultation and certification

11.9      Both Strategic and the external reviewers raised issues concerning the tender evaluation criteria and their application. The external reviews identified a range of process issues, including a lack of clarity as to the meaning of individual criteria; the potential for duplication of criteria; and the standard of documentation recording the evaluation processes and outcomes. The committee considers that there is scope to improve future practices in the procurement of services from the Air Transport Standing Offer Panel, including:

• procurement planning, particularly the identification of tender requirements;
• communication with potential tenderers;
• the evaluation of tender responses; and
• developing the source evaluation report.

Procurement Planning

11.10         The committee considers that procurement planning could be improved in two respects. First, key planning decisions should be documented at a level of detail that is commensurate to the level of probity risk. In particular, the business case for re-testing the market should be documented thoroughly. The committee notes the findings of the AFCD review in this regard.[8] In light of proponent grievances about the tender requirements discussed above, the committee also observes the importance of documenting in detail the evidentiary basis for these requirements. Rigorous documentation is particularly important where the revised requirements are greater or more stringent than those of the current contract.

11.11         Secondly, it is important that future procurement plans make sufficient contingency for decisions to re-test the market. Such decisions must be made as early as possible before the expiration of the current contract to allow sufficient lead time for a re-tender if required. The committee is concerned that the time pressures bearing upon the 2010 tender process contributed significantly to the problems encountered. It questions whether the transition between the 2008 and 2010 contracts could have been better planned. The committee acknowledges that the tight timeframes in the 2010 tender process were driven by a contract commencement date that was aligned to operational requirements in the MEAO. However, it questions whether the decision to re-test the market and the identification of the tender requirements could have been made earlier. This would have minimised the time pressures on the 2010 tender process.

Communication with potential tenderers

11.12         To manage the risk of proponent grievances, the committee considers that communication with potential tenderers could be strengthened in several respects. First, future procurement strategies should continue to include a requirement that standing offer panel members are provided with advance notice of any decisions to re‑tender, prior to the release of the request. As noted by the Deloitte Review, this was not adhered to in the 2010 tender process. In the committee's view, early notification may assist potential tenderers in meeting narrow timeframes and help prevent perceptions that such timeframes may be motivated by uncommercial interests.

11.13         Secondly, the committee notes the importance of providing a clear and consistent explanation to potential tenderers of how Australian industry participation is assessed in the tender evaluation process. That is, its consideration within the overall assessment of value for money. The correspondence between Strategic and 1JMOVGP in April 2010 indicates that there may be some industry confusion about this matter.[9]

11.14         Thirdly, the committee considers that aspects of the tender process could be better communicated to potential tenderers, to ensure clarity of understanding and minimise the risk of potential proponent grievances. These include:

(a) providing potential tenderers with an explanation of the reasons for re‑tendering and the changed tender requirements;

(b) providing potential tenderers with an explanation of how the evaluation criteria referred to in the request document will be assessed; and

(c) including in the request documentation an express statement of Defence's preference for a single aircraft solution (or any other preferred solution that is identified in future requests), and the fact that alternative solutions will be considered.

Tender evaluation

11.15         The committee notes the shortcomings identified by the external reviews in the development of the tender evaluation plan, the tender evaluation criteria, the documentation of evaluation results and other matters of process in relation to the tender evaluation. Such failings increased the likelihood of mistakes such as the possibility of double counting the same issue because of the lack of clear guidance in the TEP.[10] The committee endorses the observations made by the external reviews and, in particular, considers that Defence should take action on the following matters:

(a) In the development of the TEP, there should be early and ongoing consultation with relevant line areas within Defence on the evaluation processes—in particular, advice should be obtained from FIS on the agreed pricing model and the financial evaluation processes.[11]

(b) The request documentation and the deed of standing offer should identify expressly:

(i) provisions which are minimum conditions of participation, minimum form and content requirements or other essential requirements; and

(ii) how Defence will treat completed requests that do not comply with requirements expressed in mandatory language.[12]

(c) Future TEPs should document the following, to ensure a transparent, consistent and complete assessment of all tender responses:

(i) a full outline of the tender evaluation methodology, in particular the scoring methodology and pricing models adopted by the TEB and the TEWGs;[13] and

(ii) detailed guidance as to how the tender evaluation criteria are to be applied, to ensure that there is no overlap between criteria, such that issues are assessed multiple times.[14]

(d) The tender evaluation process should be improved through the following measures:

(i) ensuring effective consultation between elements of the tender team during the evaluation process—particularly between the technical and financial TEWGs;[15]

(ii) requiring the TEB to produce a separate report on its initial compliance assessment against the evaluation criteria for which it is responsible;[16] and

(iii) ensuring that the SER transparently and accurately communicates the evaluation processes, and explains the outcomes in detail. In particular, future SERs should include the matters identified by the AFCD and AGS Reviews, and subsequently incorporated into the revised SER.[17]

11.16         As noted earlier the committee had two particular concerns relating to the evaluation process—the observation about unequal treatment on the leasing arrangements with the respective aircraft owners and a mathematical error. The committee notes that the concerns about the unequal treatment were rectified in the final SER.

11.17         In its submission, Defence noted that the TEB was reconvened to re-validate the SER. In the process a mathematical error was discovered and reversed the order the second and third ranked tenderers.[18] When asked about the consequences stemming from this mistake, Mr Brown agreed that it was a good point. He stated:

If it had been close, which it was not—and thankfully we found it before the contract was awarded, as a result of this investigation the gap was such that it did not matter, but it did swap two to three on a fuel calculation. It was a computation within the spreadsheet where the error was found. Again it goes to the point of the probity auditor, 'Have you had an audit done of the Excel spreadsheet?

11.18         Along similar lines, Dr Watt told the committee that 'the real difference is that the tenders were chalk and cheese'—'that is the thing that is hard to get away from in all of this'.  The committee notes, however, the statement in the AFCD review, noted earlier, that there 'was very little difference between the top three ranked tender options'.[19]

11.19         This mathematical miscalculation and the confusion over the unequal treatment of the leasing arrangements highlights the critical importance of having in place robust protocols and important safeguards governing the evaluation process. The committee is of the view that Defence should ensure that tender documentation, especially in respect of scoring and pricing models is clear, unambiguous and detailed, that there is transparency and consistency in the TEP and that calculations are independently verified. 

Recommendation 5

11.20         The committee recommends that Defence:

• In line with the findings of the AFCD Review, considers strategies for the improved documentation of the business case for any future decisions to re-test the market for the provision of air sustainment services to the MEAO.
• Reviews its procurement plan for the current MEAO contract, to ensure that sufficient lead time is provided for the making of any future decisions to re-test the market, and the planning and execution of a procurement process.
• In all future procurements of air sustainment services to the MEAO:

(a) continues to include in procurement strategies a requirement that members of the Air Transport Standing Offer Panel are given advance notice of any decisions to re-tender the contract, prior to the release of the RFT; and

(b) ensures that such requirements are implemented.

• Implements strategies to ensure that potential tenderers have a clear and accurate understanding of how Australian industry participation is taken into account in the evaluation of tender responses, as part of the overall value for money assessment.
• On the release of future requests for air sustainment services to the MEAO, implements the following actions to minimise the risk of potential proponent grievances:

(a) provides potential tenderers with an explanation of the reasons for re‑tendering the contract and any changes to tender requirements from the previous request;

(b) provides potential tenderers with an explanation of how the evaluation criteria in the request documentation will be assessed; and

(c) includes in the request documentation, where applicable, an express statement of Defence's:

(i) preferred solution for meeting tender requirements, including technical specifications; and

(ii) intention to consider alternative solutions.

• As a matter of priority in future tender processes for the provision of air sustainment services to the MEAO, takes action on the tender evaluation issues identified by the Deloitte, AGS and AFCD Reviews, as documented at paragraph 11.15 of this report.

Conclusion

11.21         The committee found that the decision to re-tender was sound and supported by operational needs and that Defence could justify the change in specifications from the previous contract. It is also of the view that although the time frame for tenderers to prepare and lodge their tender was tight, it was not intended to deliberately disadvantage any tenderer.

11.22         In respect of the quality of documentation, consultation and certification, the committee found much room for improvement. Because of failings in this area, the committee noted the potential for confusion, inconsistency and errors in calculations during the evaluation, and in some cases the realisation of this potential. Although the SER was re-validated and confirmed the successful tender as the top ranked tenderer, the problems identified in the process cannot help but to undermine the committee's confidence in the robustness of the decision.

Navigation: Previous Page | Contents | Next Page