Chapter 11
Findings on tender design
11.1
This chapter outlines the committee's findings and recommendations on
the design of the tender. That is, the technical requirements and tender
conditions set out in the request documentation. In particular, the committee
is required under its terms of reference to consider the following matters
relevant to the design of the tender:
- the requirements of tenders and how effectively these will be
met;[1] and
- the methodology and adequacy of the decision process and whether
the services to be supplied in the contract were determined on the basis of
objective and supportable, current and likely future requirements or were
structured so as to unfairly advantage a particular respondent.[2]
11.2
During the course of the inquiry a number of matters were raised not
only with respect to the design of tender documents but with the process of
preparing and releasing tender documents and in the evaluation of tender
responses. The committee considers the adequacy of communication and
consultation, the quality of the documentation, including the TEP and the SER
and the checking and verification of calculations in the SER. It looks at key
issues raised before the committee pertained to:
- the decision to re-tender;
-
the technical specifications, in particular the cargo volumetric
capacity and preference for a single aircraft solution;
- the tender response period; and
- the tender evaluation criteria and their application.
Decision to re-tender, specifications and timeframe
11.3
It was alleged by Strategic that the first three matters did not reflect
operational need, were not commercially justifiable in terms of value for
money, and were designed to disadvantage the existing operator in favour of a
foreign, broker‑based solution.
The evidence
11.4
Having regard to the factors cited by Defence, the committee is
satisfied that there was a sound business case for re-tendering the 2008
contract. Defence identified an opportunity to achieve significant cost savings
as a result of the global financial crisis which created excess capacity in the
commercial air charter industry; a decline in demand for international
passenger air travel, shrinking aviation industry profitability, idle aircraft
and falling charter rates.[3]
It also noted that re-tendering was appropriate in light of the numerous
amendments to the 2008 contract which 'had significantly altered' the agreement.[4]
These changes were in areas including fuel allocation, routing, block hours
flown, pricing structure, the aircraft used and consequent load-splitting
arrangements in relation to cargo.[5]
The committee notes the findings of the Deloitte review that the decision to
tender was 'based on a reasonable expectation of achieving a better value for
money outcome for the Commonwealth'.[6]
The AFCD review stated that the decision to re-tender was based on 'valid
commercial and operational considerations'.[7]
11.5
The committee is also satisfied that the design of the RFT was supported
by operational need, and was not intentionally designed to advantage or disadvantage
individual respondents. It accepts Defence's line of reasoning that there was
sufficient evidence of operational need to justify the increased cargo
volumetric requirements in item 7.13 of the request, and the preference for a
single aircraft solution.
11.6
Indeed, the balancing of these considerations was legitimately a matter
of discretion for Defence as purchaser. In the absence of evidence suggesting
that the decision to include these requirements was unreasonable, or was based
on incorrect (or no) factual information, there would be little utility in the
committee making a different assessment of how operational needs should be
reflected in the tender requirements. In this respect, the committee notes the
findings of the AFCD and Deloitte reviews that the tender requirements were
functionally specified in the request, and that consideration was given to
cargo-splitting solutions put forward by respondents. The committee also notes
the evidence of Defence that the 2010 contract has achieved significant
financial savings on the previous contract, and that the additional aircraft
capacity has been utilised in the services performed to date.
11.7
The committee makes clear, however, its ongoing interest in this matter.
It is concerned to ensure that the technical requirements identified in
requests for the provision of air sustainment services—whether under the
current contract or under future contracts—continue to be based on documented
evidence of operational need. Accordingly, the committee requests that Defence
provide it with periodic reports on the ongoing performance of the 2010
contract, including in the realisation of projected savings, the continuing
need for the increased cargo volumetric requirements and the contractor's
compliance with the tender requirements (see recommendation 8).
11.8
The committee accepts the evidence suggesting that the eight-week tender
response period was reasonable. While the limited timeframe inevitably meant
that some respondents would not be in a position to meet the aircraft
certification requirements, the committee acknowledges that this factor must be
weighed against competing considerations. These include, for example,
operational needs and the costs associated with granting a longer response
period to allow potential tenderers to obtain certification or source aircraft
as necessary. Accordingly, the committee considers that the decision to require
an eight-week response timeframe was reasonably open to Defence.
Areas for improvement—documentation, consultation and certification
11.9
Both Strategic and the external reviewers raised issues concerning the
tender evaluation criteria and their application. The external reviews
identified a range of process issues, including a lack of clarity as to the
meaning of individual criteria; the potential for duplication of criteria; and
the standard of documentation recording the evaluation processes and outcomes.
The committee considers that there is scope to improve future practices in the
procurement of services from the Air Transport Standing Offer Panel, including:
- procurement planning, particularly the identification of tender
requirements;
- communication with potential tenderers;
- the evaluation of tender responses; and
- developing the source evaluation report.
Procurement Planning
11.10
The committee considers that procurement planning could be improved in
two respects. First, key planning decisions should be documented at a level of
detail that is commensurate to the level of probity risk. In particular, the
business case for re-testing the market should be documented thoroughly. The
committee notes the findings of the AFCD review in this regard.[8]
In light of proponent grievances about the tender requirements discussed above,
the committee also observes the importance of documenting in detail the
evidentiary basis for these requirements. Rigorous documentation is
particularly important where the revised requirements are greater or more
stringent than those of the current contract.
11.11
Secondly, it is important that future procurement plans make sufficient
contingency for decisions to re-test the market. Such decisions must be made as
early as possible before the expiration of the current contract to allow
sufficient lead time for a re-tender if required. The committee is concerned
that the time pressures bearing upon the 2010 tender process contributed
significantly to the problems encountered. It questions whether the transition
between the 2008 and 2010 contracts could have been better planned. The
committee acknowledges that the tight timeframes in the 2010 tender process
were driven by a contract commencement date that was aligned to operational
requirements in the MEAO. However, it questions whether the decision to re-test
the market and the identification of the tender requirements could have been
made earlier. This would have minimised the time pressures on the 2010 tender
process.
Communication with potential
tenderers
11.12
To manage the risk of proponent grievances, the committee considers that
communication with potential tenderers could be strengthened in several
respects. First, future procurement strategies should continue to include a
requirement that standing offer panel members are provided with advance notice
of any decisions to re‑tender, prior to the release of the request. As
noted by the Deloitte Review, this was not adhered to in the 2010 tender
process. In the committee's view, early notification may assist potential
tenderers in meeting narrow timeframes and help prevent perceptions that such
timeframes may be motivated by uncommercial interests.
11.13
Secondly, the committee notes the importance of providing a clear and
consistent explanation to potential tenderers of how Australian industry
participation is assessed in the tender evaluation process. That is, its
consideration within the overall assessment of value for money. The
correspondence between Strategic and 1JMOVGP in April 2010 indicates that there
may be some industry confusion about this matter.[9]
11.14
Thirdly, the committee considers that aspects of the tender process
could be better communicated to potential tenderers, to ensure clarity of
understanding and minimise the risk of potential proponent grievances. These
include:
(a) providing potential tenderers with an explanation of the reasons for re‑tendering
and the changed tender requirements;
(b) providing potential tenderers with an explanation of how the evaluation
criteria referred to in the request document will be assessed; and
(c) including in the request documentation an express statement of Defence's
preference for a single aircraft solution (or any other preferred solution that
is identified in future requests), and the fact that alternative solutions will
be considered.
Tender evaluation
11.15
The committee notes the shortcomings identified by the external reviews in
the development of the tender evaluation plan, the tender evaluation criteria,
the documentation of evaluation results and other matters of process in relation to the tender evaluation. Such failings increased
the likelihood of mistakes such as the possibility of double counting the same
issue because of the lack of clear guidance in the TEP.[10]
The committee endorses the observations made by the external reviews and, in
particular, considers that Defence should take action on the following matters:
(a) In the development of the TEP, there should be early and ongoing
consultation with relevant line areas within Defence on the evaluation
processes—in particular, advice should be obtained from FIS on the agreed
pricing model and the financial evaluation processes.[11]
(b) The request documentation and the deed of standing offer should identify
expressly:
(i) provisions which are minimum conditions of participation, minimum form
and content requirements or other essential requirements; and
(ii) how Defence will treat completed requests that do not comply with
requirements expressed in mandatory language.[12]
(c) Future TEPs should document the following, to ensure a transparent,
consistent and complete assessment of all tender responses:
(i) a full outline of the tender evaluation methodology, in particular the
scoring methodology and pricing models adopted by the TEB and the TEWGs;[13]
and
(ii) detailed guidance as to how the tender evaluation criteria are to be
applied, to ensure that there is no overlap between criteria, such that issues
are assessed multiple times.[14]
(d) The tender evaluation process should be improved through the following
measures:
(i) ensuring effective consultation between elements of the tender team
during the evaluation process—particularly between the technical and financial
TEWGs;[15]
(ii) requiring the TEB to produce a separate report on its initial compliance
assessment against the evaluation criteria for which it is responsible;[16]
and
(iii) ensuring that the SER transparently and accurately communicates the
evaluation processes, and explains the outcomes in detail. In particular,
future SERs should include the matters identified by the AFCD and AGS Reviews,
and subsequently incorporated into the revised SER.[17]
11.16
As noted earlier the committee had two particular concerns relating to
the evaluation process—the observation about unequal treatment on the leasing
arrangements with the respective aircraft owners and a mathematical error. The
committee notes that the concerns about the unequal treatment were rectified in
the final SER.
11.17
In its submission, Defence noted that the TEB was reconvened to
re-validate the SER. In the process a mathematical error was discovered and
reversed the order the second and third ranked tenderers.[18]
When asked about the consequences stemming from this mistake, Mr Brown agreed
that it was a good point. He stated:
If it had been close, which it was not—and thankfully we found
it before the contract was awarded, as a result of this investigation the gap
was such that it did not matter, but it did swap two to three on a fuel
calculation. It was a computation within the spreadsheet where the error was
found. Again it goes to the point of the probity auditor, 'Have you had an
audit done of the Excel spreadsheet?
11.18
Along similar lines, Dr Watt told the committee that 'the real
difference is that the tenders were chalk and cheese'—'that is the thing that
is hard to get away from in all of this'. The committee notes, however, the
statement in the AFCD review, noted earlier, that there 'was very little
difference between the top three ranked tender options'.[19]
11.19
This mathematical miscalculation and the confusion over the unequal
treatment of the leasing arrangements highlights the critical importance of
having in place robust protocols and important safeguards governing the
evaluation process. The committee is of the view that Defence should ensure
that tender documentation, especially in respect of scoring and pricing models is
clear, unambiguous and detailed, that there is transparency and consistency in
the TEP and that calculations are independently verified.
Recommendation 5
11.20 The committee recommends that Defence:
- In line with the findings of the AFCD Review, considers
strategies for the improved documentation of the business case for any future
decisions to re-test the market for the provision of air sustainment services
to the MEAO.
- Reviews its procurement plan for the current MEAO contract, to
ensure that sufficient lead time is provided for the making of any future
decisions to re-test the market, and the planning and execution of a
procurement process.
- In all future procurements of air sustainment services to the
MEAO:
(a) continues to include in procurement strategies a requirement that
members of the Air Transport Standing Offer Panel are given advance notice of
any decisions to re-tender the contract, prior to the release of the RFT; and
(b) ensures that such requirements are implemented.
- Implements strategies to ensure that potential tenderers have
a clear and accurate understanding of how Australian industry participation is
taken into account in the evaluation of tender responses, as part of the
overall value for money assessment.
- On the release of future requests for air sustainment services
to the MEAO, implements the following actions to minimise the risk of potential
proponent grievances:
(a) provides potential tenderers with an explanation of the reasons
for re‑tendering the contract and any changes to tender requirements from
the previous request;
(b) provides potential tenderers with an explanation of how the
evaluation criteria in the request documentation will be assessed; and
(c) includes in the request documentation, where applicable, an
express statement of Defence's:
(i) preferred solution for meeting tender requirements, including
technical specifications; and
(ii) intention to consider alternative solutions.
- As a matter of priority in future tender processes for the provision
of air sustainment services to the MEAO, takes action on the tender evaluation
issues identified by the Deloitte, AGS and AFCD Reviews, as documented at
paragraph 11.15 of this report.
Conclusion
11.21
The committee found that the decision to re-tender was sound and
supported by operational needs and that Defence could justify the change in
specifications from the previous contract. It is also of the view that although
the time frame for tenderers to prepare and lodge their tender was tight, it was
not intended to deliberately disadvantage any tenderer.
11.22
In respect of the quality of documentation, consultation and
certification, the committee found much room for improvement. Because of
failings in this area, the committee noted the potential for confusion,
inconsistency and errors in calculations during the evaluation, and in some
cases the realisation of this potential. Although the SER was re-validated and
confirmed the successful tender as the top ranked tenderer, the problems
identified in the process cannot help but to undermine the committee's
confidence in the robustness of the decision.
Navigation: Previous Page | Contents | Next Page