6. Measurement and evaluation

6.1
The Committee observed feedback from various inquiry participants related to the perceived success of development assistance initiatives in the Step-up measures. Most commentary was of a qualitative nature, or merely acknowledged the delivery of a project but did not quantify the benefits.
6.2
The Department of Foreign Affairs and Trade (DFAT) submission reflected this in its primary submission:
The success of the Step-up will be measured over the long term; the messages we receive from Pacific partners, together with our performance processes, will help us evaluate our success. These exchanges will be central to assessing our initiatives, and, where necessary, adapting them further, including in the face of COVID-19.1
6.3
The Step-up initiatives are designed to bolster sustainable economic development in the region, provide shared security and stability and a sense of interconnectedness. Successful outcomes from projects would be expected to deliver a benefit in at least one of these areas.

Challenges to quantifying success of Step-up

6.4
The key to building on project successes is being able to identify and quantify metrics of success. DFAT advised of those activities in Step-up delivered to date, and mentioned some as being successful deliveries, but mostly in a qualitative sense, for example, projects were reported as making ‘good’ progress.2
6.5
The Committee noted that a number of projects were in their formative or inaugural years, including, for example, the Australia Pacific Security College which was launched in November 2019.3 This made quantitative analysis less meaningful given the objective will be realised over the longer-term. Other initiatives have experienced obvious delays or pauses due to the pandemic, for example, the first year of studies for the Pacific Australian Secondary Schools Scholarships could not commence as planned in July 2020, as was anticipated in 2019.4 Quantitative evaluation analysis is therefore not expected until projects complete at least an annual cycle.
6.6
In 2019 DFAT commenced a review for a new international development policy. The review received 53 submissions, the feedback of which fed into the development of the Partnerships for Recovery: Australian Official Development Assistance package, to address pressing COVID-19 related concerns and the way forward—as discussed in Chapter 2.5 The review received feedback on potential measurement and evaluation inclusions in the new model of official development partnerships.6
6.7
As previously mentioned the review was paused to deal with the pressing demands of the pandemic and a revised ‘Step-up suite’ was announced.7 As such, the last evaluation cycle of the original format Step-up program was the 2019-20 year, with six months affected by COVID-19 impediments.8
6.8
Clearly, the most pronounced quantifiable measure of success in the Pacific island region, especially given remoteness in the vast Pacific Basin, are low recorded COVID-19 cases and deaths, as at 31 January 2022.9 Adjusting for big increases in cases and deaths suffered by Papua New Guinea and Fiji (and excluding the territories and compacts of association of the United States of America, New Zealand and France), there were 3 618 COVID-19 cases and 12 deaths across the independent Pacific islands.10 This is despite having some of the poorest countries in the world, with under-resourced health systems.11
6.9
This success was highlighted by diplomats at the September 2020 Canberra roundtable, including the observation by Dr Lafoy, the Official Representative of New Caledonia to Australia:
Regarding health security, the Pacific has been acknowledged …on the international stage for its successful management of the health impact of the COVID-19 pandemic. New Caledonia is currently COVID-free, with no deaths and only 23 cases registered previously.12

Timeliness of delivery

6.10
Submitters commented on the delivery timeframe of some projects, noting some particular successes in 2020 given the impediments created by COVID.
6.11
For example, the Australian Government’s PacificAus TV13 initiative, delivered by Free TV Australia, provided, on schedule, the agreed suite of at least 1000 hours per year of free-to-air Australian content programs to broadcasters in seven Pacific island countries14 during 2020.15 The program delivered a satellite dish to Nauru and Tuvalu to enable transmissions; each installed by locals guided remotely by the PacificAus team who were unable to be on site. Free TV emphasised that the project was delivered within agreed project parameters:
Despite the significant impacts of COVID-19, PacificAus TV launched on 25 May 2020, on time and under budget and has been enthusiastically received by partner Pacific broadcasters. Its success is evidenced by the scheduling of the programs supplied and the fact that within two months of launch, the Australian Government was looking to expand the PacificAus TV initiative to Tonga and Samoa.16
6.12
The pandemic has necessarily interrupted the timelines of delivery of other initiatives to preserve the health of island communities and Australian development partners and participants. Any arrangement which involved crossing international borders, with the exception of labour mobility scheme exemptions, were put on hold.17

Program resilience

6.13
The COVID-19 pandemic has been an extreme test of the resilience of the Step-up program and outcomes of development efforts in the recent past.18
6.14
The Department of Foreign Affairs and Trade (DFAT) reported on the overall outcomes delivered in the 2019-20 annual report that:
Through our Pacific Step-up we strengthened the longstanding people-to-people, cultural, economic and security links between Australia and our Pacific family. We established new missions in Palau and the Cook Islands and signed a new partnership with Fiji based on family (vuvale in Fijian) principles of trust, respect and mutual support. We worked quickly to support our Pacific neighbours to prepare and respond to the pandemic. We established a humanitarian corridor to move essential supplies and personnel through the region, and reprioritised development assistance to address urgent health, economic and social needs in both the Pacific and Timor-Leste as part of our broader COVID-19 immediate response package.19
6.15
In DFAT’s 2020-21 Annual Report, expenditure on budgeted development assistance in the Pacific was summarised and appeared to be tracking well, despite the impediments of the cycle:
In 2020–21 the department’s Office of the Pacific managed whole‑of‑government coordination of Australia’s estimated $1.44 billion in development assistance to the Pacific, of which the department delivered an estimated $1.3 billion. In addition, we delivered an estimated $293.8 million in temporary, targeted and supplementary measures to respond to the impacts of COVID‑19 across the Pacific and Timor‑Leste. This included an estimated $200 million delivered under the Pacific COVID‑19 Response Package (approximately $194 million in economic support and $6 million in aviation support) and $93.8 million in additional vaccine support, including technical advice and delivery support.20
6.16
DFAT’s Annual Report commented on the significant impediments to delivery of programs in the year to June 2021:
COVID‑19 brought unprecedented implementation challenges. Health risks, border restrictions and reduced air traffic limited the movement of people and goods across the Pacific region. Our overseas posts and many implementing partners moved to partial remote work. In-country capacity was limited.
We pivoted our existing development programs to respond to COVID‑19 while maintaining critical long‑term investments that helped Pacific island countries to tackle their most significant development challenges. We designed new programs (such as those associated with vaccine delivery) and instituted more regular reporting on risks and safeguards to account for the challenges of remote delivery.21
6.17
Some Step-up initiatives and facilities, perhaps due to their infant status, were agile enough to switch from their primary purpose to quickly delivering crisis-focussed support. For example, the planned Pacific Fusion Centre22 has been set up in Canberra with an interim purpose of improving regional information sharing and advising key decision-makers about the potential regional security impacts of COVID-19.23
6.18
Already noted, the inaugural year of the PacificAus TV program was delivered within agreed terms and timeframes during the pandemic, including the delivery and installation of infrastructure which had been planned to be done with a project team on site:24
Despite the coronavirus pandemic significantly affecting Free TV’s ability to access and deliver the necessary equipment to Partner Pacific broadcasters, finalise licence agreements and to access sports programming, the PacificAus TV service was successfully launched on 25 May 2020.25
6.19
DFAT provided an overall qualitative assessment of Step-up in 2019-20:
Most Step-up initiatives remain on track, while a number have been paused due to travel restrictions and border closures. We are adapting our major initiatives, like the Pacific labour mobility scheme and the Australian Infrastructure Financing Facility for the Pacific (AIFFP), to respond to new circumstances. We have assessed our overall performance on our Step-up objectives as ‘partially on track’.26
6.20
The Department also noted that engagement work undertaken in the months prior to the pandemic assisted virtual communications in 2020:
Between July and March we supported 33 Australian ministerial-level visits to the Pacific and two visit to Australia by Pacific island Prime Ministers. This high level of engagement enabled discussions with Pacific counterparts to continue virtually as COVID-19 unfolded.27
6.21
The Department reported that its performance analysis in 2020-21 was hampered by the pandemic yet reported that it was ‘on track’:
In 2020–21, COVID‑19 affected the ability of our staff and implementing partners to collect and assess monitoring and evaluation data. While we made greater use of remote approaches, limited data availability in some cases impacted performance assessments.28

Stakeholder feedback

6.22
The Committee received feedback on the value of the Step-up program from the people of the Pacific islands and from those engaging with the region. Pacific island diplomats gave general feedback on the Step-up initiatives at a Parliament House roundtable held on 4 September 2020, which addressed the JSCFADT’s Pacific related inquiries which were concurrently undertaken. For example Dr Yves Lafoy, Counsellor and Official Representative of New Caledonia to Australia outlined the benefits of Step-up initiatives:
With regard to the fourth pillar of the Pacific step-up, Australians and New Caledonians have longstanding people-to-people links across business, education, police cooperation, health care and defence, the latter being a power still exercised by the French state. We are grateful to the Australian government for enabling New Caledonia to be eligible for the New Colombo Plan since 2017 and providing scholarships to New Caledonian students. In terms of regional security, the third pillar of the Pacific step-up, New Caledonia, since its admission as a full member of the Pacific Islands Forum in 2016, and in the spirit of the Boe Declaration, contributed to the launching of the Pacific Fusion Centre last year in Canberra. New Caledonia has seconded a number of environmental scientists for a period of three months.29
6.23
The High Commissioner for Solomon Islands, His Excellency Mr Robert Sisilo, welcomed international support for health and COVID-19 measures:
Australia, New Zealand, the People's Republic of China and Indonesia have committed more than 100 million Solomon Island dollars to fund our fight against COVID-19. On top of that, Australia committed A$8 million to support the health sector. It also provided an additional 25,000 items of personal protection equipment and 1,000 RNA extraction kits.30
6.24
His Excellency Mr John Ma'o Kali, High Commissioner for Papua New Guinea said:
Some might want to say that we are joined at the hip by geographic proximity. We have common strategic interests. We have deep people-to-people links. We have mutual respect and cooperation across all spheres. I believe that [Comprehensive Strategic and Economic Partnership (CSEP)], which will be detailed later on, is really crucial to developing and enhancing the connections which have been shared over decades between Australia and Papua New Guinea. CSEP intends to deepen bilateral cooperation across security, trade and investment, governance, development cooperation, health, education, gender equity and equality, climate change, and people-to-people and institutional links.31
6.25
Mr Teata Terubea, Director, Asia Pacific Division, Ministry of Foreign Affairs and Immigration, Kiribati, opined:
Kiribati takes this opportunity also to acknowledge with sincere gratitude and appreciation for the various development assistance and support that the government of Australia has provided to the Pacific Island countries and to commend the advancement of the government of Australia’s Pacific Step-up initiatives to strengthen economic resilience, enhance our security cooperation and deepen personal and community connections across the Pacific region.32

Australia’s development program performance assessment

6.26
The ‘revised’ Step-up development program (which currently includes Pacific Partnerships for recovery) has been established with three key evaluation elements:
A three-tier framework for reporting on the overall context, annual results and effectiveness of Australia’s COVID-19 development response efforts.
Whole of Government COVID-19 Development Response country and regional plans setting out expected outcomes, key results and supporting investments.
Performance indicators for global programs and strategic partnership agreements with multilateral organisations.33

Figure 6.1:  Australia’s Development Program—Performance System

Source: DFAT, Partnerships for Recovery: Australia’s COVID-19 Development Response, May 2020, p. 27. <https://www.dfat.gov.au/sites/default/files/partnerships-for-recovery-australias-covid-19-development-response.pdf> viewed 24 February 2022.

Committee comment

6.27
The Committee notes the difficulty in observing quantifiable successes in the Step-up program to date from inquiry evidence received.
6.28
However, given the start-up of longer-term initiatives in 2019 and 2020, coupled with the unprecedented (and ongoing) pandemic interference which led to the pivoting of some existing initiatives—and in May 2020 a supplementary post-COVID recovery package—this is not unexpected.
6.29
Furthermore, the Government’s review for the new international development policy went into hibernation in 2020 to free resources to focus on COVID-19 priorities. The Committee expects the development of the new policy and its appropriate evaluation criteria to follow the finalisation of the review. However, the Committee acknowledges that performance assessment elements have been released for the ‘Step-up plus’ COVID recovery development framework (the original program with tweaks, plus Partnerships for recovery), which is designed to lead health security, stability and economic recovery out of the pandemic.
6.30
The evaluation elements of the current two year development program should include quantifiable evaluation criteria, the results of which can be compared over budget years.
6.31
In the absence of robust, quantitative and comparable measures of success (or otherwise), the Committee has acknowledged that qualitative comments about the perceived value of Australia’s current engagement in the Pacific island region is largely positive.
6.32
The Committee recognises that programs should be developed not only taking account of the voices of the various communities involved but involving them directly (as in disaster preparedness and climate adaptability measures). As such, it will be necessary to also maintain qualitative evaluation criteria to assess not only the data, and whether deliverables have been met, but how the Pacific island people—and by extension, the diaspora—feel about the strength of the relationship through the development partnership activity. That is, the relationship should be strengthened through strong, respectful and sustainable engagement and not merely project deliverables.
Senator the Hon David Fawcett
Chair
Joint Standing Committee on Foreign Affairs, Defence and Trade
15 March 2022
Mr Dave Sharma MP
Chair
Foreign Affairs and Aid Sub-Committee
15 March 2022

  • 1
    Department of Foreign Affairs and Trade (DFAT), Submission 52, p. 14.
  • 2
    DFAT, Submission 52, pp. 7-10.
  • 3
    DFAT, Submission 52, p. 9.
  • 4
    DFAT, Submission 52, pp. 9-10.
  • 5
    Internet Archive, Archive of ‘DFAT, Submissions on a new international development policy’ webpage, 18 May 2021. <https://web.archive.org/web/20210518142947/https://www.dfat.gov.au/aid/new-international-development-policy/Pages/submissions> viewed 24 February 2022.
  • 6
    For example, Australian Council for International Development’s submission to the review: <https://www.dfat.gov.au/sites/default/files/australian-council-for-international-development-performance-framework.pdf> viewed 24 February 2022.
  • 7
    The revised Step-up comprised supplementary measures (Partnerships for recovery) and redirected assistance measures (including changes to the original intent of Step-up initiatives); DFAT, Australian Development Budget Summary 2020-21, May 2020, p. 1. <https://www.dfat.gov.au/sites/default/files/pbs-2020-21-dfat-aid-budget-summary.pdf> viewed 15 February 2022.
  • 8
    DFAT, Department of Foreign Affairs and Trade Annual Report 2019-2020, September 2020, pp. 28-31.
  • 9
    The total cases recorded at 31 January 2022 is 103 618 and 1 410 deaths (excludes the data for the US territories and those in its compact of free association, those under the New Zealand realm and and the French territories). Papua New Guinea (PNG), which had a low reported infection rate to official population until February 2021, accounts for 37 145 of these cases and 597 deaths. Fiji (where cases escalated after opening the international border in December 2021) accounts for 62 855 cases and 801 deaths. The Pacific Community, Covid-19: Pacific Community Updates, 31 January 2022. <https://www.spc.int/updates/blog/2022/02/covid-19-pacific-community-updates> viewed 2 February 2022.
  • 10
    The Pacific Community, Covid-19: Pacific Community Updates, 31 January 2022. <https://www.spc.int/updates/blog/2022/02/covid-19-pacific-community-updates> viewed 2 February 2022.
  • 11
    Development Policy Centre, ANU, Submission 60, p. 2, ‘three Pacific countries are among the 30 poorest countries in the world: Kiribati (19th), Solomon Islands (21st), and Vanuatu (28th).’
  • 12
    Dr Yves Lafoy, Counsellor and Official Representative of New Caledonia to Australia, Committee Hansard, 4 September 2020, p. 7. In context, Dr Lafoy also noted that New Caledonia holds 25 per cent of the world's nickel resources and benefits from financial transfers from France. It has levels of development and infrastructure of a developed country. (At 31 January 2022 New Caledonia had recorded 20 622 COVID-19 cases and 283 deaths; Pacific Data Hub, COVID-19 cases in Pacific Island Countries and Territories, PDH.stat, <https://stats.pacificdata.org> viewed 2 February 2022)
  • 13
    For more details on the rollout see: PacificAus TV, What is PacificAus TV?. <www.pacificaustv.com/whatispacificaustv> viewed 24 February 2022.
  • 14
    Papua New Guinea, Fiji, Vanuatu, Solomon Islands, Kiribati, Tuvalu and Nauru; Free TV Australia, Submission 65, p. 5.
  • 15
    2020 was the first year of the three year contract; Free TV Australia, Submission 65, p. 3.
  • 16
    Free TV Australia, Submission 65, p. 3.
  • 17
    This applied to the Pacific Secondary Schools Scholarships Program, with the inaugural intake due in July 2020 delayed; DFAT, Submission 52, pp. 9-10.
  • 18
    Further complicating outcomes of Step-up initiatives in 2020 were the impacts of two significant tropical cyclones in the region.
  • 19
    DFAT, Department of Foreign Affairs and Trade Annual Report 2019-2020, September 2020, p. 14.
  • 20
    DFAT, Department of Foreign Affairs and Trade Annual Report 2020-2021, September 2021, p. 61.
  • 21
    DFAT, Department of Foreign Affairs and Trade Annual Report 2020-2021, September 2021, p. 61.
  • 22
    Established to build technical analytical capacity to help inform the region’s responses to major security challenges; DFAT, Submission 52, p. 6.
  • 23
    DFAT, Submission 52, p. 9.
  • 24
    Free TV Australia, Submission 65, p. 7.
  • 25
    Free TV Australia, Submission 65, p. 7.
  • 26
    DFAT, Department of Foreign Affairs and Trade Annual Report 2019-2020, September 2020, p. 29.
  • 27
    DFAT, Department of Foreign Affairs and Trade Annual Report 2019-2020, September 2020, p. 29.
  • 28
    DFAT, Department of Foreign Affairs and Trade Annual Report 2020-2021, September 2021, p. 69.
  • 29
    Dr Yves Lafoy, Counsellor and Official Representative of New Caledonia to Australia, Committee Hansard, 4 September 2020, p. 7.
  • 30
    His Excellency Mr Robert Sisilo, High Commissioner, Solomon Islands, Committee Hansard, 4 September 2020, pp. 7-8.
  • 31
    His Excellency Mr John Ma'o Kali, High Commissioner, Papua New Guinea, Committee Hansard, 4 September 2020, p. 9. The Papua New Guinea Australia Comprehensive Strategic and Economic Partnership (CSEP) was entered into in August 2020. For details see Prime Minister of Australia, Papua New Guinea-Australia Comprehensive Strategic and Economic Partnership, Media Statement, 5 August 2020. <https://www.pm.gov.au/media/papua-new-guinea-australia-comprehensive-strategic-and-economic-partnership> viewed 27 February 2020.
  • 32
    Mr Teata Terubea, Director, Asia Pacific Division, Ministry of Foreign Affairs and Immigration, Kiribati, Committee Hansard, 4 September 2020, p. 9.
  • 33
    DFAT, Partnerships for Recovery: Australia’s COVID-19 Development Response, May 2020, p. 26.

 |  Contents  |