Part IV
Transparency and accountability
This final part of the report is concerned with the way in
which Australia assesses and reports on the effectiveness of its aid to
Afghanistan. To this stage of the report, the committee has identified a number
of Australian-funded programs, their achievements and in some cases their
shortcomings. The committee has also considered the mechanisms through which
Australia channels its funds—ARTF, multilaterals, NGOs and the PRT in Uruzgan.
While acknowledging the impressive gains that have been made in Afghanistan with
Australian support, the committee has not yet determined whether Australian
funds are being used to best effect—whether they provide value for money.
In this part of the report the committee looks at the ways
in which Australia holds those delivering aid to account for their performance
and how Australia evaluates its own performance and importantly reports its
findings.
Chapter 17
Evaluating the effectiveness of Australian aid
17.1
Australia has provided over $700 million in aid to Afghanistan since
2001 with tangible results.[1]
The question before the committee is whether this aid was used most
effectively. There are suggestions that some projects have
not measured up to expectations—AliceGhan and the Australian Leadership Awards Scholarships
for Afghanistan students. There are other projects where the indications are
that, while impressive on paper, the achievements on the ground may not be as
substantial as initial indicators suggested. In this regard, some witnesses
referred to schools being built but without substantial evidence to show
increased school attendance. In this chapter, the committee looks at the
transparency of Australia's aid funding to Afghanistan: at how Australia
monitors, analyses and evaluates the effectiveness of its own performance.
17.2
The committee looks first at the monitoring and assessment of programs
delivered through Afghanistan's national budget via the ARTF, multilateral
organisations and NGOs.
ARTF
17.3
The committee has looked in detail at the transparency and
accountability mechanisms employed to ensure that the ARTF uses its funds
effectively and efficiently. Based on solid evidence, it found that the fund is
open and transparent and subject to a high level of scrutiny. Even so, there
were aspects of its accountability that could be strengthened. Importantly, the
recent independent review of the ARTF, part-funded by Australia, highlighted
the importance of 'intensive and detailed reporting'. It found that there is a
need to achieve a 'consistent, comprehensive and critical tracking and
reporting system'.[2]
With regard to the fund's National Solidarity Program and the Community
Development Committee model, it was also looking for a critical assessment of
achievements against political-social, mobilisation and livelihood objectives.[3]
17.4
The committee believes that the findings of this independent review in
relation to robust tracking and reporting systems and undertaking critical
evaluations of achievements as they affect, for example, livelihoods, should hold
true for Australian aid programs in Afghanistan.
Mutual accountability
17.5
A number of witnesses also recognised the importance of the Afghan
Government improving its performance as part of a mutual accountability
framework. They underlined the importance of attaching measurable conditions to
assistance in order to engender positive incentives for the transition decade
and to foster accountability. In managing the transition process, CARE was of
the view that shifts towards on-budget aid through the government should be
sequenced on the basis of demonstrated progress against sector-specific
benchmarks in state capacity and accountability at central and sub-national
levels.[4] Professor Maley noted that, if one is
working at the central level, one needs some pretty tough conditions attached
to funding.[5]
Budget transparency
17.6
Budget accountability is one area where Afghanistan has made some
headway. Since 2008, the country has made 'steady and impressive progress'
toward greater budget transparency, particularly in the last two years. On the
Open Budget Index in 2010, Afghanistan scored 21 out of 100 and subsequently
jumped 38 points to register 59 in the 2012 survey. Although with considerable
room to improve on budget transparency, Afghanistan has demonstrated a
willingness to provide greater information. For the first time, it now
publishes a Pre-budget Statement, the Executive's Budget Proposal and a Citizens
Budget.[6]
The 2012 open budget survey attributed Afghanistan's improved transparency to:
...the political will of the leadership of the Ministry of
Finance, as well as the government's desire to improve its international
image...Donor organisations and international financial institutions also
increasingly focused their attention on fiscal transparency as a means to
reduce corruption in the country. Their pressure, coupled with technical
assistance provided to the Ministry of Finance, facilitated quick improvements.
As part and parcel of these developments, civil society organizations and
researchers have started engaging with the government, primarily through the
Ministry of Finance, on budget-related issues, publishing budget analyses and
organizing public awareness campaigns through the media, and conducting
meetings and workshops to highlight the importance of budget transparency for
citizen monitoring and government accountability.[7]
17.7
The Survey noted that in 2010 donors committed to channelling up to 50
per cent of their aid through the budget 'on the condition that the government
make its budget more transparent and increase its spending capacity'.[8]
Clearly, with encouragement and technical support from the international donor
community, the Afghan Government can undertake reforms necessary to improve
transparency and accountability. Having conditions that are tangible, specific
and measurable are important, as is the concerted effort and good will of the
Afghan Government, the international community and civil society to help the
government meet those conditions.
17.8
This one example of improved governance is, however, not matched in
other areas. The committee has noted that the perception of corruption in
Afghanistan has changed little since 2001. Indeed, Afghanistan still has a long
way to go to improve governance across a number of facets. The worldwide
governance indicators shows that Afghanistan remains in the bottom 0–10th
percentile on voice and accountability; political stability and absence of
violence; government effectiveness; regulatory quality; rule of law and control
of corruption.[9]
Moreover, in some cases, Afghanistan has reversed its performance—a few years
ago, in voice and accountability and government effectiveness Afghanistan
managed to score in the 10–25th percentile but has since fallen back.
Importance of monitoring and
evaluation
17.9
Although multilateral organisations (including the World Bank through
its ARTF) and international NGOs deliver the bulk of Australia's ODA to
Afghanistan, the Australian Government is ultimately accountable for how its
money is spent and its effectiveness. AusAID provided the committee with a
detailed account of the measures it takes to ensure that its partners in
delivering development assistance do so efficiently and effectively.
17.10
Australia has signed a memorandum of understanding with the Government
of Afghanistan on development cooperation, which includes some conditions.
According to AusAID, the agreement contains 'a number of quite robust
commitments including fiduciary and administrative capacity within the
country's institutions in Afghanistan at national and local levels.' It also includes
'a number of commitments on tackling corruption and improving public
expenditure systems'. She explained that it is through this framework that Australia's
increased aid to Afghanistan over the next few years would take place.[10]
17.11
Dr Bizhan noted, however, that some of these measures in the memorandum
of understanding were broad and could be interpreted differently by various stakeholders
in Australian and Afghan. He referred to commitments by the Afghan Government
to fight corruption and build effective administration. For example, the Afghan
Government has undertaken to 'make tangible progress toward a democratic
society, where the equality of men and women, and the active participation of
both in Afghan society are respected'. Other equally broad commitments include
reducing corruption; addressing injustices and increasing people's access to
justice; managing revenue; and building capacity for accountable and fair
service delivery.[11]
17.12
Clearly, it would be helpful if the benchmarks against which
improvements could be gauged were concrete, practical and, indeed, measurable such
as the extent to which Afghanistan had developed its own institutions, their
strength and resilience and capacity to deliver services.
Multilaterals and NGOs own systems
17.13
AusAID stated that it promotes effectiveness and accountability for Australian
aid to Afghanistan in a number of ways. It works through credible development
partners, such as the World Bank, that have demonstrated in-country experience
and effectiveness and have robust monitoring of fiduciary risk management
systems in place.[12]
AusAID informed the committee that it employs a range of other management and
evaluation approaches, including direct monitoring, monitoring through trusted
partners, communities and third parties.[13]
Monitoring
17.14
According to AusAID, it ensures that all its contractual agreements with
implementing partners contain explicit provisions against fraud and corruption
and it engages closely with implementing partners to ensure that they have
robust scrutiny systems in place. It holds its partners to account through
strict reporting requirements against their agreed deliverables and monitors
and reviews programs directly wherever it is safe to do so.[14]
According to AusAID, it responds quickly if it detects any financial
irregularities and encourages its partners to do the same in those
circumstances.[15]
17.15
Looking more specifically at Uruzgan, Mr Dawson indicated that since AusAID's
early engagement in the province, it has been looking to build its capacity for
assessment. He explained:
Initially the work with the liaison office was one way of
providing access to information about circumstances in the province; but over a
period of time our own capacity to provide that same information and level of
analysis of developments in the province has increased.[16]
17.16
Mr Dawson mentioned the PRT and the embassy and DFAT people on the
ground who talk to the Afghan Government and other stakeholders—including community
groups—to provide direct contact with Australian agencies operating in
Afghanistan.[17]
According to Mr Dawson, AusAID's capacity for monitoring and evaluation had
increased 'very substantially' since the agency engaged a liaison office in
November 2010. These include support over three years for the annual Afghan
people's survey. The survey canvasses the views of Afghan citizens from all
provinces on a wide variety of issues, including economic development,
political participation, corruption and the status of women.[18]
It does not, however, specifically canvass views on Australian-funded projects.
17.17
In partnership with another service provider, AusAID is also to deliver
a specific Uruzgan monitoring and evaluation program—a data collection system,
which will monitor and analyse AusAID's programs in Uruzgan. The program commenced
in October 2012 and will run for one year, initially. It will include an online
database to house collated information on baselines and the results of
Australia's activities. According to Mr Dawson, it will go a substantial way to
addressing information gaps and limitations that AusAID and other development
partners face in Uruzgan. Significant access to information that AusAID obtains
through its own sources and direct contacts in Uruzgan will supplement the
program.[19]
17.18
Mr Dawson also referred to AusAID forming relationships with other
organisations operating in Afghanistan that can provide information 'on the development
circumstances and the progress of transition in different parts of the country'.[20]
He explained that given this combination of other organisations and AusAID's
own increased resources, the agency was confident of its capacity to understand
and assess the situation in the province.[21]
17.19
The committee has considered AusAID's detailed description of the
measures it takes to ensure the effective delivery of aid in Afghanistan. In
this context, the committee notes that attention is heavily focused on intentions
and process—steps taken to improve and strengthen administrative procedures—but
not on the actual outcomes, particularly what reaches intended beneficiaries
and what it means for them. As Dr Bizhan pointed out:
The basic issue should be to measure the effectiveness of aid
as it influences targeted recipients or objectives rather than 'so called
efficiency in aid administration'.
17.20
Dr Bizhan observed that sometimes 'an efficiently administered project
could have a negative impact'.[22]
Professor Maley also noted that compliance with process was relatively easy to
assess. He noted, however, that the exact outcomes of policy initiatives may be
'far from obvious', which may be 'tempting to replace appraisal based outcomes
with appraisal based on process'. He concluded that ultimately, development
policies need to be judged by outcomes, not by processes.[23]
For example, Professor Maley noted that:
With trained teachers and basic teaching materials one can
run a basic school without a dedicated building, but a school building without
teachers is simply an aggregation of bricks, mortar and concrete.[24]
17.21
The committee appreciates that having correct processes and procedures
in place is important, but that is only part of the picture. The committee now
turns to consider the effectiveness of Australia's aid program: not on process
and inputs but on what has been delivered on the ground.
Effectiveness of aid as delivered
and used
17.22
For many years, the committee has commented on Australian-funded aid projects
that have failed in a number of aspects but importantly in their sustainability.
The committee is referring to projects completed successfully but then underutilised,
neglected, or abandoned completely because of shortfalls in resources to cover
operational and maintenance costs or the community's reluctance to use them.
17.23
The problem for the committee is that such deficiencies tend to surface
after the completion of a project—when there are insufficient means to operate
and maintain a facility or it is not suitable for the environment or intended
community. For example, when children are not attending a new school or
patients not using a health clinic; when farmers do not have access to, or are
unable to take advantage of, improvements in crop production or animal
husbandry. There is also the risk of aid having unintended consequences such as
laying the foundations for future conflict by favouring particular individuals.
As mentioned already, there is the possibility that Australian funds in Uruzgan
may have helped to empower individuals looking to promote their own interests
rather than those of the local communities. In particular, the committee
mentioned Matilluah Khan. There appears, however, to have been no real analysis
or serious consideration given to understanding or appreciating the longer term
consequences of such funding.
17.24
The committee has only a few specific examples of Australian-funded
projects in Afghanistan that had fallen short or not delivered to their
potential. Some of the evidence before the committee, however, hints at other
instances of the ineffective use of aid funds. For example, Professor Maley
argued that the record for achieving local capacity building in Afghanistan was
'notably patchy'. He cited the successful work that went into preparation for
the 2004 election. At that time, the Joint Electoral Management Body, in which
Australian experts were actively involved, adopted a very positive approach to
local capacity building. The intention was to have a strong cohort of trained
Afghan staff able to do the bulk of the technical work required for a proper
process. According to Professor Maley, the United Nations failed to make
effective use of these skilled personnel with little effort made to retain
their services for the 2005 parliamentary elections. In his assessment:
The net result was that the 2005 election went over budget,
and the 2009 presidential election was marred by very serious fraud. The lesson
here is that the failure to engage in effective local capacity building can
have potentially grave long-term consequences.[25]
17.25
The current concern is that, with the anticipated decline in government
revenue and the closure of the PRTs, some projects in Afghanistan will be
unsustainable. The committee has highlighted the importance of Australian aid
focusing on sectors with a proven track record. To do so, decision makers must
rely on a sound understanding of what is working in Afghanistan and,
importantly, likely to prove durable in a vulnerable economic and security
environment. Thus, evaluation of projects, with their accompanying lessons to
be learnt, is central to this process.
17.26
From the committee's perspective, however, there is a dearth of hard-nosed
and searching examination of projects funded by Australia, even for those that
have drawn criticism. There is no publicly available assessment of the
AliceGhan project or of the circumstances that led to the delayed visas for
Afghans invited to a workshop at the ANU. It would appear that an independent
review of the Australian Leadership Awards Scholarships for Afghanistan was
precipitated by allegations of fraud.
Difficulty evaluating projects in
Afghanistan
17.27
The committee accepts that there are major obstacles to evaluating the
effectiveness of development assistance to Afghanistan. In particular, the inability
to move about the country means that aid officials do not have the
opportunities to talk to local people in order to receive feedback, to gather
data, inspect, monitor and evaluate a project on the ground. Professor Maley
referred to the limitations personnel in Afghanistan encounter in monitoring
and evaluating aid projects:
Most foreign embassies are swathed in security constraints
which make it exceedingly difficult for them to perform...some of the basic tasks
of diplomatic reporting. [26]
17.28
He made clear that this statement was not a reflection on Australian
staff deployed to Afghanistan, some of whom, in his opinion; were outstandingly
competent people.[27]
Professor Stephen Howes and Mr Jonathan Pryke also accepted the view that evaluating
aid effectiveness in Afghanistan was 'not an easy exercise':
Information is scarce, and feedback difficult to obtain. While
measures can be taken to mitigate them, these problems are very much in the nature
of aid given the fundamental geographical disconnect which underlies all aid:
the fact that aid funds are raised in one country and disbursed in another.
Evaluating aid to Afghanistan raises special challenges. It
is very difficult to visit Afghanistan. There is a huge shortage of impartial
information.[28]
17.29
While they acknowledged that monitoring and evaluation was 'harder in
Afghanistan than just about anywhere else in receipt of Australian aid', they argued
that this situation was all the more reason for the Australian Government itself
to monitor and evaluate its aid programs.[29]
17.30
As noted earlier, Australian civilians in Uruzgan are likely to pull
back to Kabul. In this regard, Professor Howes argued that Australian personnel
would not be able to do as much fieldwork because of the security situation,
but again that drawback was no reason for failing to produce annual performance
reports or for Afghanistan not being part of the transparency reform underway
in the aid program.[30]
Independent evaluation
17.31
The high level of corruption in Afghanistan underscores the importance
of transparency in Australia's aid program and robust evaluation and reporting.
Professor Maley emphasised that Australia has a strong legislative framework to
address the problem of bribery of foreign officials, but it was difficult to
put into effect when dealing with the Afghan environment. In his view, careful
monitoring of on-the-ground activity was one way to begin to deal with this
problem. He then stated, however, that it was 'precisely this kind of
monitoring which seems unlikely to be sustainable in the long run with the
mooted withdrawal of Australian personnel from Uruzgan to Kabul'.[31]
17.32
Professor Maley noted the importance of putting in place mechanisms that
would ensure the effective, independent appraisal of how Australia's policy
initiatives have affected the situation on the ground in Afghanistan. He
underlined the importance of independence in the appraisal process—that this
aspect matters most.[32]
According to Professor Maley:
...understandably, aid agencies may have a subliminal tendency
to value what they have done, just as parents can often see in their children a
beauty which is invisible to all other observers.[33]
17.33
In his view, for this reason the widely-reported termination of the
relationship between AusAID and the Liaison Office (TLO) was unfortunate.[34]
17.34
CARE Australia also highlighted the need to establish a set of
benchmarks and targets for the phased transition.[35]
For example, Mr Poulter underlined the critical importance of 'monitoring,
against benchmarks—what actually gets down to the most affected in society and
gets out of Kabul'.[36]
Committee view
17.35
While restrictions hamper evaluation, they are not an excuse for failing
to do so. Indeed, they underscore the need for sound and thorough analysis and
assessment of projects so that Australia can improve on its delivery of aid. If
a donor country, such as Australia, is committed to the effective delivery of
aid then it would also welcome open and independent scrutiny of the projects it
funds.
17.36
Mr John Eyers, however, conducted a survey of published evaluation and
reviews of Australian aid to fragile and conflict-affected states (FCA) and
found:
...readers must gather together for themselves the observations
they contain about where the performance of aid programs has been impaired by
countries’ fragile situations; and while there are references to innovations
intended to address difficulties particular to FCA countries, most of these are
not followed by reporting in later years on how successful or otherwise they
had proved to be.
Similarly, readers must make their own inferences about how
the effectiveness of Australia’s aid in FCA countries compares with that of
programs in other countries.
As far as he could discern, none of the published evaluations
or reviews had addressed this question directly.[37]
17.37
Professor Howes and Mr Pryke argued that Australian aid to Afghanistan
had hardly been evaluated at all. In particular, they noted that it was
remarkable that AusAID had 'not thought it necessary to provide a report by
management on its aid to Afghanistan even though there are so many questions
around whether it represents value-for-money'.[38]
According to Professor Howes and Mr Pryke, the Australian aid program in general
has become more transparent and monitoring and evaluation had improved over
time. With regard to Afghanistan, however, they argued that practice had lagged
even when it came to the internal management reports.[39]
They cited a number of indicators that applied as at mid-September 2012 in support
of this finding:
- AusAID had released a number of evaluations from recent years—not
one related to Afghanistan;
- most countries that receive significant volumes of Australian aid
now had 'transparency pages' on AusAID's website where key strategies and
documents were provided—but not for Afghanistan;[40]
- since 2006 AusAID had released an Annual Performance Report, in that
year and/or one more recently for nearly every bilateral aid recipient—Afghanistan
is one of the few exceptions, and the only one for a major aid program
(certainly the only one in the top ten);[41]
and
- the Office of Development Effectiveness was established in 2006
and has conducted several country and sectoral evaluations—but never a country evaluation
of Afghanistan or a sectoral evaluation which draws on Afghan experience.[42]
17.38
With regard to annual performance reports, Professor Howes noted that one
annual performance report, for 2010, had been published for Afghanistan. He
argued, however, that given Afghanistan:
...is the fourth largest program, you would expect a report
every year, and that is something that the country program is responsible for.
So, even if they are sitting in the embassy, they can still write it.[43]
17.39
Professor Howes acknowledged that to its credit, AusAID had made some
effort to evaluate projects in Afghanistan—it financed a Feinstein study and,
until recently, the Liaison Office.[44]
He understood that because of the security situation, AusAID officers would not
be able to do as much fieldwork but, as noted earlier, that was not a reason for
failing to produce annual performance reports or for Afghanistan not being part
of the transparency reform underway in the aid program. Overall, he concluded
that the same arrangements that apply to other aid programs should apply to
Afghanistan: that Afghanistan should be quickly pulled into line with the rest
of the aid program with respect to transparency, program monitoring and
evaluation.[45]
17.40
It should be noted that since Professor Howes and Mr Pryke lodged their
submission, AusAid has produced a 2011 Annual Program Performance Report for
Afghanistan and, on 21 December 2012, launched the Afghanistan Transparency
Page.[46]
The question then arises whether these measures, together with other reporting
mechanisms, provide the level and quality of information indicative of robust
analysis and assessment of Australia's aid programs in Afghanistan.
17.41
The Director General, AusAID, Mr Baxter, informed the committee that one
can get a sense of the effectiveness of Australia's bilateral aid in a country
context in the agency's annual report to Parliament. He explained that AusAID
reports extensively on the progress of each of its country programs against
individual MDGs. He argued that the agency provides a higher level of detail in
its annual report to parliament than ever before and further that it is the only
organisation in the Commonwealth required to report to Cabinet on the totality
of its program on an annual basis. In his view, AusAID receives a level of
scrutiny that does not apply to any other agency in the Commonwealth.[47]
17.42
While AusAID's Annual Report provides information on the amount spent in
Afghanistan, and describes some of the programs funded by Australia, it does
not give any indication of the effectiveness of programs. Some of the
achievements listed cannot be attributed directly to Australian funds but more
generally to the international donor community, for example achieving a longer
life expectancy for Afghan women. The committee agrees that although the report
may give a sense of the effectiveness of Australia's bilateral aid to
Afghanistan, it in no way provides analysis or evaluation.
17.43
Similarly, the committee argues that while the Afghanistan Annual
Program Performance Report provides a wealth of information on AusAID's
development assistance to Afghanistan, it is mainly descriptive, provides
little hard data and makes it difficult to determine how Australian aid to
Afghanistan is performing. The document is strong on describing programs and
activities; on detailing inputs such as the amount of money disbursed to
various organisations and on intentions but extremely weak on analysis and
evaluation. Often the report lists a range of achievements without any direct
connection to a specific Australian program. There are numerous examples of broad
statements bordering on meaningless. For example AusAID's staff in Kabul will
work closely with the Afghan Government and continue to play an active and
influential role in donor coordination (pp. 21–22). Such statements simply
invite more questions—what is meant by influential, how has it changed
behaviour? Moreover, the statement appears to be one of intention, which is
worrying since one would assume that such activity is the very bread and butter
of Australian diplomacy.
17.44
The Annual Program Performance does contain a table meant to show
progress against objectives. Firstly, the four stated objectives are broad:
- enhancing basic service delivery in health and education;
- supporting rural development and livelihoods;
- improving governance and the effectiveness of the Afghan
Government; and
- supporting vulnerable populations.
17.45
Secondly, the ratings are crude indicators of performance with all four above
objectives obtaining the rating of 'will be partly achieved within the
timeframe of the strategy'.[48]
The committee has no idea what to make of the ratings.
17.46
Where the performance program does give an indication of results on the
ground such as 1,578 farmers trained in improved cropping techniques and 5,016
in improved livestock management, there is no indication whether Australian
funds contributed fully or only partially through the Aga Khan Foundation to
this result. More importantly, there is no assessment as to the extent that the
projects have in fact changed practices for the better and whether Australia's
contribution was a cost effective way to help the farmers.[49]
17.47
Throughout this report, the committee has referred to observations about
the construction of schools but with attendance not matching enrolments or without
reference to retention rates. Local residents have referred to 'white
elephants'. Such observations may be unfair, but the committee believes that they
should be tested, otherwise the achievements trumpeted may well mask little or
no real gains for the Afghan people.[50]
17.48
There can be no denying that improvements due to Australian aid have
been made. But considering the hundreds of millions of dollars spent in
Afghanistan to help the people rebuild their country and their lives, it would
be unusual not to have some obvious improvement. The committee does not want to
appear to be too critical, but without a robust evaluation of Australia's aid
projects in Afghanistan, there can be no genuine understanding of whether the
various programs represent value for money and are likely to make a lasting
difference for the better for the Afghan people.
ADF projects
17.49
The reconstruction work undertaken by the ADF in Uruzgan came under
harsh criticism for its failure to evaluate the effectiveness of its
development assistance.
17.50
Save the Children, Oxfam and World Vision noted the importance of
evaluating projects. It noted:
Development and reconstruction projects implemented by the
ADF in Afghanistan have not been independently evaluated for
cost-effectiveness, impact or sustainability. Nor has the ADF, in its financial
reporting, disaggregated its aid operations in Afghanistan from its military
operations.[51]
17.51
An Australian Council for International Development publication also
found that the ADF does not appear to disaggregate its aid operations from
military operations in Afghanistan and further that ADF-supported development
projects have not been evaluated for cost-effectiveness, impact or outcome'.[52]
AID/WATCH argued that the extent to which problems (cost-effectiveness, focus
on strategic goals, quick fix projects and poor accountability structures) apply
to Australian assistance was unclear due to 'a lack of transparency in aid
delivered by the military'.[53]
17.52
Defence in its submission informed the committee that circumstances in
Afghanistan militated against the conduct of formal cost/benefit evaluations.
These included: the overall security situation; the relatively small scale of
the individual projects undertaken by the military Reconstruction and Task
Force and ADF Managed Works Team; and the time imperatives to consistently
deliver immediate and visible benefits to local communities.[54]
Defence made clear that the extent of its monitoring and evaluation finished
when the construction was complete, the defect liability period had expired and
the project handed over to the relevant government authority. Defence does not
go back to completed projects to do an evaluation as to effectiveness.[55]
17.53
The committee understands the ADF's position that it is not an aid
agency. Nonetheless, millions of dollars have been expended on substantial
reconstruction work in Uruzgan as part of a whole-of-government effort. The
committee cannot accept the lack of any subsequent assessments of the
effectiveness of this type of development assistance. The committee has
evidence that the quality of work produced under ADF supervision is high but understands
that while a project can be 'beautifully constructed' it may not be operational.[56]
17.54
Throughout this report, the committee has quoted from departmental
officials or official documents referring to Australia's 'integrated
whole-of-government effort involving interlinked, diplomatic and development and
military objectives' in Afghanistan.[57]
It would seem that AusAID, in the absence of any Defence evaluation, could have
been involved in monitoring and reporting on these facilities. AusAID made
clear that it does not 'assess, evaluate or monitor ADF projects for
effectiveness or how they fit with the MDGs'.[58]
This lack of coordination, of long term vision calls into question the working
of this so-called integrated whole-of-government effort.
17.55
There does not appear to have been any serious analysis of Australia's
whole-of-government approach in Afghanistan. DFAT made clear that it 'does not
generate separate reports/recommendations on the whole-of-government
performance in Afghanistan.[59]
Whole-of-government
17.56
John Eyers, who undertook a survey of evaluations of Australian aid to fragile
and conflict-afflicted states, noted that the effectiveness of Australia's
whole-of-government overseas aid was a surprising gap in recent such
evaluations.[60]
He suggested that the independent reviews of Australia's aid to fragile and
conflict-affected countries, including Afghanistan, would provide more
evaluation of the parts played by agencies other than AusAID and with more
prompt publication.[61]
The committee agrees with this observation.
Committee view
17.57
Based mainly on Annual Reports, the committee has provided a detailed
description of the development activities undertaken by the various agencies in
Afghanistan. Generally, the accounts are simply descriptions providing no indication
about the extent to which they reflect the effectiveness of the aid. Where this
activity has been part of a multilateral contribution it is difficult to
discern the effect of Australia's contribution. The committee has seen no
evidence suggesting that Australian government agencies delivering aid to
Afghanistan have attempted any genuine critical evaluation of the effectiveness
of Australian aid, including an assessment of the cost-effectiveness of aid
programs. Information is available on the inputs and when recording outcomes
the information is often restricted to quantitative information such as
schools, clinics and roads built but with no indication about how these
facilities are making a difference: that is the quality of the change that is
being achieved. Such reporting presents an incomplete picture and may mask
serious underachievement.
17.58
One of the main difficulties, however, is to comprehend the extent to
which the projects have had a lasting positive effect, particularly in light of
running costs—the need for trained people, to pay salaries, and to fund the
operation and maintenance of the project. This consideration is particularly
relevant in a fragile and conflict-affected country already dependant on aid to
provide essential services and with an uncertain future. Clearly, there is a
need for periodic systematic follow-up to determine whether the project was and
remains viable after completion and when aid funding for it has ceased.
17.59
In this regard, a number of witnesses also looked at the stated
objectives of Australia's aid program in Afghanistan. To make the reporting
more robust, the committee notes that the development goals need to be clear
and specific. Thus, while measuring development against the MDGs may provide a
general sense of the effectiveness of aid, it is too broad to give any certain
indication that particular Australian programs were value for money.
Reporting
17.60
The Australian Government agreed with the recommendation of the Independent
Review of Aid Effectiveness that all Australian government departments and
agencies adopt a three-tiered reporting system in relation to their use of ODA
funds. The three tiers for reporting would be:
- progress against development goals;
- the contribution of Australian aid; and
- operational and organisational effectiveness.[62]
17.61
The committee has drawn attention to the lack of rigour in AusAID's
reporting on ODA to Afghanistan. In light of the committee's findings on the inadequacy
of analysis and reporting on the effectiveness of Australian aid, the committee
believes that the three tiers of reporting required from other government departments
and agencies also needs to meet higher standards.
17.62
For a start, the committee believes that either operational
effectiveness should be better defined or a fourth tier should be added. This
forth tier would require an assessment and evaluation of the effectiveness of
ODA funds delivered. It would not be about process, about vague connections to
improvements in development goals but about assessing the way in which the
intended recipients of the aid have experienced direct real, sustainable and
beneficial changes to the way they live. The reporting would focus on quality
over quantity, it would go beyond recording the construction of facilities to providing
an account of their use—attendance at school or health clinics with measurable
indicators to demonstrate improved education or health standards, farmers using
roads to transport their goods to markets etc. Departments and agencies would
be required to show how their projects have taken into account the maintenance
and operational costs and the skilled people need to operate or manage
facilities. It would also require the department or agency to explain how their
projects form part of a whole-of-government coherent strategic development plan
for the recipient country including projects it is intended to complement.
Recommendation 33
17.63
The committee recommends that AusAID review its Afghanistan Annual
Program Performance Report in order to ensure that the document reflects its
title—program performance report. This means that the report's main aim would
be to convey information on:
- the performance of programs—value for money;
- the program's effect on the lives of its recipients;
- the benefits delivered to intended recipients and how they
align with their needs;
- the sustainability of the benefits; and
- how programs relate to and complement other Australian-funded
programs.
It should contain a
section providing a comprehensive account of the effectiveness of Australia's
whole-of-government effort in Afghanistan.
Recommendation 34
17.64
The committee recommends that the Australian Government implement new
reporting and evaluation requirements for departments and agencies delivering
Australian ODA that are timely, consistent, transparent and more stringent.
They should also require information on:
- the aid program's objectives and how it contributes to a
coherent, whole-of-government development plan;
- the medium and long-term prospects for the sustainability of
each project within the program including allowances made for continuing
operational costs—such as salaries, maintenance and repair; and
- the monitoring and evaluation mechanisms for tracking and
assessing the effectiveness of projects after their completion.
Unless there is a
compelling reason otherwise, reporting and evaluation reports should be
publicly available from AusAID’s website.
Recommendation 35
17.65
The committee recommends that the Office of Development Effectiveness
conduct a critical analysis of the effectiveness of Australia's ODA to
Afghanistan with a particular emphasis on the sustainability of projects and Australia's
whole-of-government effort.
17.66
Although the committee believes that the best and strongest critics of
the effectiveness of aid should be the agencies themselves, it recognises the
critical role of independent scrutiny, especially of parliamentary oversight.
Parliamentary oversight
17.67
The committee does not share AusAID's confidence in the robustness of
its evaluation and reporting on Australia's ODA to Afghanistan. Professor Howes
suggested there should be more parliamentary reviews of aid.[63]
The committee agrees. It believes that a dedicated parliamentary committee is
needed to provide regular, systematic and rigorous scrutiny of Australia's ODA
and stands ready to inquire into relevant matters as they may arise.
Recommendation 36
17.68
The committee recommends that the Parliament consider establishing a
parliamentary standing committee or dedicated subcommittee of an existing
standing committee charged with examining and reporting on Australia's ODA. Among
other benefits, this committee could be the catalyst needed to improve the
standard of reporting on Australia's ODA, especially Australia's
whole-of-government effort in delivering overseas aid. It may also be a means
of raising public awareness of the work being done with Australia's ODA.
Request to Auditor-General
17.69
In the introduction, the committee highlighted the miscalculation of
Australian ODA since 2006.
17.70
With this in mind, the committee requests that the
Auditor-General consider conducting an audit of Australia's ODA to Afghanistan
with a view to determining whether the guidelines for classifying funding as
ODA are appropriate, well understood and applied properly.
Conclusion
17.71
The committee understands the difficulties confronting donors in
delivering aid to Afghanistan and accepts that some projects will inevitably
suffer setbacks. It also acknowledges the work of Australian personnel in
Afghanistan and commends their commitment. Even so, the committee believes that
the contribution that Australia is making in Afghanistan should come under
close and critical scrutiny.
Senator Alan Eggleston
Chair
Senate Foreign Affairs, Defence and Trade References
Committee
Navigation: Previous Page | Contents | Next Page