Chapter 24

Chapter 24

Evaluation and accountability

24.1      This chapter is based on the premise that evaluation improves performance. The committee starts by looking at the importance of evaluating a peacekeeping operation. It then describes the approaches taken by the ADF and the AFP to assessing their performance in peacekeeping activities. The committee considers the performance indicators needed by government to assess adequately the success or otherwise of an operation and the challenges of ensuring that lessons learnt from an operation are captured for future operations. Finally, the committee looks at the evaluation of a peacekeeping operation as an important accountability tool.

 

Evaluating peacekeeping operations

24.2      Evaluation is used to improve performance in a number of important ways. It provides feedback to decision makers so that they are better able to assess the objectives of a mission and, if required, change the mission's mandate and make informed and necessary adjustments to the conduct of the operation. Such assessments may also identify deficiencies in training and resources and can be used to make immediate improvements to the peacekeepers' preparation or equipment. In the longer term, evaluations add to the body of knowledge and understanding of peacekeeping operations and are central to developing best practice doctrine and procedures. Finally, evaluations are an important accountability tool whereby those responsible for an operation are answerable for the conduct of the mission.

 

Agencies' current evaluation practices

24.3      The committee believes that although peacekeeping is a whole-of-government undertaking, government agencies have a responsibility to assess their own performance.[1] In the committee's view, self assessment enables an agency to identify the strengths and weaknesses in its performance and provides the necessary impetus for change and improvement. It follows that agencies should have 'mechanisms in place to measure [their] effectiveness'.[2] Both the ADF and the AFP are aware of the importance of evaluating their performance in peacekeeping operations.

ADF

24.4      According to the ADF, it develops its training objectives through incorporating lessons learnt, both from its own experience and that of other countries.[3] It uses a 'Defence-wide evaluation system' to capture and disseminate those lessons. The ADF Activity Analysis Database System covers 'all ADF operations and major exercises and has been in operation for the last decade'.[4]

24.5      The ADF informed the committee that it has operational analysis teams of two to three personnel, from the Defence Science and Technology Organisation (DSTO) and the ADF. They 'deploy regularly to identify and record lessons'.[5] The teams consider lessons from ADF operations, from a Defence perspective, but also in cooperation with other government agencies. The reporting is passed up the chain of command as appropriate.[6] The ADF explained further:

...the ADF Warfare Centre and the Centre for Army Lessons act as repositories for ADF lessons learned. Previous peacekeeping operations are analysed, and lessons applied to doctrine. These are also incorporated into standard operating procedures and result in changes to tactics, techniques and procedures. In addition, these lessons inform consequent mission rehearsal exercises for elements preparing to deploy or replacing elements already deployed.[7]

24.6      The post-deployment debriefing process also provides an opportunity for the ADF to obtain feedback from their peacekeepers in order to improve procedures and operations.

24.7      It should be noted, however, that the committee received evidence suggesting there were shortcomings in the ADF's debriefing process. Captain Wayne McInnes, who served with ATST-EM, claimed that the team received no debriefing in East Timor. In his words, they basically prepared their 'kit for inspection by AQIS and 48 hours later were out of the country'. On return to Australia, there was no debrief either. He explained, 'There were no observations, lessons learnt, how can we do things better or what went wrong'.[8]

24.8      The committee discusses more fully the ADF's approach to lessons learnt later in this chapter.

AFP

24.9      As noted in Chapter 21, the AFP mission debrief is mandatory. It is conducted through an anonymous questionnaire covering three stages of deployment.[9] The committee has also referred to the joint project funded by the AFP and the Australian Research Council, 'Policing the Neighbourhood'. One of the key objectives is to develop 'an analytical framework by which Australian police assistance missions can be better understood, assessed' and modified to 'contribute effectively, equitably and sensitively to the improvement of law and order in host countries'. The AFP is also collaborating with the University of Queensland in 'the development of performance measures to assist in evaluating AFP contributions to peace and stability operations, and capacity development missions'.[10]

24.10         These projects indicate the willingness of the AFP to address seriously the matter of evaluation. This approach is particularly important in light of the literature that points to deficiencies in the way peacekeeping operations are evaluated. The committee now looks at the difficulties organisations have in determining their effectiveness in a peacekeeping operation.

 

Performance indicators

24.11         In its report on public diplomacy, the committee discussed the difficulties measuring the effectiveness of government programs. For example, Mr Meert, ANAO, noted that because it is easy to measure, 'a lot of the agencies are stuck at the activity measure' and hence 'struggling with how to determine effectiveness'.[11] He suggested that a range of indicators are needed to ascertain whether the activities being undertaken are 'having the desired effect'. He noted that there are methods available to measure changes in attitudes or perceptions. He suggested, however, that 'One indicator on its own may not give you the result but a range of indicators may give you that indication'.[12] Based on the advice from ANAO, the committee noted that if ANAO were to undertake a performance audit, it would likely concentrate on the performance indicators a department uses to evaluate the effectiveness of its programs and how it sets targets.[13] ANAO would be looking to see whether a department has the mechanisms in place to evaluate its own programs.

24.12         This observation on having performance indicators is particularly pertinent for peacekeeping operations. The United States General Accounting Office noted in 2003 that the UN DPKO acknowledges that it 'needs better indicators by which to measure the progress peacekeeping operations are making in attaining sustainable peace. It stated:

Although some measures for the peacekeeping operations are quantifiable [include minimising and containing cease-fire violations and opening roads and removing roadblocks], the United Nations faces challenges in developing results-oriented measures about conditions in the country that the peace operations are supposed to improve.[14]

24.13         It stated further:

Although U.N. missions are using measures of progress for their operations, most measures are tasks and outputs rather than measures of underlying conditions in the country that the peace operation is to improve.[15]

24.14         For example, in East Timor a key objective was to increase the capacity of the national police force to provide internal security. The US General Accounting Office noted, however, that the number of police 'does not measure the quality of their training and whether they are improving security in the country'.[16] It warned against relying on measures that are process oriented and do 'not measure changes in country conditions that the peacekeeping operations were working to improve'.[17]

24.15         When considering exit strategies in Chapter 8, the committee made similar findings about having performance indicators that are credible and useful. It noted that an exit strategy should relate back to the objectives set out in the mandate which must be clearly defined, realistic and attainable and based on a sound understanding of all facets of the problem. The committee found further that the strategy should contain milestones against which the outcomes of peacekeeping and peace enforcing measures can be assessed. These benchmarks, however, should be more than indicators of 'technical' achievements and while identifying key attainments such as an election, should also take cognizance of, and mark progress toward, the ultimate goal of sustainable peace.[18]

24.16         Sergio Vieira De Mello, Special Representative of the Secretary-General and Transitional Administrator for East Timor, also used East Timor as an example to highlight the need for substance in determining milestones. He noted that success would be judged not just on the number of schools rebuilt or roofs replaced. The yardstick would be the extent to which the mission assists the people of East Timor to fully realise 'their independence as masters of their own future' and their own 'independent State'. Using what are called 'technical achievements' may give a false reading of the situation.[19]

24.17         In its consideration of exit strategies, the committee also noted the danger of relying on indicators that do not provide a complete assessment of the operation. The committee acknowledged that most observers agree that the ADF pacification in Solomon Islands was a success with over 6,000 militiamen arrested, over 9,000 charges laid and more than 3,000 guns confiscated. It went on to cite the findings of a recent report that concluded that, while civil stability had brought security, these gains would 'prove temporary if the underlying economic stagnation that led to civil unrest is not addressed'. Thus, while measures such as the number of local police trained, schools rebuilt or weapons confiscated are quantifiable, they may mask the fact that deep-seated conflicts remain with the potential to flare up and return violence and instability to the country.[20]

24.18         The committee notes that Assistant Commissioner Paul Jevtovic in an interview stated that one of the indicators the AFP uses to measure its performance is 'how we develop our counterparts in that country [Solomon Islands], so our success is ultimately measured on the development of good future leaders'.[21] This benchmark is an improvement on just quantifying the number of police trained and is looking to measure change that indicates real achievements have been made. The question remains, however, what indicators the AFP would use to determine 'the development of good leaders'. Having credible and useful performance indicators is critical to effective evaluation but they are also difficult to measure.

Committee view

24.19         The committee believes that to be effective, performance indicators must anticipate the difficulties of achieving sustainable peace. They should not be tasks or outputs but rather objective result-oriented measures that effectively plot progress toward achieving the objectives of the mission. In this way, they provide the necessary feedback to determine whether there is a need to change the objectives and conduct of the mission and what these changes may be. The committee accepts, however, that formulating performance indicators that measure shifts in attitude or fundamental changes in a society is not easy. Nevertheless, the committee believes that it could and should be done.

24.20         Another important reason for evaluating performance is to build up a body of knowledge and understanding that can be applied to future operations.

 

Capturing lessons learnt

24.21         The UN has for sometime been concerned that it did not have a process 'for converting lessons into policies and procedures that could guide subsequent operations'. In a report to the General Assembly in 2007, the Secretary-General stated:

To be 'learned', lessons identified need to be validated and endorsed by the Organization in the form of standardized guidance materials, and guidance needs to be disseminated and its implementation monitored. The identification of lessons and good practices does not itself lead to performance improvement. It must be accompanied by a process that seeks to implement improvements in the way operations are managed and conducted, which must also be monitored and re-evaluated on an ongoing basis.

...Learning lessons may be a natural process, but sharing and implementing lessons across different missions is not. Although most staff members say that they regularly engage in learning lessons, unless systematic efforts are made to document and share those lessons, their impact remains limited to local teams.[22]

24.22         As noted earlier, the ADF asserted that it incorporates lessons learnt from missions to develop its training objectives. It has operational analysis teams that deploy regularly to identify and record lessons, and the Warfare Centre and the Centre for Army Lessons act as repositories for ADF lessons learnt. In this way, previous peacekeeping operations are analysed, and lessons applied to doctrine.

24.23         Professor Raymond Apthorpe and Mr Jacob Townsend, however, doubted the efficacy of the ADF's learning lessons process. In general, they noted that many organisations 'tend to misunderstand the phrase "lessons learnt"':

At the end of a peacekeeping operation, identifying problems and summarising these under the title 'lessons learnt' obscures the otherwise obvious point that the lesson has yet to be learnt—or the problem would not have occurred. Easy access to in-house evaluations of Australian Defence Force (ADF) peacekeeping operations eluded us, so we will restrict ourselves to the general point that identifying problems is easy; the hard part is making real and lasting change.[23]

24.24         Professor Apthorpe and Mr Townsend therefore suggested that the committee investigate 'the degree to which [organisations] have learned any lessons from their "lessons learnt"'. They continued:

A warning sign would be the same issues re-appearing in many lessons learnt sections of evaluations. In our experience, it is not uncommon that an organisation seems to be cutting and pasting 'lessons learnt' from one evaluation to the next, which indicates dysfunction in its learning processes.[24]

24.25         Furthermore, in their view, 'lessons are often not presented in a learnable form' and 'expressed in a way that cannot inform future policy or strategy'. They argued:

Like proverbs, two equally sensible suggestions can be in direct contradiction...Like proverbs, these 'lessons' are post facto comments that cannot prepare us for the future unless we place them in context. If organisations are to learn from their experiences, then the 'lessons learnt' process needs to identify contributing contextual factors that can be recognised in new scenarios, so that something learnt in one context might be transferable to another.[25]

24.26         The committee believes that it would be of value to have an audit of the ADF's operational analysis teams, the Warfare Centre and the Centre for Army Lessons to determine the effectiveness of their work in capturing the lessons learnt from current and recent peacekeeping operations.

 

Request to Auditor-General

The committee requests that the Auditor-General consider conducting a performance audit on the mechanisms that the ADF has in place for capturing lessons from current and recent peacekeeping operations including:

  • the adequacy of its performance indicators;
  • whether lessons to be learnt from its evaluation processes are documented and inform the development or refinement of ADF's doctrine and practices; and
  • how these lessons are shared with other relevant agencies engaged in peacekeeping operations and incorporated into the whole-of-government decision-making process.

24.27         The committee has confined this request to the ADF because, as noted earlier, the AFP has commissioned the University of Queensland to develop performance indicators. An ANAO audit should provide guidance for the AFP, and indeed for all relevant government agencies, in further developing their performance indicators. The audit should also provide an insight into how effectively government agencies are sharing information from lessons learnt and contributing to improving Australia's whole-of-government performance in peacekeeping activities.

Recommendation 34

24.28         The committee recommends that the relevant government agencies jointly develop standard measurable performance indicators that, where applicable, would be used across all agencies when evaluating the effectiveness of their peacekeeping activities (also see Recommendation 36).

 

Whole-of-government evaluation

24.29         The government also confronts the challenge of devising effective performance indicators when assessing its whole-of-government performance in a peacekeeping operation. The task of measuring the effectiveness of the whole-of-government contribution means examining issues such as interoperability and CIMIC. Such matters raise questions about who should assess this overall performance and how an agency's assessment of its performance feeds into a whole-of-government appraisal. Having credible and useful performance indicators is also important. Ensuring that information is shared and lessons learnt across government agencies poses problems for the government. Also, the government must decide who is to monitor and ensure that the lessons learnt are implemented.

24.30         In this regard, the committee believes that the proposed peacekeeping institute would have a vital role in the evaluation and continuous improvement of Australia's peacekeeping performance. It believes that the institute is the ideal mechanism for ensuring that Australia has:

  • appropriate performance indicators to measure the success or otherwise of its whole-of-government performance in peacekeeping activities;
  • a repository for lessons learned; and
  • a central body responsible for ensuring that doctrine and practices are developed and refined in light of past experiences.

24.31         The establishment of the institute would not in any way counter or make redundant the work on peacekeeping of the ANAO, should it undertake the audit, or of projects such as that underway by the University of Queensland. The institute would complement and indeed add value to the findings of such organisations. The committee believes that it could play a vital role in building a culture of learning and continuous improvement.

 

Independent assessment

24.32         The committee also notes the importance of obtaining independent outside evaluation as a means of gauging the performance of Australian peacekeepers and the success of a peacekeeping operation.[26] Such information comes from the government and people of the host country as well as Australia's partners in an operation. The proposed institute could also take responsibility for gathering this type of information.

 

Accountability

24.33         Accountability lies at the heart of efforts to improve the government's management of its peacekeeping activities. Not only does it provide a means for parliamentary and public scrutiny of government expenditure in this area but it also allows people outside the executive government to make an informed assessment of the government's performance. In this way, Australians may not only feed into the overall evaluation of Australia's performance in peacekeeping operations but they are better placed to hold the government answerable for its performance. 

Annual reports

24.34         The starting point for examining Australia's involvement in peacekeeping is to understand the nature of the contribution that the government makes through each of its departments and agencies. It is difficult to find comprehensive information about Australia's whole-of-government commitment to peacekeeping operations. Defence provides clear information in its annual report and on its website about current international deployments, including each operation's aims, the number of personnel and other capabilities deployed.[27] However, Defence does not distinguish peacekeeping operations from other deployments. Its annual report lists operations under the headings 'Operations contributing to the security of the immediate neighbourhood', 'Operations supporting wider interests' and 'Peacetime national tasks'.

24.35         Referring to the Defence Cooperation Program (DCP), outlined in Chapter 20, the committee is concerned that Defence's Annual Report 2006-2007 offered no examples of the expenditure in the Defence Cooperation Program's $80 million (approx.) capacity-building activities.

24.36         Similarly, the AFP provides information about its international missions on its website and in its annual report.[28] The latest AFP Annual Report has been enhanced by the inclusion of a snapshot providing a summary of the number of personnel deployed to each operation as at the end of the financial year.[29] The AFP also does not identify deployments as 'peacekeeping operations': its snapshot distinguishes missions as 'United Nations missions', 'AFP Capacity Building missions' and 'Regional Assistance missions'.

24.37         Several other government departments refer to their involvement in peacekeeping operations in their annual reports. There is, however, no centralised source providing comprehensive information about the whole-of-government contribution. DFAT's annual report provides an overview of its activity with relevant countries or regions in support of its Outcome 1: 'Australia's national interests protected and advanced through contributions to international security, national economic and trade performance and global cooperation'.[30] In this context DFAT reports, for example, that it 'continued to coordinate Australia's whole of government contribution to the Regional Assistance Mission to Solomon Islands (RAMSI)'.[31]

24.38         In response to the conclusions of an ANAO report, DFAT's 2006–07 annual report also lists the government agencies contributing to RAMSI. This is an improvement, but falls short of the ANAO's suggestion, which was to include at a minimum each agency name, its contribution in financial and human resource terms and a brief description of its role.[32]

24.39         In this inquiry, the committee considered Australia's involvement in a number of current peacekeeping operations, not only RAMSI. In response to a question as to whether the ANAO's recommendation would be applied to other operations with multi-agency input, DFAT responded that this 'will depend on the nature of the individual operation'.[33]

24.40         Despite the inadequacy of easily accessible information, it is clear from the nature of the missions and their mandates that Australia's contribution to peacekeeping operations has extended well beyond military and police involvement. In an answer to a question on notice, DFAT informed the committee that the following departments and agencies have contributed staff to peace operations in Bougainville, East Timor or Solomon Islands: Defence, AFP, DFAT, Attorney-General's Department, AusAID, Australian Electoral Commission, Customs, Finance, Office of Financial Management, National Archives, Treasury and Department of Veterans' Affairs.[34]

Committee view

24.41         Australia's contribution to peacekeeping operations now extends well beyond the military. It is important that this whole-of-government contribution is accompanied by whole-of-government reporting, so that the Parliament and the Australian public can identify the size and nature of the resources allocated by government to peace operations.

Recommendation 35

24.42         The committee recommends that the Australian Government designate an appropriate agency to take responsibility for the whole-of-government reporting on Australia's contribution to peacekeeping. This means that the agency's annual report would include a description of all peacekeeping operations, a list of the contributing government agencies, and, for each relevant agency:

  • a description of its role in the operation;
  • the agency's financial contribution to the operation during that reporting year;
  • the peak number of personnel deployed by the agency during the reporting year and the date at which the peak occurred; and
  • the number of personnel deployed as at the end of the reporting year.

Recommendation 36

24.43         In light of the committee's discussion on the adequacy of performance indicators, the committee also recommends that the agencies reporting on peacekeeping activities provide in their annual reports measurable performance indicators on the effectiveness of these activities.

24.44         Following on from this consideration of how the government reports its whole-of-government contribution to peacekeeping, the committee notes that there appears to be a similar lack of coherence in explaining the government's policy on peacekeeping.

 

White paper on peacekeeping

24.45         The committee is of the view that the changing nature and expanding scope of Australia's involvement in peacekeeping operations have profound implications for Australia as a member of the UN and a long-time contributor to peacekeeping missions. Further, peacekeeping operations are no longer the domain of the military, with a range of government agencies and NGOs now needing to work together as an integrated team to achieve the mission's objectives. Peacekeeping operations can also be costly and dangerous undertakings, with failure a real prospect. Moreover, international doctrine and practice on issues such as the responsibility to protect, the significance of exit strategies, interoperability, CIMIC and women and peacekeeping, to name just a few, continues to develop and poses challenges for policy makers.

24.46         These developments have a direct bearing on the formulation of Australia's policy on peacekeeping and on its decisions about the composition and structure of its deployment, the training and preparation of its personnel and how it coordinates its effort. Yet to date, there is not a single coherent policy document that covers the joint efforts of all government agencies contributing to peacekeeping. The committee believes that it is time for the government to produce a white paper on peacekeeping that would explain the whole-of-government policy on peacekeeping operations, including the factors that shape the government's decision on Australia's involvement in such operations. The committee envisages that the paper would discuss the whole range of complex matters involved in peacekeeping, including matters raised in this report. It would also provide detail on the recently-announced Asia–Pacific Centre for Civil–Military Cooperation and how this initiative fits into the broader policy on peacekeeping.

24.47         The committee does not favour the proposed white paper forming a subset of the new Defence White Paper. Although peacekeeping may figure prominently in this document, the committee believes that because of the involvement of many key government agencies in today's peacekeeping operations, the government should produce a separate white paper on peacekeeping. The production of a white paper would provide the government and the relevant agencies with the opportunity to review their policies and practices and to better understand how their activities contribute to the whole-of-government effort. It would also require the government to articulate its policy across the full spectrum of Australian peacekeeping activities, thereby allowing more informed public scrutiny of this important area of government engagement.

Recommendation 37

24.48         The committee recommends that the Australian Government produce a white paper on Australia's engagement in peacekeeping activities.

 

Conclusion

24.49         The committee has underlined the importance of evaluation as a means of improving performance both in the short and long term. It has also referred to the role that the committee's proposed peacekeeping institute could have in evaluating operations and capturing the lessons learnt. In the following chapter, the committee discusses in detail the proposed institute.

Navigation: Previous Page | Contents | Next Page