Chapter 2 Audit Report No.5 2011-12

Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework

Introduction

2.1                   Performance reporting mechanisms have enabled the Parliament and the public to better understand government operations and how public funds are spent to achieve the policy objectives of government. Over time, the focus has shifted from models which showed where funds were being sent, to models that gave a clearer picture of eventual outcomes resulting from the expenditure of public money.[1]

2.2                   The first of these models was the Outcomes and Outputs Framework which was adopted from the 1999-2000 financial year until 2008-2009.[2]

Adoption of the Outcomes and Programs Framework

2.3                   In 2005, the then Shadow Minister for Finance, Mr Lindsay Tanner MP, released a discussion paper, Operation Sunlight: enhancing budgetary transparency, which noted deficiencies in the Outcomes and Outputs Framework:

The government’s outcomes and outputs framework was intended to shift the focus of financial reporting from inputs (programs, expenses and recipients) to outputs and outcomes i.e. actual results. While this is worthy in theory, it has not worked. Basic information on inputs was lost in the changeover, and reporting of outcomes is seriously inadequate.[3]

2.4                   Following a change of government in November 2007, the then Minister for Finance and Deregulation, the Hon Lindsay Tanner MP, requested that Australian Democrats Senator Andrew Murray undertake a review into the Operation Sunlight discussion paper. The subsequent report, known as the Murray Review, made 45 recommendations, including one which sought to improve reporting of government activity to the public by reporting expenditure at the program level. This recommendation was agreed to by the government, and the Outcomes and Programs Framework was developed. Amongst the key elements of this framework was the requirement for annual performance reporting on the delivery of programs and achievements against a set of Key Performance Indicators (KPIs).[4]

Key Performance Indicators

2.5                   The purpose of KPIs within the Outcomes and Programs Framework is simple: to make available information that can provide an accurate but succinct performance story of the results of government actions. The Department of Finance and Deregulation (Finance) policy advises that this performance information should be able to be used by the public to ensure a government is living up to its commitments, used by the government to assess the performance of a program, and used by program investors to determine opportunities for program improvement or modification.[5]

2.6                   To create a useful KPI, there are several methodologies that can be used. Ensuring KPIs fit the chosen methodology ensures they are of use to both the entity, and stakeholders who wish to use the KPIs to measure progress through reporting mechanisms such as Portfolio Budget Statements (PBSs) and annual reports.

Linking KPIs to annual reports and PBSs

2.7                   Under the Outcomes and Programs Framework, government entities are required to report against the approved list of programs for which they are responsible in PBSs and in their annual reports. Guidance to agencies for mandatory performance reporting notes that ‘descriptions of processes and activities should be avoided. Rather, reporting should be aimed at providing an assessment of how far the agency has progressed towards outcomes.’[6]

2.8                   KPIs should be drafted in a way that allows for direct comparison with data contained in PBSs and departmental annual reports. Providing the user with a ‘clear read’ through these documents constitutes best practice for government entities.

The ANAO Audit

Audit objectives and scope

2.9                   The objective of this audit was to assess how effectively entities had developed and implemented appropriate KPIs to support stated program objectives. To address the audit objective, the Australian National Audit Office (ANAO):

n  undertook a desktop review of the published effectiveness KPIs for 89 programs across 50 Financial Management and Accountability Act and Commonwealth Authorities and Companies Act entities within the General Government Sector;

n  supplemented this desktop review with more detailed analysis of four entities—the Australian Customs and Border Protection Service (Customs); Fair Work Australia (FWA); the National Film and Sound Archive (NFSA); and the Department of Resources, Energy and Tourism (RET)—including the reporting of performance in each entity’s annual report; and

n  assessed the role of Finance in administering the Outcomes and Programs Framework, including the preparation of guidance material for entities.[7]

Overall audit conclusion

2.10               The ANAO concluded that many entities found it challenging to develop and implement KPIs that were effective in providing quantitative and measurable information to allow for an informed assessment and reporting of achievements against stated objectives.[8]

2.11               According to the audit report, entities tended to rely on qualitative effectiveness KPIs, which were difficult to measure. The ANAO suggested that qualitative indicators may be better used to supplement quantitative indicators, to provide insights into the factors responsible for the success or otherwise, of a program.[9]

2.12               The desktop review of 89 programs, across 50 entities in the General Government Sector, conducted by the ANAO found that many programs had KPIs that were one, several, or all of the following:

n  non-specific;

n  non measurable;

n  not clear as to whether they were achievable;

n  not relevant or not linked to program objectives; and

n  not timed.[10]

2.13               Overall, a third of the entities reviewed had effectiveness KPIs that were appropriate in terms of being specific, measurable, achievable, relevant and timed, a third were mixed (often differing significantly at the program level), and a third required much further development.[11]

2.14               Further, the ANAO found that performance information in many annual reports was not sufficient to allow external stakeholders to understand the progress being made by entities in meeting program objectives, and suggested that trend data would assist stakeholders in determining whether performance was better or worse than previous years.[12]

ANAO recommendations

Table 1           ANAO recommendations, Audit Report No.05 2011-12

1.	To develop more meaningful and measurable effectiveness Key  Performance Indicators (KPIs), the ANAO recommends that entities build into their business planning processes the requirement to: ·         periodically review program objectives to provide assurance that they are clearly defined and well suited for their purpose; and ·         develop KPIs that have an appropriate emphasis on quantitative and measurable indicators, including targets. Customs, FWA, NFSA, RET response: Agreed.
2.	The collection and use of information on costs associated with the delivery of individual programs is an important component of the Government’s Outcomes and Programs Framework. To support this reform the ANAO recommends that entities assess the extent that they currently use relevant costing information to identify program support costs, and take steps to allocate these costs to applicable programs. Customs, FWA, NFSA, RET response: Agreed.
3.	To ensure the ongoing currency and appropriateness of the Outcomes and Programs Framework, the ANAO recommends that the Department of Finance and Deregulation: ·         reviews the development and implementation of effectiveness KPIs to determine the extent to which expected improvements in the measurement and achievement of program objectives is being realised; ·         includes in its guidance to entities a suggested diagnostic tool and methodology, such as the SMART criteria, to further assist entities to review and evaluate the usefulness of their KPIs; and ·         develops more expansive policy guidance for entities on how to reference performance reporting for programs delivered through national agreements. Finance response: Agreed-in-principle.

 

The Committee’s review

2.15               The Committee held a public hearing on Wednesday 29 February 2012 with the following witnesses:

n  Australian National Audit Office

n  Department of Finance and Deregulation

2.16               The Committee took evidence on the following issues:

n  progress towards implementing ANAO recommendations

n  KPI methodologies

n  developing KPIs

n  linking KPIs in cross-agency projects

n  monitoring and review of KPIs, including the new role of the Auditor‑General

n  KPI reporting of Federal-State agreements.

Progress towards implementing ANAO recommendations

2.17               The Committee asked Finance about progress being made towards implementing the recommendations made in the ANAO audit report.

2.18               Finance indicated that the findings of the audit were not a major surprise to the Department, as there had been conceptual and practical difficulties surrounding KPIs identified in the past.[13]

2.19               Finance advised that it had commenced a review of approximately 600 of the 5,500 KPIs reported in departmental Portfolio Budget Statements prior to the audit report, and that the findings of this review would enable further understanding of the issues identified by the ANAO in its report. Further, Finance advised that it planned to make recommendations to the Government to further improve the guidance it provided to entities in relation to KPIs.[14]

2.20               The ANAO noted that it was working with Finance to encourage the implementation of its recommendations and that Finance would receive full assistance in this matter from the ANAO.[15]

KPI methodologies

2.21               The purpose of KPIs within the Outcomes and Programs Framework is to provide information that can give an accurate but succinct performance story of the results of government actions. Policy provided by Finance advises that this performance information should be able to be used by the public to ensure a government is living up to its commitments, the government to assess the performance of a program, and program investors to determine opportunities for program improvement or modification.[16]

2.22               To ensure KPIs are able to allow for the measurement and assessment of the achievement of program objectives, it is important that they are drafted using a clear methodology. Drafting KPIs without using a clear methodology potentially leads to KPIs that do not enable comparison with a program’s objectives, and its eventual outcomes.

2.23               Entities are required to report against the approved list of programs for which they are responsible in PBSs, and in their annual reports. Guidance to entities for mandatory performance reporting through annual reports notes that ‘descriptions of processes and activities should be avoided. Rather, reporting should be aimed at providing an assessment of how far the agency has progressed towards outcomes.’[17]

2.24               There are several methodologies that can be used to create KPIs, with the ANAO referencing the use of the SMART criteria in its report.[18] The SMART criteria notes that KPIs must be:

n  Specific – Clear and concise to avoid misinterpretation of what is to be achieved

n  Measureable – Able to be quantified and compared to other data to show trends

n  Achievable – Practical, reasonable and credible given expected conditions

n  Relevant – Informative and useful to stakeholders

n  Timed – Specifying a timeframe for achievement and measurement.

2.25               The audit found that Finance had removed references to the SMART criteria in its guidance to entities about the preparation of KPIs for the 2010-11 Budget and that guidance related to the development of KPIs was reduced in length and detail. The ANAO suggested, given responses by agencies in regard to the usefulness of guidance provided by Finance, that ‘it would be beneficial if Finance revisited this previous guidance, and suggested a diagnostic tool and methodology’.[19]

2.26               Taking into consideration the audit report findings, the Committee enquired whether Finance had reintroduced the references to the SMART criteria in its latest advice to agencies, or if it had included alternative advice to assist agencies in the development of KPIs.

2.27               Finance replied that the removal of the SMART criteria had been part of an attempt to streamline the guidance provided to agencies. Finance agreed with the ANAO’s finding that there was value in including a methodology in the guidance provided to entities and that the inclusion of a methodology or methodologies would form part of Finance’s advice to entities in future.[20]

2.28               When pressed as to whether methodologies would form part of the advice provided by Finance to entities for the 2012-13 Budget, Finance noted that it would also take time for entities to familiarise themselves with the guidance provided, and that Finance planned to have guidance including methodologies for drafting KPIs out to agencies well in advance of the 2013-14 Budget.[21]

Developing KPIs

2.29               Finance also provides support to entities in relation to Outcomes and Programs Framework reporting responsibilities through two key areas – the Budget Framework Branch (BFB) and Agency Advice Units (AAUs).

2.30               The Committee further examined the role played by Finance’s BFB in the development of KPIs. The BFB develops and implements proposals to enhance and simplify aspects of financial frameworks, and provides guidance and advice on PBSs, performance measurement, and related reporting.

2.31               The Committee sought further clarification on the differences between the BFB and AAUs, with Finance advising:

[BFB] is responsible for the guidance that Finance produces on how KPIs should be developed and how Portfolio Budget Statements should be developed.

[AAUs] have a role in providing advice to government on costing proposals but also on the opportunities related to new policy proposals, costings and the like that present themselves at a whole-of-government level.[22]

2.32               One of the roles of an AAU is to assist its related entities with the preparation of KPIs. A Finance memorandum circulated to all entities for the preparation of the 2010-11 PBSs advised:

Agencies need to advise their AAU in Finance if there are any changes to their KPIs for information purposes so that trends can be recorded. If a KPI changes from last year’s budget of Portfolio Additional Estimates Statements, agencies should footnote in the KPI table, a summary of the change and whether they have met the previous KPI at the program level.[23]

2.33               AAUs within Finance during the course of the audit advised that they often had not been contacted by entities planning to change or amend KPIs; and that AAUs did not review or otherwise assess the KPIs developed by entities.[24]

2.34               The Committee asked Finance whether AAUs should take more responsibility for reviewing and assessing KPIs developed by entities to ensure their appropriateness and whether they fit recognised methodologies.

2.35               Finance indicated that rather than identify a specific unit, it is preferable to look at the various issues and consider how the Department as a whole can contribute. Finance confirmed its commitment to improved agency interactions and support.[25]

2.36               Following the public hearing, the Committee sought additional advice from the ANAO and Finance as to how to ensure that the real world impact of Government programs is always judged.

2.37               Both the ANAO and Finance noted that some programs lend themselves to quantitative assessment; especially those with tangible products or that are quite specific and deal with only a single element of an issue. Examples include programs such as the provision of a specific training course to a set number of teachers or other products that focus on delivery of payments or grants.[26]

2.38               However, ANAO and Finance also noted it is much more challenging to measure performance where a program seeks to address less tangible areas, with complex and multi‑factorial issues, such as homelessness or the decline of regional Australia.[27]

Linking KPIs in cross-agency projects

2.39               Finance guidance recommends that entities link program and budgetary reporting to programs in other Commonwealth entities that contribute to the same government objective. The audit found that linking of programs between entities was undertaken in different ways in different entities and no one common method of linking was evident.

2.40               The audit report notes:

While responsibility for determining the most appropriate approach to coordinating programs rests with individual entities, more specific Finance guidance to entities on how to link similar programs that straddle a number of entities, but contribute to the same government objective, would be beneficial in promoting consistency.[28]

2.41               Noting its ongoing interest in this area, the Committee asked whether Finance had made any decision to provide more specific guidance to entities on how to link similar programs that extend over a number of entities but contribute to the same government objective.

2.42               Finance noted the difficulties in linking KPIs in cross-agency projects, identifying it as a broader issue on how accountability works, and how reporting takes place. Finance indicated that there was a broader structural and framework dimension to the issue, as it also required connections across PBSs.[29]

Monitoring and review of KPIs

2.43               The ANAO emphasised the importance of agencies taking a strategic approach to the selection of KPIs that provide information that will assist in the evaluation of program performance over time. The ANAO also suggested a higher profile be given to the importance of agencies undertaking periodic program evaluation, as a way to encourage agencies to focus on KPI development in the program design phase.[30]

2.44               Finance summarised the way forward noting that:

To help ensure that we are in the best position to judge the real‑world impact of government programs we need to be able to both (a) assess the performance and effectiveness of programs, and (b) undertake effective whole-of-government coordination and implementation.[31]

2.45               Finance also advised that several pieces of work are underway, including a review of the Key Performance Indicators of Commonwealth programs to inform the development of meaningful advice and ensure KPIs are incorporated within a ‘coherent reporting framework’.[32]

2.46               In addition, recent amendments to the Auditor-General Act 1997 have given the Auditor-General the power to undertake audits of KPIs as presented in an agency’s Portfolio Budget Statements. In its appearance before the Committee, the ANAO noted that a project plan had been developed to undertake a pilot audit of a selection of agencies, with a view to the findings informing the development of the ANAO’s KPI audit methodology. [33]

2.47               The ANAO also advised that it has been working with Finance to ensure the audit processes align with developments of the framework.

It is envisaged that this relationship will provide some additional synergy in the enhancement of the performance information to the parliament and the public.[34]

KPI reporting of Federal-State agreements

2.48               The Committee has previously looked into the reporting of outcomes obtained from programs funded through COAG arrangements, recommending that the Department of the Prime Minister and Cabinet and other central agencies investigate steps to ensure PBSs and annual reports provide a more comprehensive picture of the performance and outcomes of projects funded under national partnerships across government.[35]

2.49               The audit report indicated that payments to State and Territory governments for the delivery of programs funded under COAG arrangements are included in the Treasury’s PBSs, and Treasury does have some KPIs associated with these payments. However, the report also noted that Treasury KPIs relating to these payments were solely concerned with the process of providing payments, rather than measuring the objectives associated with the programs for which the payments are made.

2.50               Finance’s guidelines indicate that Commonwealth entities affected by Federal Financial Relations should expand any non-financial information provided for the planned performance of the programs, and to link or cross-reference to programs where payments are made by Treasury.[36]

2.51               The audit found that there is currently only limited guidance for entities on how to incorporate the performance of programs funded under national agreements into PBSs and annual reports. Further, there was variability in the way entities included KPIs for those programs in their own PBS, with reporting either at a very high level or in some cases non‑existent.[37]

2.52               Finance noted that it had commenced discussions with the Department of the Prime Minister and Cabinet, the department responsible for guidance related to annual reports, expressing optimism that there would be opportunities to improve reporting on the performance of national agreements through both PBSs and annual reports.[38]

2.53               In an answer to a question on notice, Finance noted it was still ‘exploring options’ to align performance reporting information in the National Agreements of the Intergovernmental Agreement on Federal Financial Relations Framework with the PBSs to assist in establishing a ‘clear read’ between documents.[39]

Committee comment

2.54               Taking into consideration the findings in the audit report and the evidence provided by the ANAO and Finance, the Committee raises the following issues for attention:

n  guidance provided by Finance to entities to develop and report against KPIs has been reduced in detail over recent years;

n  while Finance recommends that entities link program and budgetary reporting to programs in other Commonwealth entities that contribute to the same government objective, there is no common method of linking;

n  current communication practices between agencies and their relevant AAUs in Finance has resulted in sub-optimal outcomes for both agencies and AAUs; and

n  there is only limited guidance for entities on how to incorporate the performance of programs funded under national agreements into PBSs and annual reports.

2.55               The conclusions made by the ANAO through its audit report suggest that better information needs to be made available to entities by Finance to improve the quality of KPIs developed by entities, and that entities need to more clearly consider that their KPIs should be measured against their annual reports and PBSs.

2.56               The Committee notes the comment made by Finance that the findings of the audit did not come as a surprise to the Department. However, if this is the case, Finance should have been able to provide advice to the ANAO that it was doing more to rectify the perceived problems prior to the ANAO concluding its audit.

Development of KPIs

2.57               The removal of detailed guidance that was previously provided by Finance to entities to assist them in the preparation of KPIs is an issue of some concern to the Committee. A reduction in guidance can be warranted if the guidance is no longer relevant, or creates ambiguity or confusion. However, this justification does not seem evident in the removal of a suggested methodology for the drafting of KPIs.

2.58               The Committee considers it prudent to reinstate KPI methodologies in the Finance guidance provided to agencies as soon as practicable to ensure that KPIs are fit for purpose. Based on the findings in the ANAO report and evidence heard, the Committee is of the view that clear methodologies presented to agencies would provide a tool to improve the quality and usefulness of KPIs.

2.59               The Committee notes Finance’s comment that one methodology may not suit all circumstances.[40] However, providing at least an option for a structured starting point may be helpful to agencies.

 

Recommendation 1
	That the Department of Finance and Deregulation include at least one recognised Key Performance Indicator methodology in its written guidance to government entities about the preparation of Key Performance Indicators.

 

2.60               To ensure that appropriate methodologies are used, and to assist in a clear read over time, entities should explicitly state which methodology has been used for the drafting of each KPI.

 

Recommendation 2

The Joint Committee of Public Accounts and Audit recommends that: n  the Department of Finance and Deregulation provide advice to all government entities that when providing new or amended Key Performance Indicators (KPIs) to their relevant Agency Advice Unit, the  methodology used in the preparation of the KPIs must also be available for review; and n  Finance consult with the Department of the Prime Minister and Cabinet to consider a requirement for agencies to state the ‘KPI methodology used’ in their annual reports.

Communication and engagement

2.61               Evidence contained in the audit report and discussed at the Committee’s public hearing suggested the relationship between an entity and its relevant AAU generally relied upon the entity providing information to the AAU.

2.62               While the Committee agrees that Finance as a whole can help to improve entities performance through better interactions, it is clear that relationships between entities and their relevant AAUs requires improved communication and interaction, and a more proactive stance on the part of Finance. Given AAUs are often not contacted when an entity chooses to change or amend a KPI, it is clear that some entities are unaware of the assistance Finance can provide in this area.

2.63               While an AAU is primarily required to provide advice to an entity when advice is sought, it should also be looking to ensure that entities are meeting their requirements. That AAUs play no detailed role in reviewing and assessing KPIs is perhaps understandable given the autonomous nature of entities, however, there is a role here for Finance to play should an entity require advice. Further, AAUs should be examining KPIs when they are submitted, checking that they fit an agreed methodology, and that they enable clear comparison with an entity’s PBS and annual report.

2.64               The Committee is somewhat heartened by evidence provided by Finance in the review of other audit reports that Finance is beginning to proactively engage agencies, but special attention needs to be paid to the role AAUs can play in ensuring agencies meet their obligations.

Progress

2.65               The Committee notes the review currently being undertaken into approximately 10 per cent of the total number of KPIs currently active in the Australian Public Service and believes the findings of this review will enable Finance to better engage entities that are currently not preparing KPIs that conform to best practice.

2.66               The opportunity this review presents for Finance also extends to an opportunity to improve interaction between Finance and entities. The Committee believes that once entities with KPIs that do not conform to best practice have been identified by Finance, they should be directly engaged by the relevant AAU to improve the quality of their KPIs. These agencies should then be reassessed by Finance in the following financial year to determine whether improvements have been made following assistance by Finance.

2.67               The Committee is encouraged that there will be improvements in the KPIs of entities, as it is clear that both the ANAO and Finance are committed to working together to improve outcomes in this area. Full support and implementation of the ANAO recommendations and the subsequent recommendations by this Committee will further enhance improvements. 

2.68               Giving the Auditor-General the power to assess KPIs through the audit process should allow for detailed and targeted assessment of entities that have systemic difficulties in drafting KPIs that allow for clear comparison with PBSs and departmental annual reports.

2.69               The Committee believes the Auditor-General’s first pilot audit into the KPIs of a selection of agencies will provide a ‘real world’ view of Finance’s implementation of the recommendations. However, the Committee would like additional information detailing the integration of the development of KPIs for inclusion in PBSs, reporting in agency annual reports and KPI audits being undertaken by the ANAO.

2.70               The Committee notes work is being done to align performance reporting information in the National Agreements of the Intergovernmental Agreement on Federal Financial Relations Framework with the PBSs, although this appears to be in its early stages.

2.71               Further, the Committee wishes to re-emphasise the findings of its inquiry into national funding agreements, which found that development of KPIs had been poor, and that better quality data was needed to ensure the development of effective KPIs. The Committee notes that little progress has been made on this issue, as evidenced by the findings of several COAG Reform Council reports[41] and believes that more needs to be done by governments at all levels to address this issue. 

2.72               While the Committee was pleased to hear about the work being done by Finance and ANAO to improve in this area, a lot of it is still conceptual, and the Committee wishes to see Finance take ownership of this issue with a sense of urgency. As such, the Committee would like to receive a progress report with detailed evidence of practical change.

 

Recommendation 3

That the Department of Finance and Deregulation, in consultation with the ANAO, report to the Joint Committee of Public Accounts and Audit in six months from the tabling of this report on: n  progress being made on guidance for agencies to improve the development and integration of effective Key Performance Indicators (KPIs) in program design, Portfolio Budget Statements, and annual reports; n  improvements to the KPI guidance aimed to enhance cross‑agency and cross-jurisdictional KPI development, use and reporting; and n  how the ANAO’s audit methodology is envisaged to fit within and support the overall KPI framework, and support ongoing policy enhancements.