Navigation: Previous Page | Contents | Next Page

CHAPTER 3

Australian Crime Commission performance measurement

3.1        This chapter considers the ACC's Key Performance Indicators (KPIs) and the extent to which they serve as a meaningful benchmark of the ACC's performance. It examines the measurement tools utilised to inform the KPIs and to track performance over time.

Key Performance Indicators

3.2        The Outcomes and Outputs Framework (the framework) provides the basis for the government's approach to budgeting and reporting for public sector agencies and the means by which the Parliament appropriates funds in the annual budget context.[1] In 2010, the Joint Committee on Public Accounts and Audit observed that measuring key aspects of an agency's performance is a critical part of the Government's Outcomes Framework.[2] Within the context of the framework, KPIs are 'established to provide information (either qualitative or quantitative) on the effectiveness of programs in achieving objectives in support of respective outcomes'.[3]

3.3        The Department of Finance and Deregulation (DoFD) provided the following guidance for agencies in preparing the 2011–12 Portfolio Budget Statements (PBS):

Agencies should focus on reporting a strategic and meaningful level of performance indicators, demonstrating the link between the program performance indicators and the outcome.[4]

3.4        Agencies are required to provide a relevant, informative and useful range of performance indicators that can be tracked over time. In advice to entities on developing KPIs, DoFD recommended that agencies use both qualitative and quantitative information to measure program performance in their PBS and provided the following definitions:

Qualitative: this type of reporting is represented by narrative text. Agencies should identify aspirational goals or milestones that are intended to be achieved by the program.

Quantitative: this type of reporting is represented by numbers or percentages in a table.[5]

3.5        DoFD noted that KPIs must be designed to be 'capable of signalling to government, Parliament and the community whether programs are delivering intended results'.[6] Further, consistent, clear reporting on performance provides an important record of an agency's 'progress towards meeting government policy objectives, how well public money is being spent and whether planned achievements are on track'.[7]

3.6        A performance audit report from the Australian National Audit Office (ANAO) titled Development and Implementation of Key Performance Indicators to Support the Outcomes and Programs Framework emphasised the importance of an 'appropriate mix of qualitative and quantitative KPIs including targets against which progress towards program objectives could be assessed'.[8] The ANAO noted that a tendency to rely on qualitative KPIs reduces the ability of an agency to measure the results of program activities over time. Whereas:

A mix of effectiveness KPIs, that place greater emphasis on quantitative KPIs and targets, would provide a more measurable basis for performance assessment.[9]

3.7        The ANAO argued that because KPIs are statements of the pre‐defined and expected impacts of a program, it is important that they are:

• specific—so as to focus on those results that can be attributed to the particular intervention/program;
• measurable—include quantifiable units or targets that can be readily compared over time;
• achievable—realistic when compared with baseline performance and the resources to be made available;
• relevant—embody a direct link between the program’s objective and the respective effectiveness KPI; and
• timed—include specific timeframes for completion.[10]

ACC outcome and program structure

3.8        The ACC's outcome and program structure are detailed in the PBS.

Outcome 1

Reduction in the threat and impact of serious and organised crime, through analysis of and operations against national criminal activity, for governments, law enforcement agencies and private sector organisations.

Outcome strategy

The ACC will contribute to a nationally coordinated effort to reduce the threat and impact of serious and organised crime by collaborating with Commonwealth, state and territory law enforcement and related government agencies and private sector organisations. A highly developed understanding of the threats posed by serious and organised crime will underpin the ACC’s provision of specialised criminal intelligence services, and will focus operations on targets that pose the highest risk to Australian society.[11]

3.9        The ACC's single outcome is underpinned by two programs:

• Program 1.1.1—Strategic intelligence services, the performance of which is measured by two KPIs; and
• Program 1.1.2—Investigations and intelligence operations into federally relevant criminal activity, which are measured by five KPIs.

3.10      Of a total of seven ACC KPIs, three are quantitative measures of performance. The remainder are measured by way of stakeholder feedback.

3.11      Program 1.1.1 has a set of deliverables to meet the overall aim that:

The ACC's intelligence services are designed to provide Commonwealth, state and territory law enforcement and relevant government agencies with the criminal intelligence necessary to effectively and efficiently disrupt serious and organised criminal activity and reduce the vulnerabilities posed to the Australian community.[12]

3.12      The following table lists the program's KPIs, their targets and results for 2011–12.

Program 1.1.1—ACC KPIs and performance 2011–12 [13]

KPI	2011–12 target	2011–12 result
1.    ACC strategic intelligence is aligned with ACC Board endorsed National Criminal Intelligence Priorities (NCIPs)	90%	100%
2.    The understanding of serious and organised crime by partner agencies is enhanced by ACC intelligence services, as measured by stakeholder feedback.[14]	90%	70% of senior executives of partner agencies agree/strongly agree

 

3.13      In relation to the first KPI, the annual report noted that there were 38 strategic intelligence products produced during the review period which all aligned with the National Criminal Intelligence Priorities (NCIPs) including all ACC strategic products in the Picture of Criminality in Australia suite. In addition, a combined 1884 analytical and tactical intelligence products were produced during 2011–12. Of these, 93 per cent, or 1753, align to the NCIPs.[15]

3.14      The second KPI is measured by stakeholder feedback and the explanatory note in relation to it states that:

We are measuring feedback from a diverse range of state and territory law enforcement agencies, as well as Commonwealth law enforcement, regulatory, national security and policy agencies. Each stakeholder has its own role and priorities, and each has different needs for and uses of criminal intelligence. [16]

3.15      The overall aim of program 1.1.2 is as follows:

The ACC's investigations and intelligence operations underpin its criminal intelligence services by providing unique intelligence collection capabilities. ACC investigations are conducted in partnership with law enforcement agencies with the objective of disrupting and deterring federally relevant serious and organised criminal activity. In 2011-12, the ACC, under the guidance of its Board, will further focus its coercive powers determinations to more comprehensively address emerging issues in the organised crime environment.[17]

Program 1.1.2—ACC KPIs and performance 2011–12[18]

KPIs	2011–12 result
1.    Targeted ACC investigations and operations are aligned with ACC Board priorities as approved by the ACC Board.	89% of senior executives of partner agencies agree/strongly agree (target was 80%)
2.    The ACC's operational intelligence and contribution to joint intelligence investigations and operations enhance the efficiency and/or effectiveness of law enforcement efforts to disrupt and deter serious and organised crime, as measured by stakeholder feedback.	89% of senior executives of partner agencies agree/strongly agree (target was 80%) In addition:   42% of partner agencies achieved an operational success in last 12 months as a result of intelligence/ information received;   56% identified a new criminal target; and   50% identified a new law enforcement operational opportunity.
3.    The activities of targeted criminal entities are disrupted as a result of ACC intelligence, investigations and operations, and activity is undertaken to confiscate proceeds of crime	26 disruptions 97 people charged 319 charges laid 45 convictions $103.59 m proceeds of crime restrained $31.63 m proceeds of crime forfeited $4.42 m pecuniary penalty orders recovered $49.68 m tax assessments issued 16 firearms seized $5.47 m in cash seized $67.71 m estimated street value drugs seized $7.5 m value of the illicit drug production potential of precursors seized. [19]
4.    The ACC's coercive powers are effective in resolving partner agencies' intelligence gaps or investigative needs that pertain to serious and organised crime, as measured by stakeholder feedback	77% of senior executives of partner agencies agree/strongly agree (target was 90%)
5.    A national criminal intelligence database and analytical tools are available, to facilitate the sharing and analysis of criminal intelligence across jurisdictions. The annual report translates this KPI into: Availability of the Australian Criminal Intelligence Database (ACID) and Australian Law Enforcement Intelligence Network (ALEIN)	Greater than 99% (target was 98%)

 

3.16      Three of the five KPIs under program 1.1.2 are measured by a stakeholder feedback through a survey. The fifth KPI under this program measures the 'availability' of the Australian Criminal Intelligence Database (ACID) and the Australian Law Enforcement Intelligence Network (ALEIN). While the availability of ACID and ALEIN achieved greater than 99 per cent in 2011–12 and 2010–11, its use is declining.[20] In 2011–12, there were 331 664 searchers on ACID compared to 559 469 in 2010–11. The ACC noted that there has been a decline in ACID's use for the past three years and expects this trend to 'continue until reinvestment in ACID occurs and the utility of the system is enhanced'.[21]

3.17      The committee questioned whether the availability of these resources is an appropriate KPI to measure the value of ACID and ALEIN in light of the downwards trend in their use. For this reason, the committee recommends that the ACC review and re-examine this KPI.

Recommendation 1

3.18      The committee recommends that the Australian Crime Commission review and re-examine and review its Key Performance Indicator concerning the Australian Criminal Intelligence Database and the Australian Law Enforcement Intelligence Network.

Challenges in measuring ACC performance

3.19      The ACC annual report highlighted some of the complexities in measuring ACC results. It noted that statistics 'only illustrate part of the true value that we bring to reducing harm to Australia from organised crime' including the number of arrests, seizures, forfeitures and charges laid. These comprise the traditional methods of measuring law enforcement outcomes. The ACC noted that quantitative performance measures had been the mainstay of ACC performance reporting but that they alone did not adequately convey the fullest picture of the value that the ACC brings to national efforts to coordinate and combat serious and organised crime.[22]

3.20      Furthermore, over a three year period, the ACC has shifted its operating model to focus on three key areas which 'reach beyond traditional measures'. These include:

• developing the criminal intelligence holdings of the nation;
• targeting ACC resources at the highest risk targets; and
• collaborating with a wider range of partners to break the business model of organised crime.[23] 

3.21      ACC operational capabilities are directed to enabling partners to take action and to informing policy and legislative responses that 'harden the environment against organised crime by closing vulnerabilities, diminishing criminal profitability or wholly removing incentives for criminal exploitation'. Such an approach makes for 'less conventionally and less easily measured results'.[24] However, Mr Lawler explained to the committee that work was underway to change the KPIs given that:

As we evolve it is becoming less relevant and less important in the context of the ACC's work, and more difficult, to try to map arrests, seizures and charges.[25]

3.22      The ACC has introduced a three-point grading scale applicable to direct and indirect results as part of its performance measurement framework. However, it is not clear how this grading scale has been applied and reflected in the KPIs. When this question was put to the agency, the ACC explained that it grades the level of contribution and effort expended by the ACC towards an operational result. It is currently applied to a 'limited set of quantitative measures, agency highlights, disruptions, seizure records and prosecutions' but is under examination as part of a wider review into the ACC's Performance Measurement Framework.[26] Furthermore, recognising the need for 'a better balance between quantitative and qualitative KPIs', the ACC noted that the review of its Performance Measurement Framework is expected to lead to further modifications to improve value reporting by the ACC.[27]

3.23      Notwithstanding the ACC's comments regarding the difficulties in providing quantitative KPIs and the less tangible nature of its programs which makes the development of appropriate KPIs more challenging, the committee pursued this matter during its examination. When the committee questioned the ACC about its reliance upon stakeholder feedback as a KPI measure, the committee was informed by ACC Executive Director, Mrs Karen Harfield that as the ACC's modus operandi was to work in partnership, the KPIs are directed at ensuring that the agency is 'extracting' information from its partners:

Because all the work we do is either with or for a partner, in an attempt not to simply count activities, which you can convert into quantifiable information, there was a sense of trying to achieve results against how we have improved or supported partners. That was the sort of emphasis when thinking through those KPIs, as well as how we could use stakeholder information to give us some feedback about how the products we produce and the work that we do impact on their environment.[28]

3.24      Mrs Harfield also underscored the importance for the ACC to understand and 'settle' its operating model from which it could build a performance framework:

The issue is trying to blend the balance of the quantitative, so we could have KPIs that literally went to the issue of the numbers of things that we produced in terms of intelligence, but of course that could have had a perverse outcome that people produce smaller, less insightful, not as useful pieces of intelligence but from a KPI perspective that looked like a good performance. We were conscious that we did not want to move into that sort of territory without being very clear about what it was that, from our partners' perspective as well as from an ACC perspective, we actually got the development of that performance framework right. That is work that has been carried on at the same time as still working towards KPIs that we have had for some time. The KPIs are not the only performance information that we collect or share with the board or with other partners and, clearly throughout the annual report, you see a blend of qualitative descriptors about cases as well as quantitative sorts of measures.[29]

Stakeholder surveys

3.25      The annual report noted that stakeholder surveys inform the ACC's reporting obligations against its PBS. Mr Lawler informed the committee that the ACC's KPIs 'relate principally' to the survey and have done so since 2009–10.[30] However, the committee raised a number of questions in relation to the survey including its findings, the respondent pool and the agency's overall reliance on a survey as a KPI measurement.

3.26      In 2011–12, in an effort to reduce costs and imposition on stakeholders, the ACC condensed and targeted the survey sample to focus on senior executive representatives from Commonwealth, state and territory law enforcement partners with whom the ACC works 'routinely'.[31] For these reasons, 66 senior executive representatives from 25 Commonwealth agencies and state and territory law enforcement partners were surveyed.  Conducted in May to June 2012, the research comprised an online survey and telephone interviews with ACC Board members. In contrast, previous surveys have involved a wider range of respondents. For example, the 2010–11 survey involved an 'online survey, which captured feedback from 268 respondents across 57 agencies, in-depth interviews with senior executives and focus groups with four partner agencies'.[32]

3.27      When asked about the decision to narrow the survey sample to senior executives, Mr Lawler explained the historical development of the survey:

When we started off, particularly around 2009–10, part of the issue was: who do you survey? For example, if we surveyed the whole partner agency I would say that probably 95 per cent of them would have no reference to the ACC at all because their work does not relate to organised crime. It is a case of who you try to survey and whether the response they give can be in any way meaningful. These are not just the board members; these are senior executive people whom we hoped would understand and have worked with the commission and be understanding of how the commission may have contributed or worked with them.[33]

3.28      Notwithstanding the argument that surveying senior executives is a more cost-effective and efficient approach, the use of the survey to inform most of the KPIs raises a number of concerns. As previously noted, agencies are advised to provide a balance of qualitative and quantitative KPIs while the majority of the ACC's KPIs relate 'principally' to the survey.[34] The committee questioned the ACC about the use of a survey as the primary measure under the KPIs and the extent to which it gauges ACC progress and the quality of its work. In response, Mrs Harfield informed the committee that:

The stakeholder part of the current system is an important feedback mechanism as well as then converting that into a sort of ratio number which does provide people with some sort of insight, but there is a lot of detail that sits then underneath that in terms of what we get back from a stakeholder survey. So, the bold numbers probably do not produce for you as rich a sort of picture about what it tells us.[35]

3.29      In relation to the actual survey findings, the 2011–12 survey recorded 'moderate to strong ratings' against the PBS indicators in comparison to the previous year. However the 2011–12 ratings were lower when compared to the senior executive subset of 2011.[36] The annual report does not detail why. As such an outcome is inconsistent with ACC observations that there were typically higher rates of satisfaction within senior management of partner agencies, the committee pursued this matter during its examination.[37] When questioned about the results, Mr Lawler argued that the feedback from partners and ACC Board members was 'contradictory' to the extent that discussions outside of the survey process revealed that 'board members are very happy with how the agency is performing'.[38]

3.30      Mr Lawler informed the committee that those surveyed and ACC Board members were 'quite dissatisfied with this survey process and felt that the people conducting the survey did not have a sufficient understanding of the commission'. He further explained that they felt that some of the questions were not relevant to their interaction with the ACC:

For example, they were being asked about tactical intelligence products when some of these senior people being surveyed would not see such material. They were being asked questions that were difficult for them to comment on, which is being reflected in some of the survey material.[39]

3.31      While recognising that there was 'strong support for the commission across the survey results', Mr Lawler noted that 'unless that survey is precisely targeted we could ask people about the commission who would have little input to give, and that can also reflect on the agency's performance and what stakeholders think of the agency.'[40] Furthermore, Mr Lawler explained that as part of an attempt to 'get more meaning into the actual activity' of the survey, the ACC Board has since agreed that the survey will be conducted by the ACC executive rather than an outside contractor or an independent person. In the future, the senior executive will approach stakeholders and the ACC Board and by taking a 'bottom-up approach, with feedback from all the various areas that we might operate to the commissioner or to the secretary as to how the commission is functioning'.[41] Furthermore, the ACC intends to provide each agency partner with a 'snapshot' of its interactions with the ACC. Mr Lawler explained the rationale:

There is a multitude of different facets as to how the commission interacts with its partners. That has not been captured properly—probably it is the commission's fault—so that the stakeholder can be properly informed as to what the true interaction looks like.[42]

Committee view

3.32      The committee appreciates the complexities involved for the ACC in developing meaningful qualitative KPIs that can be measured over time and acknowledges that the nature of some of the ACC's work may not be directly quantifiable.

3.33      Notwithstanding this point, the ACC has noted its work towards more effective and meaningful measurable indicators over some time. In April 2011, in recognition of the need 'review and update our strategic plan and the key performance indicators that we use to measure our performance', the ACC commissioned Project Sisco.[43] The 2011–12 annual report notes that Project Sisco was completed during the review period and provided to the ACC on 1 August 2011. While the annual report notes that implementation of the project's recommendations is 'progressing', it does not detail the extent to which the ACC's KPIs are the focus of the reform agenda.[44]

3.34      The committee's concerns regarding the ACC's reliance on the stakeholder survey and the need for a greater balance of measurable indicators is made more pressing given the fact that the key stakeholders surveyed comprise the ACC Board  which is responsible for 'providing strategic guidance' and authorising ACC work priorities'.[45] While the survey results for 2011–12 can be explained at least in part by the manner in which the survey process was managed, it is not clear to the committee what can be deduced from the survey results. Notwithstanding the survey management difficulties faced during the year in review, the committee supports the use of an independent and impartial body to conduct the process. Placing the management of the survey into the hands of the ACC executive to survey what is small subset of senior executive colleagues in partner agencies will not strengthen accountability in relation to the process nor public confidence in the results.

3.35      The committee appreciates the complexities for the ACC in developing meaningful measurable indicators of its performance. It also recognises that there are other indicators of ACC performance including the deliverables which are well constructed and informative. However, it also notes that the KPIs are limited in measuring progress towards program objectives.

3.36      The committee holds the view that the usefulness of the ACC's qualitative indicators as a basis for performance assessment would be enhanced if the agency formulated more quantitative indicators and targets against which to examine performance results. Furthermore, while the committee recognises a stakeholder survey as an important means of gaging the ACC's performance given its focus on working in partnership with other Commonwealth, state and territory stakeholders, it should be used as one of a number of tools to measure performance. The committee recommends, therefore that the ACC move towards establishing a balance between qualitative and quantitative KPIs.  

Recommendation 2

3.37      The committee recommends that the Australian Crime Commission work towards establishing a balance of qualitative and quantitative Key Performance Indicators (KPIs) which can be measured over time.

3.38      The committee recommends that the 2012–13 Annual Report of the Australian Crime Commission provide information on progress made towards establishing a balance of qualitative and quantitative KPIs.

Navigation: Previous Page | Contents | Next Page