Navigation: Previous Page | Contents | Next Page
CHAPTER 3
Australian Crime Commission performance measurement
3.1
This chapter considers the ACC's Key Performance Indicators (KPIs) and
the extent to which they serve as a meaningful benchmark of the ACC's
performance. It examines the measurement tools utilised to inform the KPIs and
to track performance over time.
Key Performance Indicators
3.2
The Outcomes and Outputs Framework (the framework) provides the basis
for the government's approach to budgeting and reporting for public sector
agencies and the means by which the Parliament appropriates funds in the annual
budget context.[1]
In 2010, the Joint Committee on Public Accounts and Audit observed that
measuring key aspects of an agency's performance is a critical part of the
Government's Outcomes Framework.[2]
Within the context of the framework, KPIs are 'established to provide
information (either qualitative or quantitative) on the effectiveness of
programs in achieving objectives in support of respective outcomes'.[3]
3.3
The Department of Finance and Deregulation (DoFD) provided the following
guidance for agencies in preparing the 2011–12 Portfolio Budget Statements
(PBS):
Agencies should focus on reporting a strategic and meaningful
level of performance indicators, demonstrating the link between the program
performance indicators and the outcome.[4]
3.4
Agencies are required to provide a relevant, informative and useful
range of performance indicators that can be tracked over time. In advice to
entities on developing KPIs, DoFD recommended that agencies use both
qualitative and quantitative information to measure program performance in
their PBS and provided the following definitions:
Qualitative: this type of reporting is represented by
narrative text. Agencies should identify aspirational goals or milestones that
are intended to be achieved by the program.
Quantitative: this type of reporting is represented by
numbers or percentages in a table.[5]
3.5
DoFD noted that KPIs must be designed to be 'capable of signalling to
government, Parliament and the community whether programs are delivering
intended results'.[6]
Further, consistent, clear reporting on performance provides an important
record of an agency's 'progress towards meeting government policy objectives,
how well public money is being spent and whether planned achievements are on
track'.[7]
3.6
A performance audit report from the Australian National Audit Office (ANAO)
titled Development and Implementation of Key Performance Indicators to
Support the Outcomes and Programs Framework emphasised the importance of an
'appropriate mix of qualitative and quantitative KPIs including targets against
which progress towards program objectives could be assessed'.[8]
The ANAO noted that a tendency to rely on qualitative KPIs reduces the ability
of an agency to measure the results of program activities over time. Whereas:
A mix of effectiveness KPIs, that place greater emphasis on
quantitative KPIs and targets, would provide a more measurable basis for performance
assessment.[9]
3.7
The ANAO argued that because KPIs are statements of the pre‐defined and expected
impacts of a program, it is important that they are:
- specific—so as to focus on those results that can be attributed
to the particular intervention/program;
- measurable—include quantifiable units or targets that can be
readily compared over time;
- achievable—realistic when compared with baseline performance and
the resources to be made available;
- relevant—embody a direct link between the program’s objective and
the respective effectiveness KPI; and
- timed—include specific timeframes for completion.[10]
ACC outcome and program structure
3.8
The ACC's outcome and program structure are detailed in the PBS.
Outcome 1
Reduction in the threat and impact of serious and organised
crime, through analysis of and operations against national criminal activity,
for governments, law enforcement agencies and private sector organisations.
Outcome strategy
The ACC will contribute to a nationally coordinated effort to
reduce the threat and impact of serious and organised crime by collaborating
with Commonwealth, state and territory law enforcement and related government
agencies and private sector organisations. A highly developed understanding of
the threats posed by serious and organised crime will underpin the ACC’s
provision of specialised criminal intelligence services, and will focus
operations on targets that pose the highest risk to Australian society.[11]
3.9
The ACC's single outcome is underpinned by two programs:
- Program 1.1.1—Strategic intelligence services, the performance of
which is measured by two KPIs; and
- Program 1.1.2—Investigations and intelligence operations into
federally relevant criminal activity, which are measured by five KPIs.
3.10
Of a total of seven ACC KPIs, three are quantitative measures of
performance. The remainder are measured by way of stakeholder feedback.
3.11
Program 1.1.1 has a set of deliverables to meet the overall aim that:
The ACC's intelligence services are designed to provide
Commonwealth, state and territory law enforcement and relevant government
agencies with the criminal intelligence necessary to effectively and
efficiently disrupt serious and organised criminal activity and reduce the
vulnerabilities posed to the Australian community.[12]
3.12
The following table lists the program's KPIs, their targets and results
for 2011–12.
Program 1.1.1—ACC KPIs and
performance 2011–12 [13]
KPI
|
2011–12 target
|
2011–12 result
|
1.
ACC strategic intelligence is aligned
with ACC Board endorsed National Criminal Intelligence Priorities (NCIPs)
|
90%
|
100%
|
2. The understanding of serious and organised crime by
partner agencies is enhanced by ACC intelligence services, as measured by
stakeholder feedback.[14]
|
90%
|
70% of senior executives of partner agencies
agree/strongly agree
|
3.13
In relation to the first KPI, the annual report noted that there were 38
strategic intelligence products produced during the review period which all aligned
with the National Criminal Intelligence Priorities (NCIPs) including all ACC
strategic products in the Picture of Criminality in Australia suite. In
addition, a combined 1884 analytical and tactical intelligence products were
produced during 2011–12. Of these, 93 per cent, or 1753, align to the NCIPs.[15]
3.14
The second KPI is measured by stakeholder feedback and the explanatory
note in relation to it states that:
We are measuring feedback from a diverse range of state and
territory law enforcement agencies, as well as Commonwealth law enforcement,
regulatory, national security and policy agencies. Each stakeholder has its own
role and priorities, and each has different needs for and uses of criminal
intelligence. [16]
3.15
The overall aim of program 1.1.2 is as follows:
The ACC's investigations and intelligence operations underpin
its criminal intelligence services by providing unique intelligence collection
capabilities. ACC investigations are conducted in partnership with law
enforcement agencies with the objective of disrupting and deterring federally
relevant serious and organised criminal activity. In 2011-12, the ACC, under
the guidance of its Board, will further focus its coercive powers
determinations to more comprehensively address emerging issues in the organised
crime environment.[17]
Program 1.1.2—ACC KPIs and
performance 2011–12[18]
KPIs
|
2011–12 result
|
1.
Targeted ACC investigations and
operations are aligned with ACC Board priorities as approved by the ACC Board.
|
89% of senior executives of partner agencies agree/strongly agree
(target was 80%)
|
2.
The ACC's operational
intelligence and contribution to joint intelligence investigations and
operations enhance the efficiency and/or effectiveness of law enforcement
efforts to disrupt and deter serious and organised crime, as measured by
stakeholder feedback.
|
89% of senior executives of partner agencies agree/strongly agree
(target was 80%)
In addition:
- 42% of partner agencies achieved an operational
success in last 12 months as a result of intelligence/ information received;
- 56% identified a new criminal target; and
- 50% identified a new law enforcement operational
opportunity.
|
3.
The activities of targeted
criminal entities are disrupted as a result of ACC intelligence,
investigations and operations, and activity is undertaken to confiscate
proceeds of crime
|
26 disruptions
97 people charged
319 charges laid
45 convictions
$103.59 m proceeds of crime restrained
$31.63 m proceeds of crime forfeited
$4.42 m pecuniary penalty orders recovered
$49.68 m tax assessments issued
16 firearms seized
$5.47 m in cash seized
$67.71 m estimated street value drugs seized
$7.5 m value of the illicit drug production potential of precursors
seized. [19]
|
4.
The ACC's coercive powers are
effective in resolving partner agencies' intelligence gaps or investigative
needs that pertain to serious and organised crime, as measured by stakeholder
feedback
|
77% of senior executives of partner agencies agree/strongly agree
(target was 90%)
|
5.
A national criminal intelligence
database and analytical tools are available, to facilitate the sharing and
analysis of criminal intelligence across jurisdictions. The annual report
translates this KPI into: Availability of the Australian Criminal Intelligence
Database (ACID) and Australian Law Enforcement Intelligence Network (ALEIN)
|
Greater than 99% (target was 98%)
|
3.16
Three of the five KPIs under program 1.1.2 are measured by a stakeholder
feedback through a survey. The fifth KPI under this program measures the
'availability' of the Australian Criminal Intelligence Database (ACID) and the
Australian Law Enforcement Intelligence Network (ALEIN). While the availability
of ACID and ALEIN achieved greater than 99 per cent in 2011–12 and 2010–11, its
use is declining.[20]
In 2011–12, there were 331 664 searchers on ACID compared to 559 469 in 2010–11.
The ACC noted that there has been a decline in ACID's use for the past three
years and expects this trend to 'continue until reinvestment in ACID occurs and
the utility of the system is enhanced'.[21]
3.17
The committee questioned whether the availability of these resources is
an appropriate KPI to measure the value of ACID and ALEIN in light of the
downwards trend in their use. For this reason, the committee recommends that
the ACC review and re-examine this KPI.
Recommendation 1
3.18
The committee recommends that the Australian Crime Commission review and
re-examine and review its Key Performance Indicator concerning the Australian
Criminal Intelligence Database and the Australian Law Enforcement Intelligence
Network.
Challenges in measuring ACC performance
3.19
The ACC annual report highlighted some of the complexities in measuring
ACC results. It noted that statistics 'only illustrate part of the true value
that we bring to reducing harm to Australia from organised crime' including the
number of arrests, seizures, forfeitures and charges laid. These comprise the
traditional methods of measuring law enforcement outcomes. The ACC noted that
quantitative performance measures had been the mainstay of ACC performance
reporting but that they alone did not adequately convey the fullest picture of
the value that the ACC brings to national efforts to coordinate and combat serious
and organised crime.[22]
3.20
Furthermore, over a three year period, the ACC has shifted its operating
model to focus on three key areas which 'reach beyond traditional measures'.
These include:
- developing the criminal intelligence holdings of the nation;
-
targeting ACC resources at the highest risk targets; and
- collaborating with a wider range of partners to break the
business model of organised crime.[23]
3.21
ACC operational capabilities are directed to enabling partners to take
action and to informing policy and legislative responses that 'harden the
environment against organised crime by closing vulnerabilities, diminishing
criminal profitability or wholly removing incentives for criminal exploitation'.
Such an approach makes for 'less conventionally and less easily measured
results'.[24]
However, Mr Lawler explained to the committee that work was underway to change
the KPIs given that:
As we evolve it is becoming less relevant and less important
in the context of the ACC's work, and more difficult, to try to map arrests,
seizures and charges.[25]
3.22
The ACC has introduced a three-point grading scale applicable to direct
and indirect results as part of its performance measurement framework. However,
it is not clear how this grading scale has been applied and reflected in the
KPIs. When this question was put to the agency, the ACC explained that it
grades the level of contribution and effort expended by the ACC towards an
operational result. It is currently applied to a 'limited set of quantitative
measures, agency highlights, disruptions, seizure records and prosecutions' but
is under examination as part of a wider review into the ACC's Performance
Measurement Framework.[26]
Furthermore, recognising the need for 'a better balance between quantitative
and qualitative KPIs', the ACC noted that the review of its Performance
Measurement Framework is expected to lead to further modifications to improve
value reporting by the ACC.[27]
3.23
Notwithstanding the ACC's comments regarding the difficulties in
providing quantitative KPIs and the less tangible nature of its programs which
makes the development of appropriate KPIs more challenging, the committee
pursued this matter during its examination. When the committee questioned the
ACC about its reliance upon stakeholder feedback as a KPI measure, the
committee was informed by ACC Executive Director, Mrs Karen Harfield that as
the ACC's modus operandi was to work in partnership, the KPIs are directed at
ensuring that the agency is 'extracting' information from its partners:
Because all the work we do is either with or for a partner,
in an attempt not to simply count activities, which you can convert into
quantifiable information, there was a sense of trying to achieve results against
how we have improved or supported partners. That was the sort of emphasis when
thinking through those KPIs, as well as how we could use stakeholder
information to give us some feedback about how the products we produce and the
work that we do impact on their environment.[28]
3.24
Mrs Harfield also underscored the importance for the ACC to understand
and 'settle' its operating model from which it could build a performance
framework:
The issue is trying to blend the balance of the quantitative,
so we could have KPIs that literally went to the issue of the numbers of things
that we produced in terms of intelligence, but of course that could have had a
perverse outcome that people produce smaller, less insightful, not as useful
pieces of intelligence but from a KPI perspective that looked like a good
performance. We were conscious that we did not want to move into that sort of
territory without being very clear about what it was that, from our partners'
perspective as well as from an ACC perspective, we actually got the development
of that performance framework right. That is work that has been carried on at
the same time as still working towards KPIs that we have had for some time. The
KPIs are not the only performance information that we collect or share with the
board or with other partners and, clearly throughout the annual report, you see
a blend of qualitative descriptors about cases as well as quantitative sorts of
measures.[29]
Stakeholder surveys
3.25
The annual report noted that stakeholder surveys inform the ACC's
reporting obligations against its PBS. Mr Lawler informed the committee that
the ACC's KPIs 'relate principally' to the survey and have done so since 2009–10.[30]
However, the committee raised a number of questions in relation to the survey
including its findings, the respondent pool and the agency's overall reliance
on a survey as a KPI measurement.
3.26
In 2011–12, in an effort to reduce costs and imposition on stakeholders,
the ACC condensed and targeted the survey sample to focus on senior executive
representatives from Commonwealth, state and territory law enforcement partners
with whom the ACC works 'routinely'.[31]
For these reasons, 66 senior executive representatives from 25 Commonwealth
agencies and state and territory law enforcement partners were surveyed. Conducted
in May to June 2012, the research comprised an online survey and telephone
interviews with ACC Board members. In contrast, previous surveys have involved
a wider range of respondents. For example, the 2010–11 survey involved an
'online survey, which captured feedback from 268 respondents across 57
agencies, in-depth interviews with senior executives and focus groups with four
partner agencies'.[32]
3.27
When asked about the decision to narrow the survey sample to senior
executives, Mr Lawler explained the historical development of the survey:
When we started off, particularly around 2009–10, part of the
issue was: who do you survey? For example, if we surveyed the whole partner
agency I would say that probably 95 per cent of them would have no reference to
the ACC at all because their work does not relate to organised crime. It is a
case of who you try to survey and whether the response they give can be in any
way meaningful. These are not just the board members; these are senior
executive people whom we hoped would understand and have worked with the
commission and be understanding of how the commission may have contributed or
worked with them.[33]
3.28
Notwithstanding the argument that surveying senior executives is a more
cost-effective and efficient approach, the use of the survey to inform most of
the KPIs raises a number of concerns. As previously noted, agencies are advised
to provide a balance of qualitative and quantitative KPIs while the majority of
the ACC's KPIs relate 'principally' to the survey.[34]
The committee questioned the ACC about the use of a survey as the primary
measure under the KPIs and the extent to which it gauges ACC progress and the
quality of its work. In response, Mrs Harfield informed the committee that:
The stakeholder part of the current system is an important
feedback mechanism as well as then converting that into a sort of ratio number
which does provide people with some sort of insight, but there is a lot of
detail that sits then underneath that in terms of what we get back from a
stakeholder survey. So, the bold numbers probably do not produce for you as
rich a sort of picture about what it tells us.[35]
3.29
In relation to the actual survey findings, the 2011–12 survey recorded
'moderate to strong ratings' against the PBS indicators in comparison to the previous
year. However the 2011–12 ratings were lower when compared to the senior
executive subset of 2011.[36]
The annual report does not detail why. As such an outcome is inconsistent with ACC
observations that there were typically higher rates of satisfaction within
senior management of partner agencies, the committee pursued this matter during
its examination.[37]
When questioned about the results, Mr Lawler argued that the feedback from
partners and ACC Board members was 'contradictory' to the extent that
discussions outside of the survey process revealed that 'board members are very
happy with how the agency is performing'.[38]
3.30
Mr Lawler informed the committee that those surveyed and ACC Board members
were 'quite dissatisfied with this survey process and felt that the people
conducting the survey did not have a sufficient understanding of the commission'.
He further explained that they felt that some of the questions were not
relevant to their interaction with the ACC:
For example, they were being asked about tactical
intelligence products when some of these senior people being surveyed would not
see such material. They were being asked questions that were difficult for them
to comment on, which is being reflected in some of the survey material.[39]
3.31
While recognising that there was 'strong support for the commission
across the survey results', Mr Lawler noted that 'unless that survey is
precisely targeted we could ask people about the commission who would have
little input to give, and that can also reflect on the agency's performance and
what stakeholders think of the agency.'[40]
Furthermore, Mr Lawler explained that as part of an attempt to 'get more
meaning into the actual activity' of the survey, the ACC Board has since agreed
that the survey will be conducted by the ACC executive rather than an outside
contractor or an independent person. In the future, the senior executive will
approach stakeholders and the ACC Board and by taking a 'bottom-up approach,
with feedback from all the various areas that we might operate to the
commissioner or to the secretary as to how the commission is functioning'.[41]
Furthermore, the ACC intends to provide each agency partner with a 'snapshot'
of its interactions with the ACC. Mr Lawler explained the rationale:
There is a multitude of different facets as to how the
commission interacts with its partners. That has not been captured
properly—probably it is the commission's fault—so that the stakeholder can be
properly informed as to what the true interaction looks like.[42]
Committee view
3.32
The committee appreciates the complexities involved for the ACC in
developing meaningful qualitative KPIs that can be measured over time and
acknowledges that the nature of some of the ACC's work may not be directly
quantifiable.
3.33
Notwithstanding this point, the ACC has noted its work towards more
effective and meaningful measurable indicators over some time. In April 2011,
in recognition of the need 'review and update our strategic plan and the key
performance indicators that we use to measure our performance', the ACC
commissioned Project Sisco.[43]
The 2011–12 annual report notes that Project Sisco was completed during the
review period and provided to the ACC on 1 August 2011. While the annual report
notes that implementation of the project's recommendations is 'progressing', it
does not detail the extent to which the ACC's KPIs are the focus of the reform
agenda.[44]
3.34
The committee's concerns regarding the ACC's reliance on the stakeholder
survey and the need for a greater balance of measurable indicators is made more
pressing given the fact that the key stakeholders surveyed comprise the ACC
Board which is responsible for 'providing strategic guidance' and authorising ACC
work priorities'.[45]
While the survey results for 2011–12 can be explained at least in part by the
manner in which the survey process was managed, it is not clear to the
committee what can be deduced from the survey results. Notwithstanding the
survey management difficulties faced during the year in review, the committee
supports the use of an independent and impartial body to conduct the process. Placing
the management of the survey into the hands of the ACC executive to survey what
is small subset of senior executive colleagues in partner agencies will not strengthen
accountability in relation to the process nor public confidence in the results.
3.35
The committee appreciates the complexities for the ACC in developing
meaningful measurable indicators of its performance. It also recognises that
there are other indicators of ACC performance including the deliverables which
are well constructed and informative. However, it also notes that the KPIs are
limited in measuring progress towards program objectives.
3.36
The committee holds the view that the usefulness of the ACC's
qualitative indicators as a basis for performance assessment would be enhanced
if the agency formulated more quantitative indicators and targets against which
to examine performance results. Furthermore, while the committee recognises a
stakeholder survey as an important means of gaging the ACC's performance given
its focus on working in partnership with other Commonwealth, state and territory
stakeholders, it should be used as one of a number of tools to measure
performance. The committee recommends, therefore that the ACC move towards
establishing a balance between qualitative and quantitative KPIs.
Recommendation 2
3.37
The committee recommends that the Australian Crime Commission work
towards establishing a balance of qualitative and quantitative Key Performance
Indicators (KPIs) which can be measured over time.
3.38
The committee recommends that the 2012–13 Annual Report of the
Australian Crime Commission provide information on progress made towards
establishing a balance of qualitative and quantitative KPIs.
Navigation: Previous Page | Contents | Next Page
Top
|