Navigation: Previous Page | Contents | Next Page
Chapter 3
Australian Crime Commission performance measurement
3.1
This chapter considers the ACC's performance against its Key Performance
Indicators (KPIs). It examines the measurement tools utilised to inform the
KPIs and to track performance over time.
Key Performance Indicators
3.2
The Outcomes and Outputs Framework (the framework) provides the basis
for the government's approach to budgeting and reporting for public sector
agencies and the means by which the Parliament appropriates funds in the annual
budget context.[1]
In 2010, the Joint Committee on Public Accounts and Audit observed that
measuring key aspects of an agency's performance is a critical part of the
Government's Outcomes Framework.[2]
Within the context of the framework, KPIs are 'established to provide
information (either qualitative or quantitative) on the effectiveness of
programs in achieving objectives in support of respective outcomes'.[3]
3.3
The Department of Finance (Finance) has provided the following guidance
for agencies in developing KPIs:
Agencies should focus on reporting a strategic and meaningful
level of performance indicators, demonstrating the link between the program
performance indicators and the outcome.[4]
3.4
Agencies are required to provide a relevant, informative and useful
range of performance indicators that can be tracked over time. In advice to
entities on developing KPIs, Finance recommended that agencies use both
qualitative and quantitative information to measure program performance and
provided the following definitions:
Quantitative: This type of reporting is represented by
numbers or percentages in a table.
Qualitative: This type of reporting is represented by
narrative text. Agencies should identify aspirational goals or milestones that
are intended to be achieved by the program.[5]
3.5
Finance noted that KPIs must be designed to be 'capable of signalling to
government, Parliament and the community whether programs are delivering
intended results'.[6]
Further, consistent, clear reporting on performance provides an important
record of an agency's 'progress towards meeting government policy objectives,
how well public money is being spent and whether planned achievements are on
track'.[7]
3.6
A performance audit report from the Australian National Audit Office
(ANAO) titled Development and Implementation of Key Performance Indicators
to Support the Outcomes and Programs Framework emphasised the importance of
an 'appropriate mix of qualitative and quantitative KPIs including targets
against which progress towards program objectives could be assessed'.[8]
The ANAO noted that a tendency to rely on qualitative KPIs reduces the ability
of an agency to measure the results of program activities over time. Whereas:
A mix of effectiveness KPIs, that place greater emphasis on
quantitative KPIs and targets, would provide a more measurable basis for
performance assessment.[9]
3.7
The ANAO argued that because KPIs are statements of the pre‐defined and expected
impacts of a program, it is important that they are:
-
specific—so as to focus on those results that can be attributed
to the particular intervention/program;
-
measurable—include quantifiable units or targets that can be
readily compared over time;
-
achievable—realistic when compared with baseline performance and
the resources to be made available;
-
relevant—embody a direct link between the program’s objective and
the respective effectiveness KPI; and
-
timed—include specific timeframes for completion.[10]
ACC's outcome and program structure
3.8
The ACC's outcome and program structure are set out in its annual
report:
Outcome 1
Reduction in the threat and impact of serious and organised
crime, through analysis of and operations against national criminal activity,
for governments, law enforcement agencies and private sector organisations.
Outcome strategy
Collaborate with law enforcement and related government
agencies and private industry as part of the national effort against serious and
organised crime.[11]
3.9
The ACC's single outcome is underpinned by two programs:
-
Program 1.1.1—Strategic criminal intelligence services, the
performance of which is measured by two KPIs; and
-
Program 1.1.2—Investigations and intelligence operations into
federally relevant criminal activity, which are measured by six KPIs.[12]
Program 1.1.1
3.10
Program 1.1.1 has a set of deliverables to meet the overall aim that:
The ACC's criminal intelligence services are designed to
provide Commonwealth, state and territory law enforcement and relevant
government agencies with the criminal intelligence necessary to effectively and
efficiently disrupt serious and organised criminal activity and reduce the
vulnerabilities posed to the Australian community.[13]
3.11
The following table lists Program 1.1.1's KPIs, their targets and
results for 2012-13.
Program
1.1.1—ACC KPIs and performance 2012-13[14]
KPI
|
2012-13 target
|
2012-13 result
|
1.
Strategic intelligence products
align with ACC Board endorsed National Criminal Intelligence Priorities.
|
90%
|
100%
|
2. Partner agencies agree or strongly that the ACC's
intelligence enhances their understanding of serious and organised crime.
|
90%
|
90% of senior executives of partner agencies
agree/strongly agree
|
3.12
In relation to the first KPI, the annual report noted that a combined
1866 analytical and tactical intelligence products were produced during 2012-13.
Of these products, 100 per cent align with the National Criminal Intelligence
Priorities (NCIPs), including all 16 of the ACC strategic products in the Picture
of Criminality in Australia suite.[15]
3.13
The second KPI is measured by stakeholder feedback and the explanatory
note in relation to it states that:
Each partner agency has its own role and priorities, and each
has different needs for and uses of criminal intelligence. We continue to work
with our stakeholders to deepen our understanding of their needs and improve
the value of our intelligence from their perspective.[16]
Program 1.1.2
3.14
The overall aim of program 1.1.2 is as follows:
The ACC's investigations and intelligence operations underpin
its criminal intelligence services by providing unique intelligence collection
capabilities. ACC investigations are conducted in partnership with law
enforcement agencies with the objective of disrupting and deterring federally
relevant serious and organised criminal activity. In 2012-13, the ACC, under
the guidance of its Board, will further focus its coercive powers
determinations to more comprehensively address emerging issues in the organised
crime environment.[17]
Program 1.1.2—ACC KPIs and performance 2012-13[18]
KPIs
|
2012-13 result
|
1.
Partner agencies agree, or
strongly agree, that the ACC's contribution enhances efforts to combat
serious and organised crime.
|
100% of managers and senior executives agree
(target was 80%)
|
2.
Partner agencies agree, or
strongly agree, that the ACC's coercive powers are effective.
|
80%
(target was 90%)
|
3.
Targeted ACC investigations and
operations are aligned with ACC Board priorities and approved by the ACC
Board.
|
100% of senior executives of partner agencies
agree/strongly agree
|
4.
The ACC's operational
intelligence and contribution to joint intelligence investigations and
operations enhance the efficiency and/or effectiveness of law enforcement
efforts to disrupt and deter serious and organised crime, as measured by
stakeholder feedback.
|
100% of senior executives of partner agencies
agree/strongly agree
|
5.
The activities of targeted
criminal entities are disrupted as a result of ACC intelligence, investigations
and operations, and activity is undertaken to confiscate proceeds of crime.
|
26 disruptions
185 people charged
332 charges laid
38 people convicted
$83.87 million proceeds of
crime restrained
$26.02 million proceeds of
crime forfeited
$159.98 million tax
assessments issued
34 firearms seized
$16.34 million in cash seized
$589.52 million estimated
street value of drugs seized
$84.74 million value of the
illicit drug production potential of precursors seized.[19]
|
6.
Availability of the Australian Criminal
Intelligence Database and the Australian Law Enforcement Intelligence
Network.
|
>99%
(target was 98%)
|
3.15
The availability of ACID and ALEIN achieved greater than 99 per cent in
2012-13 and 2011-12. In 2012-13, there were 1 302 439 searches on ACID compared
to 331 664 in 2011-12, a fourfold increase.[20]
The ACC noted that this was 'due primarily to increased multiple-entity
searches by the ACC-led National Criminal Intelligence Fusion Capability'.[21]
3.16
The committee notes that the government has responded to the
recommendation made in its last report[22]
that the ACC review and re-examine its KPI concerning the ACID and the ALIEN. In
response, the government stated that it accepts the recommendation and that the
ACC is currently undertaking a scoping study to determine a replacement system
for ACID and to improve ALEIN. In addition, the ACC has provided a new KPI for
ACID and ALEIN. The performance of ACID and ALEIN will now be measured against
'provision of a national criminal intelligence database and analytical tools,
which facilitate the sharing and analysis of criminal intelligence across
jurisdictions.'[23]
3.17
The committee will carefully examine the ACC's 2013-14 Annual Report
with a view to determining the extent to which the change in KPI has resulted
in improved performance information.
Challenges in measuring ACC performance
3.18
The annual report highlights some of the complexities in measuring ACC
results. It notes that most of the work undertaken by the ACC is classified, or
has long-term results often being realised years after the ACC's initial
involvement, and once court decisions, law reforms and changes in
community behaviour have concluded or been implemented.[24]
3.19
During its examination, the committee asked the ACC to comment on the
difficulties surrounding the measurement of the performance of the ACC, as much
of the work undertaken by the ACC is utilised by other agencies. Mr Jevtovic
stated that the ACC is eager to demonstrate the worth of investment in the organisation
for crime fighting, but that it is difficult to monitor how much of the
information is used:
...we might produce 2500 information reports, which we
disseminate to Australian law enforcement over the country. Once we disseminate
that information, there is no technical solution to monitoring which component
of our 2500 products has actually been accessed or used if the people using it
do not tell us...that is the reality and we accept that.[25]
3.20
However, the level of reporting by the ACC was improved through changes
made to the ACC's Performance Measurement framework, as recommended by the
committee in its last report.[26]
This involved adapting the framework to:
-
more accurately reflect both quantitative and qualitative
results—capturing short-term and long-term, direct and indirect impacts of
activities; and
-
measure the performance against the new Strategic Plan 2013–18,
which highlights the evolution of serious and organised crime and the need to continually
adapt and develop the right capabilities to respond.[27]
3.21
The annual report notes that these changes have had a positive outcome
on performance:
Our revised indicators enable us to better self-assess our
performance, to complement feedback from our stakeholders which we intend to
gather more regularly throughout each reporting year. This will enable us to
report in new and different ways on the value we add to the national fight
against serious and organised crime.[28]
3.22
The revised performance management framework will require the ACC to
design new systems for collecting performance data, which will be implemented
over the course of the five year Strategic Plan. The committee will examine the
ACC's 2013-14 Annual Report with a view to determining the efficiency of the
new systems for collecting and reporting on performance data over the course of
the Strategic Plan.
Stakeholder surveys
3.23
The annual report notes that stakeholder relationships are critically
important to the ACC's success in combating serious and organised crime. The
ACC uses stakeholder surveys to help inform ACC reporting obligations. In 2012-13,
the ACC revised its stakeholder research methodology. Rather than engaging
market researchers to gather data through stakeholder survey forms, the ACC
conducted its research in-house with a wider range of qualitative interviews. According
to the ACC, the result provided 'more in-depth information for analysis, [and] a
more focused indication of how [the ACC] performed', while reducing costs.[29]
Committee view
3.24
The committee appreciates the complexities involved for the ACC in
developing meaningful qualitative KPIs that can be measured over time. The
committee also acknowledges that the nature of some of the ACC's work may not
be directly quantifiable.
3.25
In its previous report the committee recommended that the ACC Annual
Report 2012-13 should include information on progress made towards establishing
a balance of quantitative and qualitative KPIs, which can be measured over time.[30]
The committee notes that the government has accepted this recommendation,[31]
and that the ACC has amended its key performance indicators for 2013-14 for
both its Portfolio Budget Statement and its annual report.[32]
3.26
The new KPIs for outcome 1 include:
-
the ACC produces useful intelligence that identifies and provides
insights on new and emerging serious and organised crime threats.
-
the ACC fills intelligence gaps through the identification of
vulnerabilities and indicators of serious and organised crime.
-
the ACC collects and maintains national intelligence holdings of
serious and organised crime threats and targets.
-
the ACC interprets and analyses national holdings to create a
national serious and organised crime intelligence picture.
-
the ACC informs and influences hardening of the environment against
serious and organised crime;
-
the ACC influences or enables the disruption, disabling or
dismantling of serious and organised crime; and
-
the ACC participates in or coordinates collaboration in joint
investigations and operations to prevent and disrupt serious and organised
crime.[33]
3.27
The committee notes these new KPIs and will closely examine the relevant
information included in the ACC's 2013‑14 annual report, to ensure that it
provides a more measurable basis for performance assessment.
Navigation: Previous Page | Contents | Next Page
|