Chapter 4 Performance reporting for national funding agreements
Introduction
4.1
This chapter examines the performance reporting requirements for national
funding agreements. Firstly, it explores the necessity for cultural change
across all levels of government to adapt to the reporting requirements for an
outcomes focused framework. The chapter then looks at the factors that must be
considered in order to provide viable, relevant data. This leads to a
discussion about data quality and collection. Finally the chapter examines some
of the improvements that are underway to address the issues identified in the
chapter.
4.2
A strong performance reporting framework is essential to provide
transparency, accountability and scrutiny of the Intergovernmental Agreement on
Federal Financial Relations (IGA FFR). These areas will be covered in Chapter 5.
Accountability and performance reporting
4.3
The IGA FFR is outcomes-based, promising a ‘rigorous focus on the
achievement of outcomes–that is mutual agreement on what objectives, outcomes
and outputs improve the well-being of Australians’.[1]
4.4
While an outcomes approach focuses on the achievement of objectives and
provides better value for money and flexibility, witnesses warned that it also
demands increased accountability. The Commonwealth Auditor‑General
advised the Committee that performance measurement was central to
accountability, explaining that parliament cannot be confident national funding
agreements are achieving the required outcomes without successful reporting:
Without clear, adequate and consistent reporting against
meaningful performance measures, the Parliament is constrained in its ability
to understand and assess how Commonwealth funding is contributing to the
achievement of value-for-money outcomes in areas covered by national funding
agreements.[2]
4.5
The Council of Australian Governments Reform Council (CRC) told the
Committee that, to ensure accountability, a strong reporting framework was
necessary, including objectives, outcomes and performance indicators:
The [CRC] must be able to assess the jurisdictions’ progress
over time in the areas covered by the national agreements, and it therefore
must have access to adequate and reliable information and data to inform its
assessments.[3]
4.6
The Committee heard that there are several issues that need to be addressed
in order to improve accountability and the reporting framework for national
funding agreements:
n cultural change;
n setting reporting
objectives;
-> clearly
defined outcomes; and
-> key
performance indicators;
n data quality;
-> timeliness;
-> comparability;
and
-> generic
data collections; and
n meaningful
interpretation.
Cultural change
4.7
Witnesses warned that achieving the benefits of an outcomes-based
framework requires significant cultural change at all levels of government. The
CRC identified the need for extensive cultural change, reminding the Committee
that the reform of federal financial relations ‘challenge[s] conventional
practices’ and will necessitate ‘[g]enuine cooperation, a commitment to
outcomes, respect for roles and responsibilities, and real accountability’.[4]
4.8
The Committee asked whether or not government departments and agencies
had accepted the need for cultural change and made the necessary adjustments.
The Victorian Government maintained that the changes had been accepted and
embraced at ministerial and central agency level but admitted that there were
problems at line agency level:
The cultural challenge that we have is one where some people
in some of the relevant line agencies …are taking a while to absorb what is a
really marked conceptual shift. …The challenge that we have is persuading some
of those who for many years in line agencies and both levels of government have
been dealing with these very prescriptive SPPs to realise that the world has
changed fundamentally.[5]
4.9
The Committee asked what the barriers were to achieving this cultural
shift. Echoing the Victorian Government’s comments, the Queensland Government cautioned
that an inputs approach was entrenched in many government departments and this mind-set
would take some time to adjust:
It is simply a matter of getting people to change the way
they think. In this case what Commonwealth line agencies require from state
agencies under a particular agreement is an issue that can be resolved in time,
but it is very hard to change overnight people’s views and expectations about
what needs to be done under a particular agreement.[6]
4.10
The Queensland Auditor-General suggested that it is more difficult to
measure outcomes than inputs and this explained the reluctance of departments
and agencies to change:
Everyone is comfortable with controlling inputs because you
can measure them; you can see them. People are less comfortable with outcomes
because they are a bit more difficult to describe and to measure.[7]
4.11
The Productivity Commission (PC) elaborated on this concept, proposing
that agencies were not comfortable being held accountable for outcomes which
are less easy to control than inputs:
Outcomes can be affected by a lot of external contextual
factors, an agency can say, ’I don’t control the unemployment rate, and the
unemployment rate is actually a major factor in homelessness, so you can’t hold
me responsible for that high-level outcome.’[8]
4.12
The Committee asked what steps are being taken by the various parties to
promote cultural change. The Queensland Government admitted that it is still
working through solutions to address this issue but told the Committee that it
is implementing a range of methods to encourage and support the necessary
change:
We are looking at a few mechanisms, but mostly in the area of
what we can do in terms of training material and documentation. For instance,
we are looking at guidance, practitioner’s toolkits and that kind of material
at a fairly technical level.[9]
4.13
The Department of the Prime Minister and Cabinet (PM&C) confirmed
that cultural change was a long term goal that required ‘an ongoing educative
exercise’ and acknowledged that the federal departments responsible for implementing
the IGA FFR had ‘underestimated the amount of work that we would need to do to
change the culture’.[10]
Setting reporting objectives
4.14
To lay the foundation for reliable performance reporting clear reporting
objectives have to be established by precisely defining the outcomes required
for each individual national funding agreement. Once objectives are clear, key
performance indicators can be developed to measure and assess outcomes.
Clearly defined outcomes
4.15
The Commonwealth Auditor-General told the Committee that outcomes must
be clearly defined and that this required ‘clarity around the policy
objectives’ of national funding agreements.[11] This advice was
reiterated by state Auditors-General and academics.[12]
The Queensland Auditor‑General told the Committee:
…it is knowing what you want and what you want to use it for
which is important.[13]
4.16
The Queensland Auditor-General went on to provide the example of the
Building the Education Revolution (BER), telling the Committee that lack of
clarity around policy outcomes hampered performance reporting for this national
funding agreement:
I am still bemused as to whether the stimulus package [BER]
was about spending the money quick or actually achieving some outcomes by way
of school buildings. If I look at the agreement, there is not a clear sense as
to should the money have just been spent and spent quickly,…or should it have
been spent well to actually deliver some buildings and some capacity.[14]
4.17
While agreeing with the need to specify measureable outcomes, the
NSW Auditor-General cautioned that this can be a difficult task in itself.[15]
He explained that outcomes must neither be too broad or too specific:
…if the outcomes are too broad it becomes very difficult to
measure whether they have been effective. For example, if they say, ‘Here’s a
bucket of money – the outcome we want is better health,’ it is quite difficult
to measure whether that has been achieved. But if they say, ‘Here’s a bucket of
money– we want you to buy this number of syringes,’ that is pretty pointless as
well.[16]
4.18
The Queensland Auditor-General identified another difficulty. Given the
multijurisdictional nature of many national funding agreements it is difficult
to articulate outcomes and specify measurements:
So there will be a number of factors that are impacting on
the achievement of the outcome, not all of which will be controlled by either
the state or the Commonwealth…[17]
Key performance indicators
4.19
Effective performance measurement is facilitated by the development of
relevant key performance indicators (KPIs). The Commonwealth Auditor‑General
advised the Committee that the shift to outcomes measurement ‘requires
performance indicators that link directly to outcomes’.[18]
Likewise, the CRC told the Committee that its ability to provide useful
performance reporting depends on developing relevant KPIs that are linked to
objectives and outcomes.[19]
4.20
The Committee asked the PC what process is used to develop the KPIs. The
PC informed the Committee that the high level indicators were developed by the Council
of Australian Governments’ (COAG) working groups and endorsed by COAG.[20]
These indicators were ‘fairly broad’ so the PC consulted with Ministerial
Council data subcommittees, PC review working groups, data providers and the
CRC to develop more specific indicators.[21] The PC advised that the
indicators are continually revised on advice from the CRC and stakeholder
consultation is repeated with each revision.[22]
4.21
Despite the attempts to align KPIs with outcomes and objectives,
witnesses repeatedly claimed that KPIs are inadequate. The Queensland Government
told the Committee that a number of KPIs are ‘not relevant’.[23]
In their submission, the NSW Government argue that KPIs are often not
appropriate:
Performance indicators should be well connected to, and
provide comprehensive coverage of, the high level objectives and outcomes of
agreements. Currently this is not the case for many NAs and NPs.[24]
4.22
Specifically, witnesses were highly critical of the profusion of KPIs. A
number of witnesses pointed to the National Healthcare Agreement as an example
of the unwieldy overuse of KPIs. The NSW Government told the Committee that
although 70 indicators may provide a useful overview of the health system it is
neither sustainable nor meaningful.[25] Further, there is
insufficient data available to measure the indicators and the CRC could only
‘meaningfully report against 25 of the 70 indicators’ in 2010.[26]
The NSW Government concedes that data could be provided for more of the
indicators but warns that the cost of achieving such improvements ‘cannot be
justified’.[27]
4.23
Ultimately, the NSW Government argues for rationalisation of KPIs:
Fewer, more meaningful indicators across the spectrum of
agreements will facilitate a sharper focus on what really matters, and make it
easier for the public to understand the performance of their governments.[28]
4.24
Another example, provided by the Queensland Government, is the National
Agreement for Indigenous Reform. This Agreement has 27 indicators and the
Queensland Government only has useful data for 14 of the KPIs and questions
whether or not it should set up programs to satisfy the other 13 indicators or
‘focus on the areas for which we do have data and time series data.’[29]
4.25
The Treasury (Treasury) acknowledged that this is an area that needs
attention and indicated that the Heads of Treasuries (HoTs) Review has addressed
the issue. Treasury told the Committee that a balance of KPIs is required that
will ‘give the best analysis of the outcome you are trying to achieve whilst
keeping [the number of KPIs] to the minimum you need to do that’.[30]
4.26
The Queensland Government, among others, drew the Committee’s attention
to the Conceptual Framework for Performance Reporting endorsed by COAG
in February 2011 which provides a general guide for developing KPIs and
addresses many of the issues raised by witnesses.[31]
Data quality and collection
4.27
Witnesses repeatedly stressed the inadequacy of the data available to
assess performance under national funding agreements. The CRC highlighted its
frustration over this ongoing issue and told the Committee its ability to
‘meaningly report on the achievement of outcomes year‑on‑year is
significantly constrained by the availability and quality of nationally
comparable data’.[32] The CRC said bluntly
that ‘accountability and transparency fail’ without quality data.[33]
4.28
National Disability Services (NDS) like many other witnesses supported
the move to an outcomes focus, however also warned that currently available
data was inadequate to measure outcomes.[34]
4.29
The Committee heard that a number of areas need to be addressed to
improve the availability and collection of data and ensure effective
performance reporting, including:
n timeliness; and
n comparability.
Timeliness
4.30
In their submission the CRC identified the timeliness of data as one of
the areas that required attention.[35] This concern was shared
by others including NDS, who provided the Committee with a relevant example of
the problems caused by time gaps in data collection. NDS informed the Committee
that the CRC had been forced to draw on data from the 2003 Survey of Disability
Ageing and Carers (SDAC) for its latest report which would suggest that the
findings do not accurately reflect the current situation:
…a six-year gap between collections makes tracking the
effectiveness of the Agreement in relation to a range of key indicators,
including employment participation, almost impossible.[36]
4.31
In his testimony to the Committee, the Chief Executive of NDS elaborated
on the distortion caused by the lack of adequate, timely data in this area:
I think the standout case there is figures on workforce
participation and employment. These are available at present only through one
source, which is the Survey of Disability, Aging and Carers, and that occurs
every six years. ... In that area where the government has such a strong focus
on increasing workforce participation, where workforce participation is such a
key driver of the economy, it seems to me ludicrous to be relying on figures in
this area that are six years out of date.[37]
4.32
The Committee pursued this matter with the Australian Bureau of
Statistics (ABS) and were told that, due to policy changes in this area, the
SDAC will be expanded and will be run every three years.[38]
Comparability
4.33
Another issue of concern to the CRC is the availability of nationally
comparable data.[39] The CRC explained to the
Committee that, while it supports the need for state and territory flexibility,
in the interests of transparency and accountability there is a need to be able
to compare data across governments.[40] The challenge for the
CRC is that different governments set different targets to achieve the outcomes
of national funding agreements.[41]
4.34
For example, the CRC told the Committee that, with regard to the
Literacy and Numeracy National Partnership, variations across jurisdictions
have included:
n proportion of
participating schools and students and the criteria for selecting participating
schools;
n domains, year levels,
size of student cohort, student characteristics and sectors for measurement;
and
n calculation of
targets and methodologies for establishing baselines and the total number of
targets.[42]
4.35
The Committee asked the ABS what steps have been taken to improve the
comparability of data across the nation. The ABS explained that the
Commonwealth processes had to be clarified before the issue could be taken up
with the states and territories.[43] Now that those processes
are established at the Commonwealth level, the states and territories are being
engaged and attempts made to collect comparable data across jurisdictions:
There is a whole range of differences in the systems in
states and territories that we are sorting out in that process as well so that
we can get common measurement not so much in the way the services and systems
work differently but in what the outcome differences are.[44]
4.36
The CRC also alerted the Committee to the need for data to be built up
over time to provide both meaningful comparisons across jurisdictions and
longitudinal comparisons of changes and trends, a point made by PM&C as
well.[45] The CRC advised that, as
the CRC has been operating for nearly three years, solid foundations have been
put in place and this aspect of data collection is improving.[46]
Generic data collections
4.37
To streamline data collection and remove some of the burden being placed
on line departments and service delivery agencies, the Committee heard that it
would be useful if generic data collections could be developed to satisfy the
requirements of various reporting frameworks. Dr Baker, Chief Executive Officer
of NDS suggested that the PC principle ‘one report, many uses’ would
significantly relieve the reporting burden on smaller agencies:
The picture from the point of view of service providers is
that they are often feeding into multiple data collections, some of which
overlap and not all of which include meaningful data items. It seems to me that
there would be sense in…auditing that and producing a reduced but more
meaningful consolidated set which may have multiple purposes.[47]
4.38
The Committee questioned if it was feasible to develop a uniform set of
KPIs that would satisfy the reporting requirements of different sectors.
Dr Baker was unsure but pointed to similar work being done to rationalise
quality compliance systems and suggested this may provide a model:
There are attempts at present to try to ... look at the different
quality systems to which organisations have to comply and do a cross-check. So,
if they have complied with one quality accreditation system, then they may have
complied, in effect, with 90 per cent of another requirement. The same
principle could apply to data as well. There will be some data required that is
distinct to one particular program or sector and there will be some that will
be common to many.[48]
Meaningful interpretation
4.39
Meaningful interpretation of data to assess the quality of outcomes was
also of concern. The Committee heard that the focus on narrow or quantitative
data could make it difficult to assess the quality of the larger outcomes being
achieved. For example, NDS explained to the Committee that while public policy
outcomes stress the participation of people with a disability ‘in all domains
of life, not just the economy but civil society’ such an outcome is very
difficult to measure:
…our capacity to measure in a meaningful way what
participation in non-economic terms means is not easy.
4.40
The Committee pursued this issue with a number of witnesses and asked
what methods were being implemented to ensure meaningful interpretation of the
data collected. In particular, the Committee wanted to know what was being done
to supplement the facts and figures of quantitative data with more sophisticated
qualitative information about the quality of outcomes. PM&C recognised the
need to assess ‘quality of life’ outcomes but maintained that this could be
done without resorting to direct qualitative methods as Mr Hazlehurst, First
Assistant Secretary, explained:
It is in those spaces where the data challenges of course
become the most challenging. It is still data. I do not think we are likely to
end up in a situation where it would go as far as qualitative data in the form
of things like reports from focus groups. However, there is plenty of data that
is collected, including of course by the ABS, which is survey data which goes
to people’s level of satisfaction with either services that they have received,
their quality of life, the amenity of their local neighbourhood, their feeling
of empowerment in the workplace,…things like that where it is not about things
that you can see as easily, it is more about people’s perceptions.[49]
4.41
While acknowledging the validity of collecting qualitative data,
PM&C informed the Committee that collecting qualitative data is not cost
effective:
The obvious thing to say about that is that those require
intensive, expensive, nationally collected surveys in order to come up with an
accurate representation of, if you like, the pattern of outcomes across the
country that one requires for administering a set of arrangements like this.
For a particular one-off purpose we are talking about national data collections
that have to be replicated over and over again…[50]
4.42
The Committee questioned if the focus on facts and figures would blur
the distinction between outputs and outcomes and inhibit the cultural
transition needed to ensure the quality of the outcomes of the IGA FFR.
PM&C held that quantitative data was the only means of measuring many
outcomes and cited the example of reducing infant mortality where a figure
indicates the quality of the outcome.[51]
4.43
To clarify the issue, PM&C used the example of the National
Education Agreement where the emphasis is on measuring student results:
The sorts of measures in there are not measures of how many students
receive X, Y and Z. …They are actually measures of achievement of outcomes by
those students. So…wherever possible, the desire has been to shift to those
measures that are not to do with counting the number of things that have been
delivered to people.[52]
4.44
The Committee asked how to ensure that both qualitative and quantitative
results were being considered in the analysis of data to guarantee that quality
outcomes were being achieved. Treasury advised that the issue must be addressed
when KPIs are being formulated at the beginning of the process:
You would expect these qualitative issues to be considered in
the policy design up front and thought about in terms of the policy that is
formulated and then looking to get an appropriate expression of the outcomes of
that and then indicators to measure it.[53]
4.45
The Committee’s questioning on qualitative results related to the
question of how to effectively assess the overall quality of outcomes in a
sector. The Committee also questioned whether sectoral assessments alone were
sufficient due to the large scale of the spending, and hence whether a more
integrated measure or limited set of high level indictors of national outcomes
might be needed. Members of the Committee asked the PC whether ‘development’ or
‘wellbeing’ indicators at a national level were being used to assess the
outcomes for national funding agreements—such as the ABS’s program into Measures
of Australia’s Progress—and whether data used for national funding agreements was
being cross linked to these types of initiatives.
4.46
The PC told the Committee that it had not been requested to use this
type of complementary measure of welfare. Further, the PC representative made a
point of detail saying that he was cautious of using indices due to the
complexities involved:
I know how to interpret a specific indicator. I know what the
numerator and denominator were. I can work out the data quality issues and the
context and I can make an informed assessment. When you start putting an index
together, you have to be very careful because the debate then becomes about
what is included in the index, what you left out of the index and how you weighted
the different components of the index.[54]
Improvements underway
4.47
The Committee is aware that, as with the implementation process, the
performance reporting framework has been reviewed by the HoTs and the CRC and
acknowledges that these bodies have made a number of recommendations to address
the issues identified in this chapter. Additionally, witnesses drew the
Committee’s attention to a range of improvements that are underway, including:
n Conceptual
Framework for Performance Reporting;
n whole-of-government data
integration project;
n CRC reports and recommendations;
and
n Steering Committee
for the Review of Government Service Provision analysis and reforms.
Conceptual Framework for Performance Reporting
4.48
The Queensland Government drew the Committee’s attention to the Conceptual
Framework for Performance Reporting endorsed by COAG in February 2011 which
provides a general guide for developing KPIs.[55] The guidelines set out
clear steps to ensure KPIs are linked to objectives and outcomes, and are
meaningful, timely and comparable.[56] The guidelines are to be
used in conjunction with the IGA FFR and the Federal Finances Circular 2010/01,
Developing National Partnerships.
Whole-of-government data integration project
4.49
The ABS advised the Committee that a whole-of-government data
integration project has been established to facilitate the collection and
comparability of data across the country.[57] The project has been
initiated by portfolio secretaries and a governance board chaired by the
Australian Statistician has been set up.[58]
4.50
The ABS told the Committee that the project will allow all information
held by Commonwealth agencies to be interrogated for statistical and research
purposes.[59] However, the ABS assured
the Committee that steps have been taken to ensure transparency and
accountability, including legislative accountability:
It is high powered in terms of liberating the data for the
sorts of purposes that…this committee is interested in, but it also has very
strict controls around privacy and confidentiality. It can only be done for
public benefit.[60]
COAG Reform Council reports and recommendations
4.51
The CRC is the key accountability body for COAG under the IGA FFR
and is tasked with reporting on performance for all National Agreements (NAs)
and National Partnerships (NPs).[61] The Committee was told
that the CRC plays a ‘pivotal role’[62] in providing
transparency and accountability for the COAG reform agenda. The Queensland
Auditor-General spoke of the ‘rigour’ that the CRC is bringing to performance
reporting.[63]
4.52
In its second annual report on the reform agenda, the CRC found that
overall performance reporting is improving and is simpler, standardised, and
more transparent.[64] However, the CRC
continues to be concerned over data quality including ‘data availability,
comparability, timeliness, frequency of collection, accuracy and the ability to
disaggregate data’.[65] The CRC has recommended
that these issues must be addressed, particularly with regard to NPs.[66]
4.53
Questioned on the seeming slowness of reform with regard to the performance
reporting framework and data quality and collection, the CRC cautioned that
reform in this area takes time.[67] The CRC said that data
development is complex and expensive, explaining that service delivery data,
drawn from administrative data, is essential for monitoring NAs.[68]
Administrative data is largely collected by state and territory governments and
changes and improvements involve considerable negotiation:
…[administrative data systems] were set in place at the state
level for the state government’s purposes or even for the service providers’
purposes, and you are trying to aggregate administrative data not only to, say,
the school system of a jurisdiction but then to jurisdictions across Australia
so that it is comparable. It takes a long time to agree how they are going to
define certain indicators and how they are going to collect the data. It
changes computer systems. It changes administrative systems.[69]
4.54
The Business Council of Australia (BCA) was critical of COAG’s slow
response rate to CRC reports and recommendations.[70]
The Committee asked the CRC if its recommendations were being responded to in a
timely manner by COAG. The CRC assured the Committee that COAG is taking action
on its recommendations.[71] The CRC advised the
Committee that many of its recommendations have been simple reforms not
requiring COAG approval and have been directly taken up by data development
committees across jurisdictions.[72]
4.55
The CRC informed the Committee that, in response to a range of
recommendations concerning the performance reporting framework from the CRC and
others, COAG had initiated the Heads of Treasuries Review (HoTs Review). More
recently, COAG has announced a review to specifically assess the performance
framework of each individual NA ‘to ensure progress is measured and all
jurisdictions are clearly accountable to the public and COAG for their efforts’.[73]
The Conceptual Framework for Performance Reporting, mentioned earlier,
details the process to be followed for this review. The Committee asked what
timeframe was in place for this review and was told that the review will be
completed by the middle of 2012.[74]
4.56
The Treasury (Treasury) and PM&C updated the Committee on the
progress of this review. The performance framework for each agreement is being
examined to ensure that there are clear links between outcomes and indicators
and that indicators are sound:
…ensuring that there is no ambiguity in indicators in how
they relate to the outcomes so that from the lay person’s or public’s point of
view there is a clear understanding; if you have an indicator, if you see a
movement or change in the data over time, you know what that is trying to
measure…[75]
4.57
Treasury told the Committee data is being reviewed to ensure its
veracity and frequency and that it is measureable over time.[76]
Treasury advised that gaps in data are being identified and the opportunity
taken to assess whether or not such data can be collected cost effectively:
If there is an absence of data in an area, you need to look at
the benefit of the data being captured versus the cost of actually doing it.[77]
4.58
In its submission to the Committee, the Secretariat for the Steering
Committee for the Review of Government Services elaborated on this principle
explaining:
n the benefits of new
data collections or improvements to collections and reporting must be
reasonably expected to outweigh the associated costs to service providers, data
agencies, reporting agencies and agencies required to respond to reports.[78]
Steering Committee for the Review of Government Service Provision
4.59
Supported by a Secretariat within the Productivity Commission, the Steering
Committee for the Review of Government Service Provision provides support to
the CRC through a range of reports.[79] Specifically, under the
IGA FFR, the Steering Committee collects and collates the performance data for all
NAs, and a number of NPs, for the CRC.[80] The Secretariat advised
the Committee that the Steering Committee also has a role in assessing the
quality of data collected for performance reporting:
…the data providers provide a data quality statement
according to the Australian Bureau of Statistics’ data quality framework. The
Steering Committee then summarises that information and adds some of its own
commentary to that in what are called ‘comments on data quality’.[81]
4.60
The Secretariat is optimistic that there is overall improvement in data
systems including in the quality, availability and timeliness of data.[82]
The Secretariat identified both the HoTs Review and the current review of the
performance framework of each individual NA as positive steps and told the
Committee that a lot of work is being done at ground level to improve data
quality.[83]
4.61
The Committee asked the Steering Committee Secretariat how advanced the
improvement in data collection was. The Secretariat explained that improvement
varies across the different areas covered by the NAs, depending on the specific
problems associated with different types of data. For example, the National
Indigenous Reform Agreement and the National Disability Agreement present
inherent difficulties that will take some time to resolve. On the other hand, data
quality and collection for the National Healthcare Agreement has made ‘rapid
progress’ due to a concerted effort across jurisdictions:
That has come about through system changes at the
jurisdiction level, where jurisdictions are doing things differently, and
through significant changes by the main collector or manager of the health
data, which is the Australian Institute of Health and Welfare. They have done a
good job of making more data available more quickly.[84]
4.62
The Committee asked if the work on improving data quality and collection
is being undertaken formally and coordinated between the Commonwealth and
state/territory governments. The Secretariat advised that the ABS and the
Australian Institute of Health and Welfare, both national data agencies, were
responsible for the data improvement process.[85]
4.63
The Committee asked if there was a timeframe for completion of the
improvement process and was told that the timeframe could vary depending on the
type of work that was needed. For example, the Secretariat explained that the
issues regarding data on homelessness for the National Affordable Housing
Agreement presented conceptual problems that would need academic research:
…developing a new methodology for counting the homeless is an
academic piece of work and it is taking academic time frames to be resolved. It
is quite a difficult conceptual issue and you want it done right.[86]
Committee comment
4.64
Performance reporting is crucial to understanding the success or
otherwise of COAG priorities and the IGA FFR itself. However, through the inquiry
the Committee found that better performance reporting will require additional
effort and sustained focus. For this to become a reality there will be a need
to:
n drive cultural
change;
n clearly set
objectives and outcomes, and develop valid key performance indicators;
n collect data of a
higher quality more quickly, while streamlining administrative burdens; and
n ensure meaningful
interpretation of the data that is collected.
4.65
Each of the above improvement points is discussed below.
4.66
Driving cultural change towards full adoption and implementation of the
principles in the IGA FFR is critical to realising the potential benefits it
promises. Achieving deep seated cultural change will require additional effort
within the Australian Public Service (APS) and across the different levels of
government. It will also require time and concerted effort. The Committee
recognises that the IGA FFR is part of a broader shift in public sector
management, with an emphasis on outcomes and enhanced accountability. Commitment
will be needed to overcome entrenched practices which do not accommodate the
fundamental principles of this new perspective on public administration.
4.67
The Committee is satisfied that the cultural change required is well
understood within central agencies and that cultural change is underway.
However, the evidence suggests that personnel on the ground in line departments
and service delivery agencies have still not grasped, or at least have not
fully adopted, the consequences of the change.
4.68
While the Committee accepts that cultural change will take time to
filter down through the various layers of the bureaucracy, it believes that
positive steps can be taken to encourage and support such change. The Committee
notes that some steps are being taken with the development of Federal Finances
Circulars and the Conceptual Framework for Performance Reporting. However,
more can still be done.
4.69
The delivery of training to all staff involved in IGA FFR processes and
performance reporting is a key additional step towards the cultural change
needed.
4.70
The Committee therefore recommends that PM&C and the central agencies
implement a structured approach to ensure that all relevant staff receive specific
training to enhance their understanding of the framework and develop the skills
required to meet performance reporting requirements. In addition to dedicated
training, the Committee recommends that relevant broader APS training be
amended to incorporate information on the IGA FFR. For example, training on the
Financial Management and Accountability Act 1997 and general public
sector administration courses should reference and explain the importance of
the IGA FFR and its principles. This is considered important to raise
awareness of the IGA FFR to APS staff generally.
4.71
The Committee also recommends that the Commonwealth works to ensure that
other jurisdictions implement a similar approach to training. Up-skilling only
APS staff is insufficient; to realise the full benefits inherent in the IGA FFR
state and territory officials will need equivalent skills and buy-in.
4.72
The Committee considers it essential that clear objectives and outcomes for
national funding agreements be negotiated, agreed and documented – and that
these are supported by valid key performance indicators. The Committee believes
that the apparent lack of clarity surrounding outcomes for different agreements
is seriously undermining the principles of the IGA FFR. If outcomes cannot be
satisfactorily agreed and articulated in a way that provides sufficient clarity
to all parties this suggests that the practicality of the principles of the
IGA FFR may need to be reconsidered. However, this should not be
interpreted to mean that every aspect of an agreement needs to be quantified or
linked to a numerical KPI for the agreement to be meaningful. Agreement and
clarity of outcomes is about mutual understanding of the end goal, not whether
the end goal can be perfectly broken down into a long list of numerical KPIs.
4.73
In this regard, the Committee stresses the need to ensure that serious
consideration is given to the relevance of KPIs, and whether these KPIs are
supported by existing data collections. The Committee notes that the Conceptual
Framework for Performance Reporting is being used to streamline KPIs with
the aim of ensuring that KPIs are measureable, relevant and directly related to
outcomes. The Committee understands that this review is well underway and that
it will rationalise and simplify KPIs. The Committee considers that this
exercise is critical to address the concerns raised over the proliferation and
meaningfulness of indicators.
4.74
Despite a potentially improved set of KPIs, the Committee remains concerned
at the ongoing problems with data quality and collection. In many instances
data needs to be collected, complied and analysed more quickly than is
currently the case. In instances where data is not currently collected, consideration
must be given to the allocation of funds to support its collection. If there is
no data available to measure a KPI and collecting the necessary data would not
be cost effective, the KPI should be removed or amended.
4.75
Regarding the reporting burden, national funding agreements should seek
to streamline reporting requirements and consolidate data collections wherever
possible. In essence, the goal should be easier data collection with fewer
‘survey’ forms. Although the Committee is aware that various attempts are
underway to improve data collection approaches, it was not clear that enough
work had been done towards the ‘single report to multiple agencies’ ideal, or towards
compiling core data sets for key national priorities.
4.76
The benefits of moving towards the ‘single report to multiple agencies’
ideal are obvious, including minimising the need to reconfigure and repackage
the data collected for each respective reporting requirement. The Committee also
sees potential benefit in developing a core set of standard data requirements for
all reporting which include key areas of national interest such as indigenous
affairs and provision of services to low socio economic status members of the
public.
4.77
The Committee therefore recommends that PM&C, Finance and Treasury report
back to the Committee on work undertaken to move towards the ‘single report to
multiple agencies’ ideal and the potential to develop a core set of standard data
requirements for key areas of national interest.
4.78
In this regard the Committee is particularly interested in the
development of the whole‑of‑government data integration project and
urges all jurisdictions to take whatever steps necessary to ensure it
progresses expeditiously.
4.79
It is not sufficient just to collect data, it is necessary to ensure
meaningful interpretation by decision makers and the community. The Committee is
concerned that the proliferation of KPIs comes at the detriment of higher level
measures or summary indicators that allow decision makers to meaningfully gauge
ultimate outcomes. Despite some suggestions that sectoral or national level
indicators (either single or multiple) are complex to compile the Committee
feels that it is important that a tangible set of indicators is available. The
Committee therefore supports moves towards better links between national
agreement reporting and complementary measures of national outcomes such as
ABS’s Measuring Australia’s Progress initiative, even if further research is
required to achieve this result.
4.80
Overall the Committee has identified a range of potential improvements
to the performance reporting framework under the IGA FFR. The Committee
acknowledges that there are many initiatives underway towards improvement, but
is keen to see more action to bring these to fruition and ensure full
implementation. The Committee notes that the CRC has already identified many of
the problems examined in this chapter, however remains concerned at the slow response
time by COAG to the CRC’s reports and recommendations. The Committee recommends
that COAG take steps to respond to the reports and recommendations of the CRC
in a timelier manner.
Recommendation 5 |
4.81 |
The Committee recommends that a structured approach be
developed and implemented by the Department of the Prime Minister and Cabinet
and other central agencies to ensure relevant staff receive specific training
to enhance understanding of the Intergovernmental Agreement on Federal Financial
Relations and develop the skills required to meet outcomes focused performance
reporting requirements. |
Recommendation 6 |
4.82 |
The Committee recommends that the Department of the Prime
Minister and Cabinet, in consultation with other central agencies, establish
processes to ensure that there is clarity of the outcomes to be achieved and
these are clearly reflected in national funding agreements. The committee
asserts that to underpin the achievement of outcomes, mutual understanding of
the end goal must drive the cultural change, the training and skill
development, and the quality and timeliness of data collection and
publication. At all times, outcomes should be the focus in the development of
all national agreements. |
Recommendation 7 |
4.83 |
The Committee recommends that the Department of the Prime
Minister and Cabinet, in collaboration with agencies such as the Australian
Public Service Commission, should lead a process to provide training across
the broader Australian Public Service which incorporates information on the Intergovernmental
Agreement on Federal Financial Relations to explain the importance of the Agreement
and its principles. |
Recommendation 8 |
4.84 |
The Committee recommends that the Commonwealth works through
the Council of Australian Governments to ensure that states and territories
develop and implement a similarly structured approach to foster cultural
change throughout departments and agencies and ensure all staff receive
relevant training to enhance understanding of the framework and develop the
skills required to meet outcomes focused performance reporting requirements. |
Recommendation 9 |
4.85 |
The Committee recommends that the Department of the Prime
Minister and Cabinet and central agencies report back to the Committee within
six months on work undertaken to move towards the ‘single report to multiple
agencies’ ideal and the potential to develop a core set of standard data
requirements for key areas of national interest. |
Recommendation 10 |
4.86 |
The Committee recommends that the Prime Minister through the
Council of Australian Governments, take steps to respond to the reports and
recommendations of the Council of Australian Governments Reform Council within
three months. |