CHAPTER 4
The Performance Framework
4.1
The committee recognises that the DES Performance Framework (Framework) is
critical to determining whether organisations have the opportunity to continue
delivering services. Consequently, it has received significant attention. There
were several issues raised during the course of the inquiry related to the
Framework including:
(a) The use of regression analysis;
(b) Quality of outcomes promoted by the performance framework; and
(c) Possible manipulation of the Framework.
Performance Framework
4.2
The Framework, commonly referred to as the 'star-system', is used to
rate DES providers to facilitate relative performance comparisons. The
Framework includes three Key Performance Indicators (KPIs): efficiency,
effectiveness and quality. A provider's star-rating is based on a measurement
of their performance against the efficiency and effectiveness KPIs. The
star-rating measures the relative performance of a provider by comparing job
seeker engagement and achievement of outcomes to all other providers nationally
using a regression analysis. Quality is measured by the requirement for all DES
providers to be certified as meeting the Disability Service Standards.[1]
An overview of the Framework was provided in Chapter 2.
4.3
The star-ratings of providers are derived through the use of statistical
regression analysis to allow the comparison of provider performances across
Australia. As the Department explains:
The Star Rating model calculates what providers could
reasonably be expected to have achieved given the given the unique set of job
seekers they have assisted in their specific labour market. The regression
model used for DES specifically takes into account the impact of the person's
disability on their relative chance of being placed into sustainable
employment. The more the provider's actual outcome rates exceed expected rates,
then the higher the performance score.[2]
4.4
The star-ratings model was initially developed by the Department in
consultation with the South Australian Centre for Economic Studies in 1999 for
Job Network services. The Department advised that the star-ratings have been
through several independent reviews since their development in 1999: Access
Economics in 2002; the Productivity Commission in 2002; and the Australian National
Audit Office in 2005.[3]
These independent reviews were generally supportive of the statistical
robustness of the star-system. The committee notes that the last review
predates the current DES program and it is unclear if any compatibility issues may
exist.
4.5
Jobs Australia – a peak body for 280 non-profit organisations that
deliver a variety of employment and other related services – put it to the
committee that:
The first thing to say about [the Framework] is that it is
not perfect but it has been continuously refined. It was then adapted to get it
as close to the best of breed that it can be. It is widely regarded, in global
terms, as world's best practice in terms of measuring the comparative
performance of providers of public employment services.[4]
4.6
At the committee's public hearing in Melbourne several witnesses
criticised the use of regression analysis as it is too difficult to understand.
Dr Ken Baker, Chief Executive Officer of National Disability Services argued
that:
I think the whole regression analysis is a very mysterious
process. It is described as the black box. I do not say the process is not
sophisticated, but it is very difficult to have confidence in it unless we can
see its inner workings and its inner workings are not publicly disclosed.[5]
4.7
Department Deputy Secretary Ms Sandra Parker explained why some people argue
that the regression model is difficult to understand:
It is a complicated model. There is no question that it takes
account of a whole range of job seeker characteristics and disabilities. It
compares people against other performers and what is expected of them to
perform in a particular labour market at a particular time with a particular
cohort of job seekers. It is a complicated model. People talk about it as a
black box because it is complicated. It is not something we have tried to hide
or restrict people from seeing.[6]
4.8
However, the Department acknowledged that it did withhold some information
about the regression model:
What we do not do is provide the whole set of data on it to
have people pull it apart and ask a million questions. We try to keep it at a
principles level and explain to people how it operates so they understand that
it is a robust model.[7]
4.9
The committee agrees that the regression model is complex and sought an
additional briefing from officers of the Department. The Department provided a
detailed briefing to the committee on 8 November. With the aid of a detailed
information session, it is much easier to understand the workings of the
regression model and how it impacts upon providers. The committee recognises
that absent of such information, the regression model could seem overly complex
and opaque.
4.10
A common criticism of the regression model put to the committee is the
ability of seemingly small changes in performance to significantly change a
provider's star-rating. Results from the regression analysis are produced based
on a national comparison of similar providers, and most providers are not in a
position to anticipate macro-level changes in employment prospects, placement
numbers and other factors that will impact upon their own star-rating. The
committee heard of examples in which a provider placed more people with
disabilities in one quarter than in the previous quarter, and yet its star
rating declined.[8]
4.11
Small providers questioned whether regression analysis was the best tool
for assessing their performance. As smaller providers deal with fewer clients,
there seems to be greater scope for small changes in outcomes to significantly
alter star-ratings.[9]
4.12
Undoubtedly the regression model is sophisticated and complex. The
committee considers that stakeholder understanding is crucial as it allows
providers to have confidence that the regression analysis fairly represents
their performance. Given the importance of the regression analysis to the
ongoing survival of service providers, the regression analysis should be open
to discussion and criticism.
Committee view
4.13
While there were numerous criticisms of the way in which the Framework
assessed the quality of outcomes, no-one was able to put forward an alternative
model to assess relative performance across such a large program. The
regression model is understandable once it is explained, but the committee can
see how the process and its outcomes are not easily understood absent a
detailed briefing from the Department. The committee considers it paramount
that providers view the regression model as robust and fair.
Recommendation 6
4.14 The committee recommends that the Department, in consultation with key
stakeholders, consider the DES performance framework regression modelling
information with a view to public release in ways that are accessible,
meaningful and helpful to advancing program objectives.
Assessing quality
4.15
The current Framework has been criticised as being too narrowly focused
on short-term numerical outcomes, rather than looking after the best, long-term
interests of clients.[10]
According to the Post Program Monitoring Survey, only a quarter of people who
achieve a 26 week outcome are employed or training three months later.[11] These figures
have been used by some to argue that too many outcomes in DES-ESS are not of a
high quality, and that this situation is encouraged by the lack of quality
measures in the Framework.
4.16
Although the quality of outcomes achieved is one of the three KPIs in
the DES-ESS program, it is not included in evaluating star-ratings of providers.
The regression analysis is limited to measuring quantitative elements. Because
of the difficulty in measuring qualitative outcomes, KPI 3 – Quality, is not
part of the regression analysis.
4.17
Quality is said to be assured though the requirement to meet the Standards.
The annual audit for Standards is a pass or fail test. Many submitters perceive
that star-ratings have become the sole determinant in deciding who has
contracts extended and who must retender. For this reason many submissions have
argued that quantity has overtaken quality as the principal concern.[12] The
Department recognises such sentiments exist in some quarters:
Some stakeholders have raised concerns that the DES
Performance Framework omits assessing against the quality of the job and that
DES providers are not financially rewarded or assessed against the number of
hours and the wages that the participant has achieved. It is therefore
determined by some stakeholders that the DES performance framework is driving
quick outcomes rather than quality outcomes and this is inconsistent with the
Disability Service Standards.[13]
4.18
The committee heard that the requirement under the Act to focus on the
best interests of a person with a disability were often in conflict with the
incentives under the current Framework. The committee was informed that quality
measures were removed from the regression analysis model because of the
difficulty of accurate measurement on the recommendation of the Industry
Reference Group. DEA succinctly expressed a regular criticism of the Framework's
lack of quality measures:
KPI 1 and KPI 2, which are part of the performance framework,
are more about speed-to-place and speed-to-outcome, and they only measure up to
six months. There is no measurement beyond six months employment. There is
nothing about the quality of the jobs, hours of work, dollar value of
employment et cetera.[14]
4.19
EDGE Employment Solutions explained how this might look in practice:
For example, a person with a benchmark of 8 to 15 hours in a
position of eight hours per week in an enclave of six people at $8.15 per hour
receives the same rating as in an individualised job of 15 hours per week with
individual support at a for-award wage.[15]
4.20
One service provider explained the mindset and behaviour – bordering on
creaming (discussed below) – that these circumstances create:
[I]f we were to focus on the performance ratings when we
visit the local crisis accommodation centre to talk about the potential to get
into work, out of the 10 residents we would focus on the two or three who are
most likely to get a job and get us our results for performance processes,
rather than the six or seven who want to have a go and try their hand at
employment and so become members of society in a productive manner.[16]
4.21
Key Employment Association argues that absence of quality accountability
measures means the Framework fails to reward best practice:
The Performance Management system imposed by DEEWR is not a
true measure of success in achieving the goals of the program. That the credit
under the Stars and Fees are equal for a short term casual job and a 4 year
apprenticeship is simple and compelling evidence of a failure in the system to
recognise and reward best practice.[17]
4.22
By way of example, KEA provided the committee with details of a program
run by their organisation designed to place people with disabilities into
apprenticeships. The program has placed 40 persons into apprenticeships in
2011, and it is anticipated that it will place 50 persons into apprenticeships
each year thereafter. KEA argues that the current Framework does not
incentivise providers to strive for outcomes like this when they can receive
the same funding for an eight hour per week job with limited future prospects.[18]
4.23
Because of the difficulty in measuring quality in the regression model
which is limited to quantitative inputs, the committee heard that longer-term employment
outcome measures could be feasibly used as one way to assess quality. Under the
Framework providers receive their largest payment (and star-rating) once a job
seeker attains 26 weeks in employment. It was suggested that a 52-week outcome
may focus providers' attention on achieving quality, sustainable employment.
4.24
The committee recognises that achieving six full months in employment is
a significant achievement for many job seekers who have previously been
excluded from the labour market, and the committee wishes to acknowledge that.
However, given that DES-ESS is a program designed to provide on-going support,
there appears no reason why longer-term outcomes could not be considered as a
way to promote quality while working within the constraints of the regression
model.
Committee view
4.25
The committee considers that quality is as important as quantity in
assessing the performance of DES-ESS. The committee received compelling
evidence that the current lack of quality incentives in the performance
framework is resulting in some job seekers being placed in positions that do
not align with their ambitions or interests. The committee considers that there
is a need to develop a robust and quantifiable quality assessment mechanism for
the provision of services under the DES-ESS.
Recommendation 7
4.26 The committee recommends that the Department develop a robust and
quantifiable quality assessment mechanism for services under the DES-ESS, and
incorporate that assessment mechanism into the performance framework.
Recommendation 8
4.27 The committee recommends that the Department consider a trial monitoring
program of 52-week employment outcomes for clients of DES-ESS providers.
Manipulation of the performance framework
4.28
A number of allegations were made during the inquiry that some service
providers were intentionally manipulating their services so as to improve their
star-ratings. For the most part, allegations were unable to be supported with
evidence proving systematic manipulation, but it seems clear that there is
potential in the current Framework for untoward practices to exist. The
committee is of the view that these allegations are serious and deserve further
consideration.
4.29
These alleged practices are colloquially categorised as parking,
scheming, creaming and churning. Broadly speaking:
- parking is a process of putting the hard-to-help cases to
one side;
- creaming is the provider prioritising clients who are
easier to place in employment;
- scheming is the practice of breaking up an employment
position into several smaller positions to achieve more outcomes; and
-
churning occurs when a person falls out of employment at
the end of six months after the final outcome has been achieved after which
they re-enter the program – often this is achieved through 'buying jobs' or
inappropriate wage subsidies.[19]
[20]
4.30
The Department informed the committee of two other practices that they
are aware of:
-
inappropriate suspensions minimised the official 'time to
placement' of a job seeker; and
-
incorrect exits where providers incorrectly inform the
Department that the participant has received 'independent worker' status to
maximise their own score.[21]
4.31
Because high star-ratings are necessary to ensure ongoing contracts with
the Australian Government, 'gaming behaviours' have emerged to maximise the
star ratings of some providers. The prevalence of these activities is unclear. Mission
Australia contended that although isolated incidents of manipulation may take
place, the regression analysis was sufficiently robust to prevent an average
provider obtaining a four- or five-star rating.[22] However, others argued
that it had become endemic, and that practices that could be categorised as
scheming and gaming were openly discussed as ways to improve performance
ratings at industry conferences.[23]
At a minimum, it was widely recognised that the Framework has the potential to
create a conflict between the best interests of job seekers, and providers
seeking to maximise performance.[24]
The Department informed the committee that:
...[W]e know about each of the concerns that have been
raised. We take very seriously every concern that is raised where it might come
to sharp practice or views about manipulating the star ratings...We have been
doing a whole range of actions to investigate the concerns.[25]
4.32
Perhaps the most common allegation was that of scheming in which one
available job was broken down into several smaller jobs. This practice, while
conceivably to the benefit of service providers who maximise their statistical
performance, does not appear to be an appealing practice from the perspective
of an employer who would be faced with higher training, administration and insurance
costs.[26]
4.33
The strongest public allegation that the committee heard was provided by
NDS' Mr Craig Harrison who related an example from a workshop – organised by
the Department and run by KPMG – in Adelaide in August 2011. At that workshop
two five-star providers allegedly gave presentations instructing other
providers on how to improve their star-ratings. Mr Harrison reported to the
committee that:
...[O]ne of them talked about putting aside government money
to the tune of half a million dollars a year for which they were buying
jobs...The other provider was running programs that were about [creaming] and
parking.[27]
4.34
The committee notes the Department's position that behaviour labelled as
scheming is not necessarily a bad thing, and can lead to quality outcomes for
job seekers in some circumstances.[28]
Nonetheless, there also remains the potential for scheming to be used to
achieve positive outcomes for providers rather than the job seeker.
4.35
There does not seem to be any way to quantify the prevalence of creaming.
Because creaming occurs when a provider only services clients who are easier to
place, prospective clients that were not assisted do not appear in collected
statistics. The Department posited that the Framework actively discourages the
practice because it awards higher ratings for difficult to place clients, but
there does not seem to be any way to ensure that providers will service all
clients equally.[29]
Equally, based on the testimony received in the Melbourne hearing, there is
little chance that a provider engaging in churning practices would come to the
attention of the Department. [30]
4.36
The Department informed the committee that the most common method by
which they become aware of providers who might be manipulating the framework is
through information supplied by other providers.[31] Complaints received by
the Department are investigated by the Department internally.[32] This reliance is
surprising in such a large-scale program that has access to a comprehensive and
pliable database of information on providers, job seekers, and other
information.
4.37
The Department claimed that: the use of inappropriate suspensions and
parking was not occurring; Australian Disability Enterprises were not being
used to manipulate the Framework; no evidence has been presented to them
regarding large-scale churning;[33]
that the use of inappropriate exits was rare and that the Department was
currently taking action to prevent this practice.[34]
4.38
The committee also attempted to clarify whether manipulative practices
are a breach of providers' contractual obligations or whether these practices,
although clearly against the intentions of the program, are legal loopholes. The
committee has not been provided a complete copy of the contract so could not reach
a definite answer on this question. The committee notes that the Department
advised during the hearing that no action has been taken against any DES-ESS
provider during the course of the contract.[35]
Committee view
4.39
The committee considers that the line between responding to incentives
and engaging in practices such as those described above is ill-defined and open
to differing interpretations. Even though the committee accepts that these
practices are unlikely to be endemic there are almost certainly some clients
who engage with service providers in good faith and are subsequently either
neglected (in the case of creaming) or placed in suboptimal jobs for their
needs (in the case of scheming). The committee is concerned with the lack of
transparency around possible scheming practices and would welcome the
Department taking measures to ensure that potential scheming practices are
scrutinised with a view to identifying cases which are not apparently in the
best interests of the job seeker. The committee would encourage the Department
to focus more closely on gaming activities, their potential impact, and ways in
which their use can be identified and managed.
Recommendation 9
4.40 The committee recommends that the Department establish regular,
proactive monitoring practices to identify incidents where the star-rating
system is manipulated by providers.
Recommendation 10
4.41
The committee recommends that the Department include in the Disability
Employment Services – Employment Support Services contract:
- Definitions of inappropriate practices; and
- Provision for the removal of providers who are found to engage in
any practices listed in the included definitions.
Recommendation 11
4.42 The committee recommends that the Department collect and publish
statistics regarding:
-
the number of reports of alleged inappropriate practice;
- action taken in response to these reports; and
- the number of providers found to have engaged in inappropriate
practices.
Senator Chris Back
Chair |
Senator Bridget McKenzie |
| |
|
| Senator Sue Boyce |
Senator Rachel Siewert
|
Navigation: Previous Page | Contents | Next Page