8. Innovation, continuous learning, experimentation, research and evaluation

Access to data

8.1Stakeholders asserted that timely, accurate data is critical to understanding the operation and performance of the employment services system and supporting policy and program design. The joint submission from the Brotherhood of St Laurence (BSL), Centre for Policy Development (CPD), and the University of Melbourne (UniMelb) stated:

[A]ccess to timely and diverse data and evidence is key to the effectiveness of a reformed employment services system, including frontline practice with jobseekers and employers. Data is required to evaluate system as well as provider performance, drive continuous quality improvement, assess the impact of work with jobseekers and employers, support the development and evolution of employment models and practice which is adapted to local labour and service markets and community characteristics, and support relevant research.[1]

8.2Stakeholders raised concern that DEWR releases limited program data, there are delays before data is released to the public, and data is not sufficiently detailed to enable a full understanding of whether the system is operating effectively.[2]

8.3Social Ventures Australia (SVA) observed that data is often difficult to access—particularly across jurisdictional and administrative boundaries. This this can impact evaluation and research capacity.[3] CVGT Employment (CVGT) noted that while some data is available via DEWR’s IT systems, providers are obliged to manually download data relating to participants on their caseload.[4]

8.4The first public release of Workforce Australia caseload data on DEWR’s website occurred in April2023, ninemonths after the start of Workforce Australia.[5] The data released was less detailed and disaggregated than data presented to this Committee during its public hearing in November2022—more than fivemonths earlier.[6]

8.5The Employment White Paper review identified the need for enhanced labour market data and analytical capability in order to understand and respond to the labour market experience of under-represented cohorts.[7] The Australian Bureau of Statistics (ABS) captures a range of data on those who are transitioning into and out of employment and the barriers that they may face.[8] In response to the findings of the Employment White Paper, the Australian Government has committed to funding the ABS to improve its data collection on barriers to job mobility and career progression, job security, quality of employment, and outcomes of employment services and income support programs.[9]

8.6Several stakeholders called for employment services data to be made publicly available in relation to a range of matters, including:

• industry trends;[10]
• participant caseloads, including changes over time;[11]
• participants in and outcomes from Workforce Australia Online;[12]
• ability to track and monitor outcomes;[13]
• compliance outcomes;[14]
• jobseekers’ goals, preferences, and experiences;[15] and
• activities contributing to jobseekers’ Points Based Activation System (PBAS).[16]
  7. Per Capita emphasised the importance of building a fit-for-purpose IT system and of collecting and sharing of data focused on customers’ needs, noting that this data can support service improvements and improved connections between jobseekers and employers.[17] Per Capita indicated that the following matters should be considered in designing data management arrangements for a new system:
• Mechanisms to collect information on services and users. Per Capita suggested that the locations of services be mapped against locations of users (for example, jobseekers), and against job vacancies. According to Per Capita, this type of analyse could be made within the IT system relating to employment services, and linked to other systems (for example, adult education and TAFEs, community services, childcare, and transport).
• Definitions formeasuring and reporting on participant progress and experience, outcomes, and impact which serve the purposes of quality monitoring, licensing, and regulatory oversight.
• Howcosts and value for money can inform commissioning and the administration of purchasing payment systems.
• The data needed to enable research and development and innovation design, as well asresearch and evaluation.
• Interfaces with the ABS and the Multi-Agency Data Integration Project, and any scope to share data with other sectors to inform investment and track collective impact.[18]
  8. Multiple stakeholders noted that de-identified data relating to other human services—including health, justice, and education—is available for public scrutiny.[19]
  9. Stakeholders also expressed concern in relation to information asymmetry between DEWR and providers, noting that DEWR has access to all data held by providers, while providers lack access even to the data they have entered into the system. This leads to duplication of effort where providers create separate databases to meet business needs.[20] The National Employment Services Association (NESA) stated:

[P]roviders enter a lot of data into IT systems, but they can’t necessarily retrieve it. They have no option but to have a separate shadow system so that when they’re asked to demonstrate evidence, they can pull things out.[21]

8.10CVGT stated that access to a live feed of data from DEWR’s IT system’s interface would assist in resolving this issue. In addition, CVGT called for establishment of a data repository to enable access granular and aggregate data, and metadata, on matters such as placement numbers, provider performance, program costs, and participant and employer feedback.[22]

8.11In its 2002 review of Job Network, the Productivity Commission recommended that program data on employment services be made available for independent scrutiny in a timely manner.[23] The Australian Government expressed in‑principle support for this recommendation, noting that implementation would carry significant resource implications.[24] The commission also indicated that concerns about evaluation methodology are best addressed by giving researchers access to raw data. It recommended consideration be given to establishing an independent panel of researchers to advise on the data needed for effective evaluations of employment services, with the panel’s views and recommendations made public.[25]

8.12DEWR publishes program and survey data on matters such as participant caseloads by region and cohort; employment outcomes (reported via Post-Program Monitoring Surveys); and data on compliance activities. DEWR indicated that providers receive operational and performance data to support day-to-day operations, including comparisons with other providers, and between sites within their organisation.[26]

8.13DEWR emphasised the need to balance transparency with privacy concerns when considering the release of data, noting increased risk of re-identification where the volume and scope of data is increased. DEWR also suggested that it does not have the technology or resources to create discrete data for research purposes.[27]

8.14DEWR also highlighted its Research and Evaluation Database, noting that this database provides episodic income support data to support longitudinal analyses and that the database is often used to aid in evaluations and support research projects. DEWR confirmed there are strict security controls in place, including—for external users—a requirement to obtain a Public Interest Certificate.[28]

The public good of public sector data

8.15There is a growing expectation that public data is made available for the public good. This is reflected in the establishment of a regulated data sharing scheme under the Data Availability and Transparency Act 2022. Key objectives of the Act include greater availability of public data, enhancing integrity and transparency in data sharing, and ensuring that data sharing is subject to appropriate controls.[29]

8.16Yilabara Solutions explained that the scheme established pursuant to the Data Availability and Transparency Act 2022 is used to facilitate access to de-identified health data and asserted that employment services should apply a similar approach to improve research, evaluation, and policy design.[30]

8.17Jobs Australia noted that Australia has some of the best data on employment services of any country in the world but puts it to limited use.[31]

8.18The previous chapter detailed benefits associated with sharing and use of Single Touch Payroll (STP) data in simplifying the administration of employment services. Stakeholders observed that greater access to STP data could also enable better tracking and evaluation of long-term outcomes for participants.[32]

8.19Stakeholders also asserted that greater data sharing between government agencies and jurisdictions is needed, noting that this can significantly improve service delivery and enhance research and evaluation.[33] The Victorian Government observed in this respect that during COVID-19, data sharing between Victoria and the Commonwealth was ‘immensely useful’ in supporting economic planning and service development.[34]

8.20The Employment White Paper review noted the need to provide greater access to data as part of the framework to support place-based approaches.[35] Jobs and Skills Australia (JSA) noted that work is underway within the ABS and JSA to improve regional labour market data, including data on vacancies and employment projections.[36]

8.21There have also been calls to address the lack of consistent data to identify disability status across services and population surveys.[37]

Evaluation and research

Evaluation of employment services

8.22DEWR advised that it conducts regular evaluations of employment services according to defined principles of independence and relevance. Evaluations are conducted internally, supported by consultation with external stakeholders who are engaged to advise on and peer-review evaluation methodology.[38]

8.23Evidence to the current inquiry indicated that there is scope to enhance evaluations. For example, WISE Employment indicated that evaluation reports are not released in a timely manner.[39] Consistent with this view, the DEWR website indicates that delays in releasing evaluation reports is increasing. For example, the stage three of the Job Network evaluation covered a period ending May 2001 and was published in May2002.[40] By contrast, the jobactive final evaluation covered a period ending December 2018 and was published in November2022.[41]

8.24The Organisation for Economic Co-Operation and Development (OECD) raised concern as to the rigour of methods used to assess performance in employment services, advocating greater use of counterfactual impact evaluation (an evaluation that estimates the net impact of a program relative to no intervention). The OECD pointed to Canada and Finland as jurisdictions that use this type of evaluation.[42]

8.25There is support for greater evaluation capability and application of counterfactual evaluations such as Randomised Control Trials (RCTs). However, recent analysis noted that just 1.5percent of government evaluations used RCTs.[43] The Paul Ramsay Foundation (PRF) outlined that undertaking RCTs in the Australian context is not straightforward, particularly for programs where access and participation can be universal. It is often necessary to use alternative or quasi-experimental methods which—while useful in evaluating impacts of elements of a program—may not be effective in assessing the program as a whole.[44] Evidence from other countries indicated that there are ethical issues that must be carefully considered and addressed when contemplating RCTs in human services, and especially with vulnerable people.[45] However, as demonstrated by international experiences—including jurisdictions with which the Committee engaged during its European delegation—these issues are not insurmountable. Several numerous international jurisdictions have demonstrated in their more sophisticated and systemic evaluation systems and use of RCTs.

8.26The Australian Government has created an Australian Centre for Evaluation within the Treasury to enable and promote high-quality evaluations across the Australian Public Service (APS). The centre will champion quality evaluations; oversee efforts to improve evaluation capability; embed high-quality evaluation plans in new policy proposals; and collaborate with international counterparts.[46] The centre will collaborate with Commonwealth agencies to ensure evaluations are conducted within a robust ethical framework.[47] On 11 October 2023, government announced the centre’s first partnership with DEWR, which will focus on the use of RCTs to evaluate different features of online employment services.[48]

8.27The Employment White Paper highlighted the critical importance of embedding evaluation into policy design and the need for continuous learning and improvement in services for jobseekers and employers.[49] Research conducted in 2018 by the UK’s Behavioural Insights Team asserts that behavioural insights and robust evaluation can improve employment policy. The research emphasised that policies and programs must be designed so as to take into account the decision-making of all jobseekers, employers, and providers and how the behaviours of these actors intersect. According to the researchers, there are three key areas which are vital to improving employment policy and create better functioning labour markets:

• The use of experimentation in policymaking should increase, and policy makers need to take a proactive approach to understanding and managing change.
• Data should be used to better target policy and empower jobseekers and workers.
• Policy makers should work to build the skills and resilience of workers.[50]
  28. The Committee heard during its European delegation that Sweden makes frequent use of RCTs, for example, to assess the impact of job matching services for refugees and the effectiveness of private service providers. This has led to the development of an ethical framework for trialling new service models and the instilling of a culture of continuous learning and improvement within the public employment service.
  29. A 2019 independent review of the APS advised that main responsibility for evaluations should continue to reside with individual agencies, but a central enabling function could provide guidance and support for agencies on best-practice approaches.[51]
  30. Stakeholders also called for evaluations of employment services to be independent of government to ensure results are viewed as unbiased, credible, and trustworthy.[52] For example, WISE Employment stated:

Providing an objective and independent evaluation of…programs and initiatives, with an emphasis on evidence-based data and being removed from perceived policy or government bias, would give clearer and more prompt indications of the effectiveness of such programs.[53]

8.31By contrast, the OECD noted that the role of the public service has been critical to the success of evaluations of labour market programs in Canada. The OECD drew attention to work by the Canadian Ministry of Employment and Social Affairs to enhance capability in that country’s public service, including via implementation of dedicated platforms for conducting analysis, and the establishment of a Chief Data Office. The OECD also indicated that keeping evaluations in-house allows Canada to dedicate additional resources to the evaluation process and enable Canada to formalise and embed best-practice evaluation approaches.[54]

8.32Stakeholders also suggested alternative approaches to research and evaluation. These tended to be co-operative frameworks involving participation by government, academia, advocates, and industry.[55] Stakeholders pointed to examples of where this approach has been successfully implemented, such as:

• The Centre for Local Innovation in Social and Employment Services at Aalborg University in Denmark.[56]
• The Centre for Work Inclusion at OsloMet University in Norway.[57]
• In Austria, which has a strong focus on continuous learning and evaluation, including a quality management system that utilises balanced scorecards for their 100 or so regional offices with partners (providers) involved. This work is led by a small internal unit utilising external partners.[58]
  33. Stakeholders indicated that cooperative approaches to evaluation are not in place in Australia. For example, Campbell Page noted that there is no centre of excellence dedicated to providing a body of research on the effectiveness of interventions and the factors which contribute to a jobseeker finding and keeping a job.[59]The Getting Welfare to Work Research Team expressed concern at a loss of research capacity in this area of public policy, and suggested that universities and academics be formally engaged in a research and advisory capacity in the employment services system.[60]

Measures of performance and service quality

8.34In addition to raising concerns as to the timeliness of release of evaluations and the methodologies used in their conduct, stakeholders indicated the scope of evaluations could be broadened to capture additional measures of success.[61] Sarina Russo Job Access (SRJA) noted that measures used in previous evaluations of employment services were exits from income support, exits from employment services programs, and reductions in income support. SRJA stated that none of these measures capture sustained employment or income security.[62] The Committee also heard that the Australian Government needs to have a much better understanding of what a quality services looks like and takes to deliver.[63]

8.35The Evaluation Strategy for Workforce Australia Employment Services 2022–2028 includes three key evaluation questions:

1Does Workforce Australia Employment Services meet the needs of key stakeholders including individuals, businesses, and providers?

2How effective is Workforce Australia Employment Services in helping individuals find sustained employment that promotes job security?

3How cost-effective is Workforce Australia Employment Services in achieving its objectives?[64]

8.36The evidence received suggested that the existing evaluation framework appears narrowly and largely focused on employment outcomes without other outcomes that a well-functioning system would achieve and value or explicitly mentioning satisfaction with services. A narrow focus on employment is also reflected in evaluation questions relating to the payment system and provider performance framework, which ask whether those elements of the employment services system are effective in driving employment outcomes without recognising that for some people preparation for employment, building human capital or social participation are also important and worthy goals and outcomes.[65] While cost effectiveness is obviously an important measure, as outlined in Chapter 2, a key criticism of Australia’s approach has been an endless focus on cost reduction with no relationship to ensuring a quality service. No evidence was provided indicating how evaluation of ‘cost effectiveness’ in Workforce Australia considers other outcomes or relates to a quality service.

8.37Stakeholders suggested a range of measures of performance and outcomes which should be considered in evaluating the employment services system. These included the following:

• feedback against a range of domains including individual wellbeing, community cohesion, and social mobility;[66]
• long-term benefits such as whether income security has observably improved;[67]
• human capital development, including improved work readiness and improved capacity for social and economic participation;[68]
• community contribution;[69]
• reduced reliance on other social services;[70]
• quality of jobs obtained;[71] and
• social return on investment.[72]
  38. Stakeholders also argued that evaluations must focus on the experience of and outcomes for the jobseeker.[73] For example, the Australian Unemployed Workers’ Union (AUWU) asserted that employment services should:
• deliver practical assistance to enable participants to find a job or improve their employability;
• have a focus on what each unemployed worker needs to get a job and not merely on the services that the provider wants to deliver;
• be fair—particularly if compulsory participation is to remain in place;
• ensure the integrity providers’ actions;
• empower jobseekers to provide feedback and be responsive to that feedback;
• enable building of rapport between jobseekers and providers (and others), noting that employment services are a human service; and
• impose only realistic expectations, taking into account jobseekers’ personal and environmental circumstances.[74]

Collaboration and sharing of best practice

8.39The Committee heard that the employment services system must enable the identification and sharing of best practice.[75] For example, Professor Jeff Borland stated:

By bringing to the fore ideas of learning and improvement, research can foster a culture for service providers of constantly coming back to the question of whether the service being delivered is working for jobseekers and of seeking improvement against current outcomes. Activities that seek to share knowledge on best practice are an important part of the influence that research can have.[76]

8.40Stakeholders raised concern that an overly competitive model for employment services is an impediment to service improvement, as providers are unable to share lessons or examples of success.[77] Stakeholders also noted that ad hoc fora established to share best practice are typically industry- or provider-led. This impacts on the willingness of market actors to participate due to heightened concerns about the sharing of intellectual property with potential competitors in future procurement processes.[78]

8.41Providers,[79] employers,[80] academics[81] and advocacy groups[82] all supported measures to enable the sharing of best practice. DEWR acknowledged that there would be value in implementing such measures.[83] In addition, there were calls to establish a Centre of Excellence to enable greater sharing within the sector.[84]

8.42Stakeholders also called on government to establish, support, and manage formal Communities of Practice, noting that government intervention may be needed to encourage provider engagement.[85] Stakeholders pointed to examples of successful Communities of Practice established with government support. For example, Yilabara Solutions drew attention to the Australian Housing and Urban Research Institute as an example of a community that brings together all levels of government, as well as an Australia-wide network of university partners.[86]

8.43Professor Mark Considine observed that a key challenge will be setting parameters for how Communities of Practice should operating noting that insights as to what is working must be rapidly identified and communicated to providers’ frontline staff.[87]

8.44The Victorian Chamber of Commerce and Industry (VCCI) stated that best practice must be viewed and celebrated through a lens of innovation, customer-centricity, and service improvement. The VCCI also emphasised that poor practice should be managed effectively and indicated that poor performance by one provider should not result in increased compliance activity for the sector as a whole.[88]

8.45The Committee also heard through its delegation trip, examples in countries like South Korea, where the use of artificial intelligence (AI) and machine learning is being applied to help to identify the optimal pathways for jobseekers to achieve outcomes.

Innovation and experimentation

8.46Several stakeholders indicated that the employment services system must enable and encourage providers to innovate and trial alternative services and expressed concern that the current system—and particularly its funding and performance management arrangements—stifles innovation and limits outcomes for jobseekers, employers, and other stakeholders.[89] MAX provided the following example:

[i]f we invest in developing bespoke interventions, like wellness programs or confidence building initiatives we must get approval from the department to use Employment Fund or fund this from our bottom line.[90]

8.47The BSL noted that providers currently bear almost all risk and cost associated with innovation—particularly if innovation results in short-term drop in performance. This is the case even if the initiative results in longer-term gains to service quality, performance, and outcomes. BSL asserted that government must share the financial and performance risks of trialling and implementing new approaches if the sector is to support innovation and continuous improvement.[91]

Framework to support trialling and experimentation

8.48Stakeholders expressed strong support for a framework which supports testing of new initiatives and delivery approaches, noting that under previous iterations of employment services there was greater scope for innovation.[92] MatchWorks and SYC Ltd (SYC) drew attention to the New Employment Services Trial (NEST), noting that the NEST included the necessary flexibility in terms of funding and performance management for providers to test new activities and delivery approaches. SYC also noted that frontline staff will generally be as innovative as program guidelines and performance measures allow.[93]

8.49NESA highlighted the Stream4 Demonstration Pilots under Job Services Australia, as well as the Empowering Youth Initiative under jobactive, as examples of models which allowed trialling of alternative service models.[94] The evaluation of Job Services Australia stated that the Demonstration Pilots were:

… a discretionary grants programme which … enabled the Department to work with leading providers to capture best practice in service delivery, explore new approaches and partnerships, with the aim of achieving improved employment and education outcomes for highly disadvantaged job seekers, including those with multiple barriers to employment.[95]

8.50Under the first Job Services Australia contract, there was also a dedicated Innovation Fund, with $41 million available to support innovative place-based solutions. Funding enabled providers to connect disadvantaged jobseekers to training and employment through trialling of alternative servicing approaches.[96]

8.51The Community Development Program (CDP) and Disability Employment Services (DES) have implemented frameworks to support innovation in service delivery. For example:

• Under Trialling Pathways to Real Jobs, the National Indigenous Australians Agency (NIAA) allows CDP providers to re-direct 25 per cent of their funding to trial new approaches which support First Nations communities find secure work.[97]
• The Department of Social Services (DSS) is trialling a Payment-by-Outcomes approach to funding innovative programs. One program, led by White Box Enterprises, involves funding 15 pilot Work Integrated Social Enterprises (WISEs) to employ up to 150 people with disability. Additional funding is available where a participant transitions into mainstream employment.[98]
  52. The PRF expressed support for ‘Innovation Zones’ which enable time-limited testing of new technologies, products, or approaches.[99] NESA noted that other countries—for example France and Belgium—have implemented zones or labs to support innovative labour market programs.[100]
  53. Internationally, there are a range of government initiatives that seek to support innovation in employment services. For example, the Committee had the opportunity to learn about ‘Le Lab’ while in France. Le Lab is a government-led project which aims to remove barriers to innovation inherent in traditional project management approaches by favouring co-design and the free exchange of ideas. The program operates out of central offices in Paris and across a regional network of sites across the country—including sites operated on an outreach basis.
  54. Stakeholders emphasised that measures must be implemented to reduce the risk to providers of trialling new approaches. For example, Dr Ann Nevile suggested that sites where a provider is trialling a service approach could be removed from formal performance assessment.[101] The Salvation Army Employment Plus (SAEP) similarly stated that DEWR should encourage innovation by giving preference to innovative providers during mid-term licence allocations and excluding trials of new initiatives from performance metrics.[102]

Committee comment

8.55Since the introduction of the quasi-market approach to procurement of service delivery, the Commonwealth has conducted massive multi-billion-dollar tenders for employment services every three to five years. The system design has then largely been subject to a ‘set and forget’ mentality, with infrequent formal evaluations of the service model and limited data sharing.

8.56As outlined in Chapter 2, one of the policy rationales for marketisation was to promote diversity and innovation. Yet the reality of the overly competitive quasi‑market that has emerged in Australia is that there is little diversity in the ways that providers delivery services to their clients. While there may have been some enabling of innovation in Job Services Australia, Workforce Australia has little in the way of an innovation or learning culture. The Commonwealth investment into employment services means it should be able understand practices that are of benefit to the hundreds of thousands of jobseekers and countless businesses that engage with providers. But a ‘black box’ approach to provider’s practices combined with a decades-long absence from direct engagement in local communities and service delivery means that the Australian Government does not have a view of what a quality service looks like or costs to deliver.

8.57The system is also overly focused on providers’ compliance, as highlighted in Chapter 7. Assurance and monitoring processes are focused on conditioning the behaviour of contracted providers and measuring ‘performance’ rather than assessing and enhancing the quality of services. The current regimented, transaction-based system stifles innovation and prohibits deviation from a departmentally determined path. This is compounded by a regulatory culture which make innovation a high-risk option for providers.

8.58The public unavailability of the very detailed and rich data on the employment services system held by government agencies, combined with a failure to make evaluations available in a timely manner and a lack of support for sharing best practice, has resulted in a hollowing out of research capacity and has limited the sector’s ability to continuously improve. Amuch greater systemic focus on the ongoing research and evaluation of service quality and effectiveness is needed.

8.59In its interim report, the Committee supported measures to make data on employment services publicly accessible. After considering additional evidence, the Committee remains firmly of the view that data must be made available—and accessible—to a greater range of stakeholders. The Committee notes that there has for decades been support for the release of de-identified data employment services, data. Further, the release of the Research and Evaluation Database indicates that DEWR has the ability to curate and manage the release of data for research and analysis.

8.60A principle of transparency by default should be adopted, whereby data is published as matter of course unless there is a compelling reason not to do so. The Australian Government should develop a comprehensive and structured program of data release to support research and enable evidence-based improvements to the sector. This should be finalised and published within 12–18 months of tabling of this report. The Committee would expect the first tranche of employment services data to also be released within this timeframe.

8.61The Committee recommends that the Australian Government:

• adopt a clear default principle of transparency whereby all data is published as matter of course unless there is a compelling reason not to do so; and
• implement a comprehensive and structured program of data release to support research capability and enable evidence-based improvements to the sector, with the program to include:

• identification of data assets—including administrative and survey data;
• a clear schedule for the release of data assets to the public;
• a comprehensive data protection framework;
• training and ongoing support in the use of the data for all officers;
• a service level requirement that administrative and survey data will be released no less than 12months after it is collected unless the data is specifically collected for the purposes of an evaluation; and
• a policy framework and IT system changes to support the integration of data from other sources including Australian Government, State and Territory Governments, and research institutions.

This program should be finalised and published within 12–18 months of the tabling of this report.

8.62The Committee recommends that the Australian Government implement a new evaluation framework for employment services, including the following key elements:

• A maximum 18-month period release date for evaluation reports following the end of the period analysed.
• Release of data used for the purposes of the evaluation for external scrutiny and analysis at the time the evaluation report is published.
• Inclusion of a broader range of metrics of effectiveness, including measures which assess the success of the system in building a jobseeker’s capacity for social and economic participation and meeting the needs of employers.
• Consideration of the satisfaction of service users including unemployed people and employers.
  63. As outlined in Chapter 2, excessive competition and a lack of collaboration and sharing of best practice has been a critical issue for employment services since outsourcing of service delivery and the adoption of a quasi-market model. There is compelling evidence to indicate that this concern has grown over time, with negative consequences including wastage, fragmentation, and lack of dispersion of new or techniques or strategies that work. In saying this, it must also be acknowledged that competition often drives innovation, and there is a trade-off.
  64. There is an enormous amount that Australia can and should learn from other jurisdictions in relation to learning and evaluation. Many countries with which the Committee engaged during the inquiry have far more structured approaches to research, learning, continuous improvement, and evaluation than Australia. This includes more routine use of trials, particularly RCTs, to test and establish an evidence based for changes and new initiatives.
  65. Building on the Committee’s conclusions in Chapter 7, the Committee considers that the Employment Services Quality Commission should have the lead responsibility to rebuild a culture across the system of collaborative research, learning, service, and quality improvement and sharing good practice. This should occur in partnership with external partners.
  66. A dedicated unit within the new Employment Services Quality Commission should facilitate co-operative research which brings together government, academics, researchers, and practitioners. Having practitioners (including providers and their staff) work within and in partnership with the commission would also ensure insights are directly translated to service-level improvements, build a clear picture of a ‘quality’ employment service, and provide guidance on quality service pricing.
  67. Key functions of the unit within the Employment Services Quality Commission should include:

• leading work to identify and share best practice with the sector, including through the establishment of Communities of Policy and Practice;
• identification and mapping of pathways to employment, including services that would best support jobseekers along those pathways and the costs associated with service delivery; and
• developing a research agenda and ethical framework for the sector, including measures to enable access to data to support research efforts and proposals for RCTs or pilots.
  68. The Committee was impressed by the approaches of the Centre for Local Innovation in Social and Employment Services in Denmark to drive improvements to the public employment services, as well as the activities undertaken by Le Lab in France. Those organisations may be used as models in establishing the new unit within the Employment Services Quality Commission.
  69. The Committee also notes that there has been a commitment made to establish a Disability Employment Centre of Excellence. Consistent with other recommendations to support a culture of learning and continuous improvement, there would be value in the functions of this Centre ultimately being subject to oversight by the Employment Services Quality Commission.
  70. DEWR would be closely engaged with the Quality Commission’s work as the enabler and active steward of the system; a provider of functions and services; the custodian of the bulk of system data; and the funder and commissioner of the service ecosystem. The Committee expects that DEWR needs to appropriately invest in ITinfrastructure, to support fully utilising the emerging techniques, such as data analytics, AI and machine learning.
  71. Complementing the new Quality Commission’s focus on learning, quality, and service improvement, DEWR would retain responsibility for service model and program level evaluations, both formal and informal. There is considerable room to enhance the framework for evaluations of employment services, including via the application of counterfactual evaluations—subject to appropriate ethical and methodological constraints—and consideration of a greater range of measures of service quality.
  72. Evaluations must have the built-in capacity to assess whether services are successful in building the capacity of jobseekers across a range of domains. This is consistent with a service model which delivers holistic support to unemployed people and enables social as well as economic participation. Evaluations should also assess the system’s ability to support employers. This will be critical in light of the demand-led measures the Committee recommends elsewhere in this report.
  73. Program evaluation is an important capability for the public service. The Committee welcomes work by government to build and strengthen this capability. DEWR should partner with key stakeholders including University researchers and the Australian Centre for Evaluation (a preferable approach to ad-hoc commissioning of large commercial consultancy led evaluations), supporting a culture of peer-review. Greater transparency in evaluations builds confidence in results, builds research capacity and a learning culture in the public sector, and encourages government to explore additional cooperative and partnership-based evaluation models.
  74. Noting the evidence regarding gaps in the evaluation framework, the Committee is of the view that government should implement a new evaluation framework. This is likely to require revisions to the current Evaluation Strategy for Workforce Australia Employment Services 2022–2028. The framework appears narrow and largely focused on employment outcomes and cost effectiveness. While these are primary goals, other considerations must form part of a more holistic approach to evaluation if the system is to move beyond the failed work first focus. Cost effectiveness should not be seen as a proxy for endless cost reductions which bear no relationship to ensuring a quality service. A focus on employment is also reflected in evaluation questions relating to the payment system and provider performance framework, which ask whether those elements of the employment services system are effective in driving employment outcomes.

8.75The Committee recommends that the Australian Government establish a unit within the new Employment Services Quality Commission responsible for continuous research and learning regarding service delivery and quality improvement and sharing of good practice. The unit should undertake its work with the Department of Employment and Workplace Relations, service partners, clients and external research partners including Universities and the Australian Centre for Evaluation, with its functions to include:

• leading work to identify and share best practice in the design and delivery of employment services;
• identifying and mapping pathways to employment, including services that would best support jobseekers along those pathways and the costs associated with service delivery; and
• developing an ongoing collaborative research agenda for the sector, including measures to enable greater access to data to support research efforts.
  76. The Committee is also concerned that providers’ access to data they have recorded in the system is restricted, noting that this may limit—or at least delay—a provider’s ability to deliver effective employment supports. Lack of access to this data also adds to administrative burden, particularly if providers are obliged to maintain ‘shadow’ systems which result in the duplication of work.
  77. Providers should have real- or near real-time access to data they record in systems that they are required to use, in the form that meets the provider’s business needs. This is in order to improve services to clients and reduce administrative burden by forcing providers to maintain data in duplicate ‘shadow’ administrative systems for accountability and compliance purposes.

8.78The Committee recommends that the Australian Government ensure all service partners can access information they enter into the department’s IT systems unless there are clear, defendable reasons for not making that data available.

8.79Existing contractual arrangements severely limit opportunities to innovate, including by discouraging providers from trialling new initiatives. Employment services must have the in-built capacity to trial and evaluate new service offerings. This is critical to identifying and sharing best practice, responding to changing labour markets, and enabling continuous improvement.

8.80Trials and pilots were previously supported under the NEST and in earlier iterations of employment services. This indicates that innovation can form part of mainstream employment services assuming policy settings do not constrain new initiatives.

8.81Measures such as removing sites used to trial service models from performance evaluations, and the implementation of dedicated sandboxes or innovation zones to enable testing of new approaches in a low-risk environment should be implemented to reduce barriers to innovation. Further, rewarding those who do innovate through giving preference during mid-term licence allocations warrants consideration.

8.82Both NIAA and DSS have implemented measures to support trialling of initiatives in the CDP and DES programs. The government should actively monitor those trials to identify lessons and examples of best practice which may support reforms to the mainstream employment services system.

8.83The Committee recommends that the Australian Government ensure a rebuilt Commonwealth Employment Services System permanently design in the capacity to experiment with innovative service delivery approaches. Trials should be subject to rigorous co‑design with stakeholders including participants, providers, peak bodies, and academic and policy experts, and should be subject to robust monitoring and evaluation. ‘Innovation zones’ should be considered along with exclusion of approved trials of new initiatives from performance metrics so as not to discourage or punish providers and services which try new approaches.

8.84In reaching its conclusions in this chapter, the Committee acknowledges there is some degree of inherent overlap between the functions proposed for the Employment Services Quality Commission and DEWR. If the Australian Government adopts these broad recommendations, then more detailed delineation of these functions can be worked out by DEWR. Wherever the lines are drawn matters far less than the important of establishing a positive culture of close collaboration between DEWR as data custodian, active system steward and enabler of the service ecosystem and the Employment Services Quality Commission as procurer and funder of services. Similarly, if the proposed Commission is established, minimising duplication with other regulators should be a design principle.

Footnotes

[1]Brotherhood of St Laurence (BSL), Centre for Policy Development (CPD) and University of Melbourne (UniMelb), Submission 256, p. 33,

[2]See, for example, National Foundation for Australian Women (NFAW), Submission 135, p. [3]; Workways Australia (Workways), Submission 239, p. 13; Per Capita, Submission 252, pages 63–64; National Employment Services Association (NESA), Submission 260, p. 92.

[3]Social Ventures Australia (SVA), Submission 232, p. 21.

[4]CVGT Employment (CVGT), Submission 106, p. 27.

[5]See Department of Employment and Workplace Relations (DEWR), Employment Services Data, www.dewr.gov.au/employment-services-data, viewed 20 November2023.

[6]Available as Additional Documents 1-5 on the Committee’s website.

[7]Commonwealth of Australia (2023), Working Future: The Australian Government's White Paper on Jobs and Opportunities, pages 182, 186.

[8]Australian Bureau of Statistics (ABS), Submission 68, p. 2.

[9]Commonwealth of Australia (2023), Working Future: The Australian Government's White Paper on Jobs and Opportunities, p. 235.

[10]See, for example, WISE Employment, Submission 169, pages 23–24; The Salvation Army Employment Plus (SAEP), Submission 199, p. 23.

[11]See, for example, CVGT, Submission 106, p. 20; NFAW, Submission 135, p. [3].

[12]See, for example, Sarina Russo Job Access (SRJA), Submission 145, p. 11; CoAct, Submission 151, p. 11; MrConKittos, Executive Chairman, Asuria People Services (Asuria), Committee Hansard, 26May2023, p.20.

[13]See, for example, CVGT, Submission 106, p. 27; Mission Australia, Submission 190, p. 11.

[14]NFAW, Submission 135, p. [6].

[15]WDEA Works, Submission 168, p. 5.

[16]Workways, Submission 239, p. 12.

[17]Dr May Lam, Senior Research Fellow, Per Capita, Committee Hansard, 19 September 2023, pages 22, 24.

[18]Per Capita, Submission 252.1, p. 4.

[19]See, for example, Professor Leila Green, Dr Kylie Stevenson, Dr Kelly Jaunzems, Ms Claire Hanlon and MrArthur Hanlon (Professor Green et al), Submission 120, p. [12]; Yilabara Solutions, Submission231, p.27.

[20]See, for example, CoAct, Submission 151, p. 7; Joblink Plus, Submission 157, p. 20; SYC Ltd (SYC), Submission 189, p. 19; MatchWorks, Submission 263, p. 41.

[21]Ms Annette Gill, Principal Policy Advisor, NESA, Committee Hansard, 14March2023, p. 37. In their submission, NESA also covered this issue. See NESA, Submission 260, p. 57.

[22]CVGT, Submission 106, p. 27.

[23]Productivity Commission (2002), Independent review of the Job Network, Inquiry Report no. 21, p. 5.32.

[24]Senator the Hon Ian Campbell, Parliamentary Secretary to the Treasurer (2019), Government Response to the Productivity Commission Independent Review of Job Network, https://ministers.treasury.gov.au/sites/ ministers.treasury.gov.au/files/2019-09/JobNetwork.pdf, viewed 20 November2023.

[25]Productivity Commission (2002), Independent review of the Job Network, pages 5.30–5.31.

[26]DEWR, Submission 254, pages 126–127.

[27]DEWR, Submission 254, p. 127.

[28]DEWR, Submission 254, p. 126.

[29]See Office of the National Data Commissioner, Data Availability and Transparency Act 2022, www.datacommissioner.gov.au/law/dat-act, viewed 20 November 2023.

[30]Yilabara Solutions, Submission 231, p. 27.

[31]Mr Bryan McCormack, Senior Advisor—Government and Policy, Jobs Australia, Committee Hansard, 20September 2023, p. 20.

[32]See, for example, Per Capita, Submission 252, p. 63; DEWR, Submission 254, p. 127.

[33]See, for example, IntoWork Australia (IntoWork), Submission 156, p. 6; WISE Employment, Submission 169, pages 23–24; SYC, Submission 189, p. 19; SSI, Submission 193, pages 17–18; SAEP, Submission 199, p.70; Yilabara Solutions, Submission 231, p.27; SVA, Submission 232, pages.21, 23.

[34]Government of Victoria, Submission 278, p. 20.

[35]Commonwealth of Australia (2023), Working Future: The Australian Government's White Paper on Jobs and Opportunities, p. 149.

[36]Mr David Turvey, First Assistant Secretary, Jobs and Skills Australia (JSA), Committee Hansard, 20 September 2023, p. 60.

[37]Disability Royal Commission (2023), Final Report: Volume 12: Beyond the Royal Commission, p. 28.

[38]DEWR, Submission 254, p. 125.

[39]WISE Employment, Submission 169, p. 23.

[40]DEWR (2002), Job Network Evaluation: Stage Three, p. 80, www.dewr.gov.au/employment-services-evaluations/job-network-evaluation, viewed 20 November 2023.

[41]DEWR (2022), The Evaluation of jobactive Final Report, pages 185, 237, 242, 319, www.dewr.gov.au/employment-research-and-evaluations/resources/evaluation-jobactive-final-report, viewed20 November2023.

[42]Organisation for Economic Co-Operation and Development (OECD), Submission 177, p. 3.

[43]Committee for Economic Development of Australia (2023), We Need More Randomised Trials to Evaluate Public Policy, Opinion Article, www.ceda.com.au/NewsAndResources/Opinion/Economy/We-need-more-randomised-trials-to-evaluate-public?s=03, viewed 20 November2023.

[44]Mr Ben Gales, Chief Delivery Officer, Paul Ramsey Foundation (PRF), Committee Hansard, 11August2023, p.29.

[45]See, for example, OECD, Submission 177, p. 3. See also OECD (2022), Assessing Canada’s System of Impact Evaluation of Active Labour Market Policies, Connecting People with Jobs, pages9–10, www.oecd.org/publications/assessing-canada-s-system-of-impact-evaluation-of-active-labour-market-policies-27dfbd5f-en.htm, viewed 20 November 2023.

[46]The Hon Dr Andrew Leigh MP, Assistant Minister for Employment, ‘Running Better Evaluations’, Daily Telegraph, 7February2023; The Hon Dr Andrew Leigh MP, Assistant Minister for Employment, Australian Centre for Evaluation to measure what works, Media Release, 25 May 2023.

[47]Commonwealth of Australia (2023), Working Future: The Australian Government's White Paper on Jobs and Opportunities, p. 238.

[48]The Hon Dr Andrew Leigh MP, Assistant Minister for Employment, First evaluation partnership for the Australian Centre for Evaluation, Media Release, 11 October 2023.

[49]Commonwealth of Australia (2023), Working Future: The Australian Government's White Paper on Jobs and Opportunities, pages 224, 235.

[50]Behavioural Insights Team (2018), Applying Behavioural Insights toLabour Markets, p. 25, www.bi.team/wp-content/uploads/2018/11/TheBehaviouralInsightsTeam-LabourMarketsReport.pdf, viewed 20 November 2023.

[51]Independent Panel of the APS Review (2019), Our Public Service Our Future: Independent review of the Australian Public Service, pages 221–222, www.pmc.gov.au/resources/independent-review-australian-public-service, viewed 20 November 2023.

[52]See, for example, CVGT, Submission 106, p. 28; Workskil Australia (Workskil), Submission 196, p. 33; APM, Submission 213, p. 21; MatchWorks, Submission 263, p.41.

[53]WISE Employment, Submission 169, p. [23].

[54]OECD, Submission 177, p. 3. See also OECD (2022), Assessing Canada’s System of Impact Evaluation of Active Labour Market Policies, Connecting People with Jobs, pages9–10.

[55]See, for example, Professor Green et al, Submission 120, p. [12]; Joblink Plus, Submission 157, p. 20; Getting Welfare to Work Research Team, Submission 191, p. 6; PerCapita, Submission 252, p.14.

[56]Getting Welfare to Work Research Team, Submission 191, p. 6; Per Capita, Submission 252, p. 66.

[57]Getting Welfare to Work Research Team, Submission 191, p. 6.

[58]Dr Ann Nevile, Submission 136, pages 2-3. Austria’s approach to learning and evaluation was discussed in bilateral meetings between the Committee and Austria during the Committee’s European delegation.

[59]Campbell Page, Submission 150, p. [7].

[60]Getting Welfare to Work Research Team, Submission 191, p. 5.

[61]See, for example, VERTO, Submission 202, p. [23]; Yilabara Solutions, Submission 231, p. 26; SVA, Submission 232, p.17; Per Capita, Submission 252, p. 63.

[62]SRJA, Submission 145, p. 6.

[63]Ms Annabel Brown, Deputy CEO, CPD, Committee Hansard, 20 September 2023, p. 2.

[64]DEWR (2023), Evaluation Strategy for Workforce Australia Employment Services 2022–2028, p. 29, www.dewr.gov.au/employment-services-evaluations/resources/evaluation-strategy-workforce-australia-employment-services-2022-2028, viewed 20 November2023.

[65]See DEWR (2023), Evaluation Strategy for Workforce Australia Employment Services 2022–2028, p. 29.

[66]See, for example, Professor Green et al, Submission 120, p. [13]; WISE Employment, Submission 169, p. 5.

[67]Australian Centre for Career Education (ACCE), Submission 149, p. 2.

[68]See, for example, SRJA, Submission 145, p. 22; IntoWork, Submission 156, p. 2; Professor Jeff Borland, Submission 171, p. 2; SAEP, Submission 199, p. 65; Yilabara Solutions, Submission 231, p. 26.

[69]See, for example, IntoWork, Submission 156, p. 2; SVA, Submission 232, p.21.

[70]IntoWork, Submission 156, p. 2.

[71]See, for example, WISE Employment, Submission 169, p. 5; SAEP, Submission 199, p. 65; SVA, Submission 232, pages 17, 21.

[72]Yilabara Solutions, Submission 231, p. 26.

[73]ACCE, Submission 149, p. 5.

[74]Australian Unemployed Workers' Union (AUWU), Submission 253, p. [10].

[75]See, for example, ACCE, Submission 149, p. 13; Campbell Page, Submission 150, p. [7]; Training Alliance Group (TAG), Submission 195, p. [2]; SAEP, Submission 199, p. 14; Per Capita, Submission 252, p. 62.

[76]Professor Borland, Submission 171, p. 15.

[77]See, for example, CVGT, Submission 106, p. 28; CoAct, Submission 151, p. 9; SVA, Submission232, p. 20; BSL, Submission 249, p.18; Dr Travers McLeod, Executive Director, BSL, Committee Hansard, 20 September 2023, p. 6; Ms Kathryn Mandla, CEO, NESA, Committee Hansard, 20September 2023, p. 21.

[78]See, for example, Asuria, Submission 246, p. 26; NFAW, Submission 135, p. [7].

[79]See, for example, Campbell Page, Submission 150, p. [7]; Joblink Plus, Submission 157, p. 7; WISE Employment, Submission 169, p. 10; TAG, Submission 195, p. [2]; Yilabara Solutions, Submission 231, p. 26; Asuria, Submission 246, p. 26.

[80]Victorian Chamber of Commerce and Industry (VCCI), Submission 259, p. 6.

[81]See, for example, Dr Nevile, Submission 136, pages 4–5; Professor Borland, Submission 171, p. 15.

[82]See, for example, SVA, Submission 232, p. 20; BSL, Submission 249, p. 77.

[83]DEWR, Submission 254, p. 82.

[84]See, for example, Campbell Page, Submission 150, p. [7]; Jobs Australia, Submission 185, p. 23.

[85]See, for example, Greater South East Melbourne (GSEM), Submission 155, p.[7]; TAG, Submission 195, p.[2]; Asuria, Submission 246, p.26; Per Capita, Submission 252, p. 12.

[86]Yilabara Solutions, Submission 231, p. 26.

[87]Professor Mark Considine, private capacity, Committee Hansard, 14 March 2023, p. 14.

[88]Victorian Chamber of Commerce and Industry (VCCI), Submission 259, p. 6.

[89]See, for example, NFAW, Submission 135, p. [10]; MAX Solutions (MAX), Submission 146, p. 11; ACCE, Submission149, p. 12; WISE Employment, Submission 169, p. 10; The BUSY Group (BUSY), Submission 227, p. 2; Submission 249, p. 18; Per Capita, Submission 252, p. 62.

[90]MAX, Submission 146, p. 30.

[91]BSL, Submission 249, pages 12, 76–77. See also, Per Capita, Submission 252, p. 12; BSL, CDP and UniMelb, Submission 256, pages 7, 19–20.

[92]See, for example, CVGT, Submission 106, p. 26; WISE Employment, Submission 169, pages11, 13, 23, 25; Workskil, Submission 196, p.33; Centre for Policy Futures (CPF–UQ), Submission 217, pages 4–5; PerCapita, Submission 252, p. 11; Community and Public Sector Union (CPSU), Submission 255, p.[14].

[93]SYC, Submission 189, p. 19; MatchWorks, Submission 263, pages 18, 23.

[94]NESA, Submission 260, pages 26, 38, 92.

[95]Department of Employment, Skills, Small and Family Business (2020), The Evaluation of Job Services Australia 2009–2012, p. 96 (footnote 215), www.dewr.gov.au/employment-research-and-evaluations/resources/evaluation-job-services-australia-2009-2012-report, viewed 20 November 2023.

[96]Department of Education, Employment and Workplace Relations (2009), Portfolio Budget Statement 2009–10: Budget Related Paper No. 1.6, p.134.

[97]See, for example, National Indigenous Australian Agency (NIAA), Submission 176, p. 4; Ms Julie-Ann Guivarra, Deputy CEO—Policy and Programs, NIAA, Committee Hansard, 17May2023, p.13.

[98]White Box Enterprises, Submission 274, p. [3].

[99]PRF, Submission 304, p.16.

[100]NESA, Submission 260, p. 92.

[101]Dr Nevile, Submission 136, p.12.

[102]SAEP, Submission 199, p. 37.