Chapter 2 Audit Report No.41 2011–12 National Partnership Agreement on Literacy and Numeracy

Introduction

2.1                   The National Partnership Agreement on Literacy and Numeracy (LNNP) was one of three ‘Smarter Schools’ National Partnerships announced in the 2008–09 Budget.[1] Commencing in 2009, the LNNP was one of the first National Partnerships operating under the Intergovernmental Agreement on Federal Financial Relations (IGA FFR), and one of the first to include reward payments to states and territories.[2]

2.2                   The LNNP was envisaged as a tool to:

… galvanise the collective resources and energy of the Australian Government and the state, territory and non-government education systems, to put in place the infrastructure and practices that will deliver sustained improvement in literacy and numeracy outcomes for all students, especially those who are falling behind.[3]

2.3                   The LNNP also aimed to ‘accelerate progress towards the ambitious literacy and numeracy target’ set by the Council of Australian Governments (COAG) to ‘halve the gap for Indigenous students in reading, writing and numeracy within a decade’.[4]

2.4                   The LNNP committed a total of $540 million of Commonwealth Government funding over four years,[5] comprising the following:

• $150 million in ‘facilitation payments’ to states and territories to support literacy and numeracy reform activities over the first two years of the partnership (2009 and 2010). These payments were to be contingent on equivalent ‘co-investments’ from states,[6] including existing or redirected funds.[7]
  
  
• $350 million in ‘reward payments’ to states based on the achievement of agreed literacy and numeracy targets over the last two years of the partnership (2011 and 2012). Targets were to be ‘ambitious’ and to ‘aim for accelerated improvement’ for schools and students involved in the partnership, but would reflect the ‘different starting points in each state’.[8]
  
  
• $40 million for research initiatives targeted at improving teaching capacity in literacy and numeracy.[9]

2.5                   In addition to the research initiatives, an agreed outcome of the LNNP was the Commonwealth’s management of a database of effective strategies and approaches known as a ‘framework for effective practice’ or ‘Evidence Base’. The Evidence Base was expected to encourage sharing of information and effective practice, leading to better informed and evidence-based decisions on literacy and numeracy. The Evidence Base was expected to be available by early 2009.[10]

2.6                   The Department of Education, Employment and Workplace Relations (DEEWR) coordinated the implementation of the LNNP. This included negotiating bilateral agreements and implementation plans with state government agencies. As part of this process, DEEWR also negotiated the reform targets that would be the basis for reward payments to states.[11]

2.7                   Approximately 1050 government and non-government schools received LNNP assistance, accounting for around 13 per cent of Australia’s student population and around 14 per cent of Indigenous and low-performing students.[12]

2.8                   In May 2012, the government committed to a $243 million extension to the LNNP in the form of a new agreement that will expire in December 2013.[13] At the time of writing, details of the extended partnership and its implementation plans were not publicly available.

The ANAO audit

Audit objective and scope[14]

2.9                   The objective of the Australian National Audit Office (ANAO)’s audit was to assess the effectiveness of DEEWR’s administration of the LNNP. The three high-level audit criteria used to form a conclusion examined the extent to which DEEWR:

• established sound administrative and payment arrangements consistent with government policy, including through its negotiation of bilateral agreements, implementation plans and reform targets;
  
  
• properly managed administrative and payment arrangements; and
  
  
• effectively monitored and reported on delivery and outcomes.[15]

2.10               The audit report also included analysis of changes in National Assessment Program – Literacy and Numeracy (NAPLAN) test results for participating schools; presented case studies of literacy and numeracy initiatives; and examined national trends in literacy and numeracy performance since the commencement of the LNNP.

2.11               In performing the audit, the ANAO consulted with education authorities in four states and other relevant stakeholders, including the Australian Council for Educational Research; the Australian Curriculum, Assessment and Reporting Authority; the COAG Reform Council; the Commonwealth Treasury; and several schools that received LNNP funding. The audit covered the LNNP’s operation from when it was signed in 2009 until March 2012.

Overall audit conclusion

2.12               The ANAO reported that, through the partnership, education authorities and schools had implemented a range of initiatives in the agreed reform areas, with positive impacts on schools, teachers and students.[16]

2.13               Despite these positive impacts, ANAO analysis of NAPLAN data found no statistically significant improvement in any state on the average results of schools receiving LNNP funding when compared to schools that did not receive funding.[17] This lack of progress was at least partially attributed to the short amount of time between the commencement of LNNP activities and NAPLAN testing, with the ANAO suggesting it may take several years until a ‘reliable assessment of the impact of the LNNP’ could be made.[18]

2.14               The ANAO concluded that ‘overall, the effectiveness of DEEWR’s administration of the LNNP has been mixed’. This was in the context of National Partnerships being a new form of program delivery and the LNNP being one of the first to include reward payments to states.[19]

2.15               The report noted that while DEEWR worked collaboratively with state agencies, the department ‘did not apply a structured approach to negotiating key implementation arrangements’, including the number of participating schools, performance indicators, and reform targets. This resulted in ‘significant variability at a state level in the coverage of the LNNP and performance indicators used, and reward targets were not necessarily demanding’.[20]

2.16               Additionally, the ANAO found that DEEWR did not progress the Evidence Base of effective literacy and numeracy strategies as promptly as envisaged under the partnership agreement, limiting the guidance available to education authorities and schools implementing LNNP programs.[21]

ANAO recommendations

2.17               The ANAO made two recommendations aimed at strengthening program and payment design for future National Partnerships, and better assessing the partnership’s impact on literacy and numeracy outcomes for participating schools following the conclusion of the LNNP.[22]

Table 2.1 ANAO recommendations, Audit Report No.41 2011–12

1.	To inform program and payment design that provides strong performance incentives for future National Partnerships, the ANAO recommends that DEEWR review the approach taken in establishing reform targets for the LNNP as the basis of reward payments, to draw on opportunities for improvement. DEEWR Response: Agreed.
2.	To assess the impact of the LNNP and different literacy and numeracy strategies, the ANAO recommends that DEEWR analyse the literacy and numeracy outcomes of participating schools at an appropriate stage following the conclusion of the National Partnership, as part of the national evaluation of the Smarter Schools National Partnerships. DEEWR Response: Agreed

2.18               The audit report also highlighted a range of broader lessons for the establishment of National Partnership payment arrangements by responsible agencies, including that National Partnerships should be designed to:

… allow sufficient time for initiatives funded using facilitation payments to significantly influence performance results, prior to measuring achievement against reform targets and making reward payments;

and

… provide the best opportunity for achievement of intended outcomes, such as by linking accessible reward funding with the coverage of planned initiatives and targeted levels of improvement.[23]

The Committee’s review

2.19               The Committee held a public hearing on Wednesday 12 September 2012 with representatives of the following organisations:

• The Australian National Audit Office
  
• The Department of Education, Employment and Workplace Relations.
  
  

2.20               The Committee took evidence on the following issues:

• Administrative strengths and improvements over time
  
• Setting of reform targets
  
• Reward payments
  
• Co-investments by states
  
• Impact of the LNNP
  
• Evidence Base
  
• Ongoing evaluation
  
• New LNNP agreement.

Administrative strengths and improvements over time

2.21               At the public hearing, DEEWR emphasised the strengths that the audit found in elements of the department’s implementation of the LNNP, including sound relationship management mechanisms and formal multilateral governance arrangements.[24]

2.22               DEEWR attributed these strengths to the National Partnerships Implementation Working Group, which was established by federal and state education ministers as an oversight body to steer the implementation of partnership agreements.[25]

2.23               DEEWR also highlighted the audit report’s finding that the mixed effectiveness of its administration was in part due to DEEWR not having access to formal guidance when developing the LNNP framework. This was in the context of the LNNP being one of the ‘earliest’ National Partnerships and one of the first to include reward payments.[26]

2.24               DEEWR pointed to improvements that were made during the LNNP’s implementation period to the process for assessing reward targets, and noted that lessons from the LNNP had helped inform guidance from central agencies.[27]

2.25               The department indicated that it accepted the ANAO’s recommendations and has applied the lessons learned from the audit in its subsequent negotiation of a new literacy and numeracy partnership, and in developing the next phase of its evaluation of the LNNP.[28] The Auditor-General agreed that the department had ‘responded positively to the report’.[29]

Setting of reform targets

2.26               As noted above, the LNNP agreement stipulated that the reform targets negotiated with states would be ‘ambitious’ and would ‘aim for accelerated improvement’ in literacy and numeracy outcomes, while reflecting ‘the different starting points’ of each state.[30] The initial targets were set by states based on 2008 NAPLAN data for reading and numeracy, supplemented by a range of local measures.[31] Targets were submitted by DEEWR to the Australian Council for Educational Research (ACER) for independent advice on their level of ambition.[32]

2.27               The audit found that ACER had used its ‘professional judgement’ to assess the ambition of the initial reform targets, and that DEEWR ‘could have applied a more rigorous approach to assess ambition’.[33] At the public hearing, the ANAO explained that it was looking for more structure or methodology in how ambition was assessed,[34] and added:

… it is a balancing act for the department. On the one hand, with the national partnership you are trying to provide flexibility to the states to develop the right implementation arrangements. On the other hand, in this national partnership we are looking for ambitious, accelerated improvements … We simply raise the point in terms of factors like the number of participating schools and students and the levels of improvement, we think the scale could have been tipped a little more towards the ambition side in providing that flexibility.[35]

2.28               The COAG Reform Council (CRC) had responsibility for independently assessing whether reform targets had been met before payments were made to states.[36] The CRC’s performance report for 2010 noted wide variations in state reward frameworks and the level of ambition of targets, and made a range of recommendations for improving the performance reporting framework.[37]

2.29               When questioned about the negative findings on the level of ambition of the 2010 reform targets, DEEWR pointed out the difficulty that it faced in setting targets based on a ‘very scant’ evidence base. The initial targets were made against only one year of NAPLAN data (for 2008), meaning no existing trends were visible. DEEWR explained that this was likely to have led to a ‘slightly conservative bias’ on the part of states.[38] However, during the setting of targets for 2011, three years of NAPLAN data were available, meaning the department could apply a more structured approach that was ‘far more transparent and robust’.[39]

2.30               The ANAO agreed with the department’s position that during the setting of targets for 2011 the process had been improved.[40]

Reward payments

Timing of payments

2.31               The audit report noted the limited time allowed under the LNNP agreement for states to demonstrate improvements in literacy and numeracy prior to reward payments being made. The final bilateral agreement was signed in February 2010, and in some cases state-level planning prior to implementation was only completed at the end of 2010. The first LNNP reward payments were based on the results of the May 2010 NAPLAN tests. The ANAO commented that any improvements measured were therefore ‘unlikely to be significantly influenced by LNNP activities’.[41]

2.32               At the hearing, the Auditor-General acknowledged that in administering the LNNP  the department was acting within the particular framework established by government, but emphasised that more time needed to be allowed for measures to be implemented before performance was assessed.[42]

Funding withheld

2.33               Of the $350 million available in reward funding to states, the Committee heard that $64 million was withheld from the final payments in June 2012 due to targets not being met.[43]

2.34               The Auditor-General noted that it was ‘encouraging’ that where targets were not met, funds were being withheld, commenting that ‘this is the system working as intended’.[44]

2.35               At the request of the Committee, DEEWR provided a state by state breakdown of the reward payments that were withheld. The data revealed that the largest amount of unspent reward funding was for New South Wales, which received only 26.67 per cent of its allocated funds. The best performing jurisdictions were Victoria and the Australian Capital Territory, which each received more than 90 per cent of the allocated reward funding.[45]

2.36               There was also wide variability between jurisdictions in both the performance against and the weighting allocated to Indigenous reading and writing targets. For example, information provided by DEEWR showed that Queensland met or exceeded two of its four Indigenous targets, and made significant progress on the other two. These Indigenous targets had been weighted to 31.5 per cent of the total performance targets. On the other hand, New South Wales did not make progress on any of its Indigenous targets, but these targets only accounted for 10 per cent of the total.[46]

2.37               The Committee noted the May 2012 media announcement that the majority of the LNNP’s unallocated Commonwealth reward funding would be re-directed to the ‘Focus Schools’ program to improve Indigenous educational outcomes.[47]

Quality of state data

2.38               A criticism in the audit report was that neither DEEWR nor the CRC had access to the data or methodological information required to verify the accuracy of performance results provided by states, risking inaccuracies that ‘may lead to corresponding inaccuracies in the allocation of reward funding’. Despite having access to a more detailed NAPLAN dataset than that which had been provided to DEEWR, the ANAO was unable to verify the states’ performance data for either 2010 or 2011, as only limited information was available on the methodologies used to calculate the reported results.[48]

2.39               The report emphasised the importance of verifying or assuring the accuracy of results to ensure that public money is spent appropriately, and suggested that for future National Partnerships, ‘administering agencies would benefit from working with states to coordinate preparation of performance results and to consider related assurance processes’.[49]

2.40               The ANAO explained at the hearing that DEEWR had only received aggregated data from the states, and reiterated that better quality assurance mechanisms were needed:

… a department like DEEWR needs to be better positioned once it receives the performance data to be able to interrogate it and check that it is accurate. That requires the school level data and a clear understanding of the methodologies used to calculate the performance results.[50]

2.41               The Committee was informed that individual states had negotiated their own reporting methodology within their implementation plans.[51]

2.42               DEEWR was asked about whether there was scope for the Commonwealth to increase its visibility over the spending of reward funding by states to ensure appropriate targeting. The department advised that its agreements with state education authorities stipulated that funds must be directed to educational outcomes, but that it had no further visibility or control over spending.[52] One of the ‘end-points’ of the LNNP, however, was to develop an evidence base on improving literacy and numeracy (see below) which would be informed by the LNNP funded activities of state education authorities.[53]

Co-investments by states

2.43               As noted earlier, the LNNP agreement stipulated that states were required to match the Commonwealth’s facilitation payments. The agreement specified that the first facilitation payments would be triggered by the negotiation of bilateral agreements and implementation plans, and that bilateral agreements would include ‘the monitoring and reporting  arrangements’ to track state co-investments.[54] Facilitation and reward payments would only be made once the Commonwealth Treasurer received advice that states had ‘complied in full with their earlier funding (co-investment) obligations’.[55]

2.44               The ANAO reported that co-investment obligations by states had not been monitored by DEEWR. The audit found that the first facilitation payments had been approved by the Minister for School Education prior to the signing of bilateral agreements, ‘so that the implementation of the LNNP was not further delayed’.[56] In the bilateral agreements, once signed, all states agreed to match or exceed the Commonwealth’s payments, however, only four states agreed to include co-investment information in their annual reports to DEEWR. No states actually reported this information.[57]

2.45               The audit also found that DEEWR’s Chief Financial Officer had made certifications to Treasury for facilitation and reward payments to be made that did not make reference to co-investments.[58] Treasury had access to data on co-investments for the relevant financial years through its role in acquitting co-investment reports on behalf of the Standing Council on Federal Financial Relations, however, it had not obtained agreement to share its data with relevant Commonwealth agencies.[59]

2.46               When questioned, DEEWR referred to the matter as a ‘gap in the process’, and explained to the Committee:

We asked Treasury for the reports, because these reports are provided to Treasury, not to DEEWR. Treasury asked permission to provide them to DEEWR. That was not given. Therefore we did not have it in place.[60]

2.47               DEEWR further informed the Committee that subsequent to the LNNP, there has been agreement that co-investment requirements would no longer be included in future national partnerships, as such requirements are ‘input controls’ (and therefore at odds with an IGA FFR principle).[61]

Impact of the LNNP

2.48               It an opening statement to the Committee, DEEWR highlighted the LNNP’s overall positive impacts:

DEEWR is particularly proud of the fact that the literacy and numeracy national partnership is making a real and positive difference in our schools, particularly for the lowest-achieving students. It is pleasing to see that Indigenous students have shown the biggest gains in reading and numeracy over the past four years. LNNP reforms have contributed to a changed culture in schools through a focus on quality teaching, leadership, transparency and parental engagement.[62]

2.49               The CRC’s performance report for 2011, released in May 2012, found that schools participating in the LNNP generally improved their results in reading and numeracy.[63]

2.50               In contrast, the ANAO’s analysis of NAPLAN results found that when comparing LNNP schools to non-LNNP schools, the partnership was ‘yet to make a statistically significant improvement’ in any state. This finding was in the context that it may still have been ‘too early for such impacts to be clearly evident’ (as noted in the above discussion on the timing of reward payments).[64]

2.51               Asked about these findings, DEEWR explained that the ANAO’s analysis was ‘testing around a mean’, whereas the department was also looking at ‘other measures’ in which there had been significant improvements.[65] The department explained that the LNNP was intended to focus on students that were falling behind, and that NAPLAN results have shown a reduction in the number of students below the national minimum standard—to ‘decrease the tail’.[66]

2.52               In particular, DEEWR noted that the proportion of Indigenous Year 3 students at or below the minimum standard for reading had decreased from 53 per cent in 2008 to 46.1 per cent in 2011.[67]

2.53               Additionally, DEEWR noted strong support for the program amongst school communities and principals, and told the Committee that when assessing the success of the LNNP more outcomes than just NAPLAN results need to be considered:

The report acknowledges that significant achievement has been made against the other key outcomes of the LNNP: increased collaboration between schools and systems in achieving literacy and numeracy reform, improved classroom practice in literacy and numeracy, and positive impact on school leadership, teacher practice and student engagement.[68]

Evidence Base

2.54               DEEWR informed the Committee that the literacy and numeracy Evidence Base, a key output of the LNNP agreement, had been launched in June 2012 in the form of a website known as the Teach, Learn, Share database. DEEWR explained that the database was being used to ‘share the success of the LNNP’ by presenting a ‘body of evidence’ developed by state and territory education authorities over the four years of the partnership.[69]

Ongoing evaluation

2.55               The audit report noted that DEEWR had begun a national evaluation of the three Smarter Schools National Partnerships, with the first phase having been completed. The ANAO suggested that ‘given the complexities in measuring the effectiveness of reform activities, it may take several years until a reliable assessment of the LNNP approach can be made’.[70] The report recommended that DEEWR analyse the literacy and numeracy outcomes of participating schools ‘at an appropriate stage’ following the LNNP’s conclusion.[71]

2.56               At the public hearing, the ANAO further explained that although there were some ‘positive signs’, it was ‘still too early to tell at this stage the overall impact’.[72]

2.57               DEEWR agreed with the recommendation, and undertook to continue monitoring the LNNP’s impact with further analysis ‘for many years’.[73] The department said it would use the findings of the audit report to inform the second phase of the LNNP’s evaluation, which was currently being scoped.[74]

New LNNP agreement

2.58               DEEWR informed the Committee that negotiations were currently underway for the new agreement, commencing in the 2013 school year. The department planned to have implementation agreements in place for the new agreement by the end of 2012.[75]

2.59               DEEWR indicated that under the new agreement it was ‘looking’ to require jurisdictions to identify ‘what strategies have been working, evidence of why they have been working and for what cohorts, and for the funding to be provided to continue and expand that particular practice within the state or territory’.[76]

Committee comment

2.60               The Committee welcomes the audit report into the LNNP and strongly supports the Auditor-General’s findings and recommendations.

2.61               While the audit report has highlighted a range of concerns, the Committee notes that the LNNP appears to be having a positive impact on literacy and numeracy outcomes and has provided a robust evidence base which education authorities can draw on for future initiatives. Moreover, DEEWR has made clear improvements to the LNNP’s implementation over time, and has responded positively to the audit report’s findings.

2.62               The Committee agrees with the Auditor-General’s comments about the need for longer term evaluation of the LNNP’s impacts. Such evaluation will help ensure future initiatives are better planned and public money is spent in the most effective way possible. The Committee is interested to hear more detail from the department on its plan for implementing the Auditor-General’s recommendation for evaluating the LNNP’s impact ‘at an appropriate stage following its conclusion’, and therefore recommends:

Recommendation 1
	That the Department of Education, Employment and Workplace Relations report to the Committee within six months on the progress of its implementation of the Auditor-General’s Recommendation 2 regarding the development of a longer term evaluation strategy for the National Partnership Agreement on Literacy and Numeracy.

2.63               It is concerning, however, that the initial performance targets set under the LNNP framework were not as ambitious as they could have been. The targets were not aimed at a level that would ‘accelerate improvements’ to the extent foreshadowed in the language of the LNNP agreement. It is also concerning that implementation plans were finalised, and performance targets set, in some instances, very shortly before the first round of reward payments were being evaluated. There may have been value in the LNNP specifying in clearer terms what ‘ambitious’ means, perhaps to the extent of including specific performance targets in the agreement at the outset.

2.64               These findings appear to strengthen the case for National Partnership agreements and implementation plans to be negotiated in tandem, as the Committee has previously suggested,[77] and as has now been included in advice to Commonwealth agencies issued by the Treasury.[78]

2.65               At the same time, the Committee was reassured to hear that where performance targets have not been met, reward funds have been withheld—as was intended under the framework.

2.66               Another key concern of the Committee is that DEEWR did not fulfil its obligation under the LNNP to monitor the agreed co-investment of literacy and numeracy funds by states. DEEWR apparently provided certifications for payments to be made to states despite having been unsuccessful in obtaining the required data from jurisdictions.

2.67               The Committee considers that the LNNP agreement was clear in its stipulation that DEEWR would confirm that co-investments had been made prior to payments being recommended to Treasury, and that state implementation plans would specify the mechanisms by which the relevant data would be provided. That only half of the implementation plans included the required details, and that, of those, none were adhered to in practice, suggests an unwarranted disregard for the terms and intent of the LNNP agreement.

2.68               Further, the inability of DEEWR, and the ANAO, to verify the accuracy of performance results provided by states suggests that departments need to pay more attention to setting up appropriate performance data monitoring and assurance mechanisms early in the development of National Partnership agreements. As was suggested in JCPAA Report 427—Inquiry into National Funding Agreements—outside expertise, either from the Auditor-General or from other sources, could be engaged to assist departments in developing performance data monitoring and assurance frameworks, central to which would be consistent agreement on what data assurance means in the National Partnership context.

2.69               The above findings suggest a general need for DEEWR and the Minister for School Education to be more active in ensuring that the terms of its National Partnership agreements are abided by during their implementation. This may require: better mechanisms to ensure the adequacy of performance targets; the introduction of assurance measures for performance data; improving Commonwealth visibility of state co-investment data; and potentially negotiating more visibility over how the funds provided to states are being spent.

2.70               The findings echo previous JCPAA comments on the need for implementation plans to better reflect the expectations of National Partnerships and to have good quality performance indicators.[79]

2.71               Towards this end, the Committee recommends:

Recommendation 2
	That, in order to help ensure the expectations of future National Partnerships are met, the Department of Education, Employment and Workplace Relations work with states and territories to develop implementation plans and reform targets at the same time as any new National Partnership agreements are developed, in accordance with advice from the Treasury.

Recommendation 3
	That where state and territory co-investment obligations are included in the terms of current National Partnership agreements, the Department of Education, Employment and Workplace Relations take responsibility for monitoring these investments and assessing them before payments are made, even if this requires negotiating more visibility of state and territory data.

Recommendation 4
	That when negotiating National Partnership agreements, the Department of Education, Employment and Workplace Relations seek external advice, potentially from the Auditor-General, on the monitoring and assurance mechanisms that should be incorporated to enable verification of performance data provided by states and territories.

2.72               Finally, the Committee strongly supports the Auditor-General’s comments that in future National Partnership agreements more time must be allowed for programs to have an impact before the criteria for reward payments are assessed. In the case of the LNNP, more time being allowed between facilitation and reward payments could have enabled more ambitious targets to be set by states, and perhaps led to more meaningful and clearly measureable outcomes.

Recommendation 5
	That the Department of Education, Employment and Workplace Relations and the Minister for School Education ensure that sufficient time is allowed in future National Partnership agreements for facilitation payments to take effect before measuring the performance of states and territories against reform targets and making reward payments.