Chapter 3

My School website

In education, good decision making is facilitated by access to relevant, reliable and timely information. Dependable information is required at all levels of educational decision making to identify areas of deficiency and special need, to monitor progress towards goals, to evaluate the effectiveness of special interventions and initiatives and to make decisions in the best interests of individual learners.FF[1]FF

Background

3.1        In 2008, the Council of Australian Governments (COAG) agreed that greater transparency and accountability for school performance was essential 'to ensure that every Australian child receives the highest quality education and opportunity to achieve through participation in employment and society'.FF[2]FF It also agreed that high‑quality reporting is important for students, parents, carers and the community and should include:

• streamlined and consistent reports on national progress, including an annual national report on the outcomes of schooling in Australia;
• national reporting on performance of individual schools to inform parents and carers and for evaluation by governments of school performance; and
• provision by schools of plain language student reports to parents and carers and an annual report made publicly available to their school community on the school’s achievements and other contextual information.FF[3]

3.2        To provide the public with information on each school, COAG agreed that the Australian Curriculum, Assessment and Reporting Authority (ACARA) would:

...be supplied with the information necessary to enable it to publish relevant, nationally comparable information on all schools to support accountability, school evaluation, collaborative policy development and resource allocation.FF[4]

3.3        ACARA would then provide information on each school and this would include:

...data on each school’s performance, including national testing results and school attainment rates, the indicators relevant to the needs of the student population and the school’s capacity including the numbers and qualifications of its teaching staff and its resources. The publication of this information will allow comparison of like schools (that is, schools with similar student populations across the nation) and comparison of a school with other schools in their local community.FF[5]

3.4        The committee majority notes that COAG intended school performance data to be published in the context of broader information about a school's students, teachers and resources.FF[6]

3.5        In August 2008 the Ministerial Council on Education, Employment, Training and Youth Affairs (MCEETYA) agreed that an Expert Working Group (EWG) convened by the Australian Government would provide a report to ministers on relevant measures to guide school evaluation, accountability and resource allocation. In late 2008, EWG commissioned the Australian Council for Educational Research (ACER) to provide advice on national schools data collection and reporting for school evaluation and resource allocation. The report recommended the use of NAPLAN results as the basis for the comparative performance of schools.FF[7]FF It noted that few countries have developed measurement scales along which gain and growth can be measured for all students. The report concluded that NAPLAN is an effective way to assess whether a school is making a difference in a student's numeracy and literacy skills by measuring improvement across the years:

The NAPLAN measurement scales enable status, gain and growth to be measured across Years 3, 5, 7 and 9 and in this sense represent world's best practice in the measurement of student progress.FF[8]

3.6        Regarding public reporting, the Australian Council of Educational Research (ACER) report recommended that:

For the purpose of providing public information about schools, a common national website should be used to provide parents/caregivers and the public with access to rich information about individual schools.FF[9]

3.7        The ACER report noted that it would be important that characteristics that are known to be correlated with student outcomes are taken into consideration:

Research consistently shows a correlation between students' socio-economic backgrounds and their level of school attainment. For this reason, the socioeconomic backgrounds of a school's student intake also must be taken into consideration in any evaluation of the school's performance...

The socio-economic backgrounds of students in a school can be measured either at the level of the school, eg, using data from the ABS census collection districts for the home addresses of the students attending the school) or by aggregating information about the SES backgrounds of individual students in the school. FF[10]

3.8        The ACER report was considered by MCEETYA in April 2009, and the ministerial council agreed that, from 2009, ACARA would be responsible for publishing relevant, nationally comparable information on '...a common national website [which would]...provide parents/caregivers and the public with access to rich information about individual schools'.FF[11]FF This would include publication of the 2008 NAPLAN data and associated contextual information. The information would enable comparison of each school with other schools serving similar student populations around the nation and with the best-performing school in each cohort of ‘like schools’. MCEETYA noted that this information is intended to support accountability, school evaluation, collaborative policy development and resource allocation.FF[12]

Use of ICSEA values

3.9        As noted by ACARA, the best way to compare the performance of schools in the NAPLAN tests would be to find groups of schools with students of similar abilities on commencing school. However, no such measures of starting abilities are available nationally.FF[13]FF The data used instead is Australian Bureau of Statistics (ABS) census data which is also used to allocate funds to non-government schools and to identify enrolling students from similar social backgrounds.FF[14]

3.10      Noting that ACER had emphasised the clear links between a student's socio‑economic background and educational outcomes, ACARA decided to use socio‑economic indices as a starting point to build a comparative tool for student populations. ACARA noted that the ABS produces four indices of socio-economic status, the Socio-Economic Indicators for Areas (SEIFA). Although the indices correlate positively with student achievement they were not designed to predict educational attainment. Therefore a new index was developed, the Index of Community Socio-Educational Advantage (ICSEA):FF[15]

ICSEA uses the SEIFA variables and school data to create an index that best predicts schools' average performance on NAPLAN tests. The variables that make up ICSEA include socio-economic characteristics of the small areas where students live (in this case an ABS census collection district (CCD)), as well as an index of remoteness, and the proportion of Indigenous students enrolled at the school.FF[16]

3.11      The steps taken to calculate an ICSEA value for each school are detailed in the ACARA submission.FF[17]FF ICSEA places schools on a numerical scale, for example:

• a school in a regional town with a student population drawn largely from relatively disadvantaged households may have an ICSEA value of about 850;
• a school in a metropolitan area which draws its students from relatively advantaged households may have an ICSEA value of about 1150; and
• a school in a remote Indigenous community may have an ICSEA value of about 540.FF[18]

3.12      ACARA noted that some schools will not have an ICSEA value because of the nature of the student population—for example, a school solely for children with intellectual disabilities. ACARA also noted:

In a small proportion of cases, ICSEA may provide an inappropriate measure of the socio-educational level of the school. This can occur in instances where there is a mismatch between students' actual levels of socio-educational advantage and that of the CCD values associated with their addresses. An example would be remote schools where the ICSEA values are inflated where a mining community is located in an otherwise disadvantaged remote community.FF[19]

3.13      Statistically Similar School Groups (SSSGs) were checked with state and territory departments and non-government sector authorities prior to the My School website going live. As a result of this checking, the ICSEA values of around 650 out of approximately 10,000 schools were revised before the website was launched.FF[20]FF These changes were reviewed by an expert panel. The use of the ICSEA index on the website was endorsed by MCEECDYA in September 2009.FF[21]

Publishing contextual information

3.14      ACARA is required to publish contextual information on schools on the My School website. Education ministers agreed the Principles for Reporting on Schooling in Australia which include:

• the protection of individual student privacy;
• not publishing comparative data without contextual information; and
• the publication of error margins, caveats and explanatory notes to ensure accurate information.FF[22]

Website launch

3.15      The design of the My School website was endorsed by MCEECDYA in September 2009, and the website was launched on 28 January 2010. It provides profiles of almost 10,000 schools, contextual information and NAPLAN results that can be compared with results from statistically similar schools.FF[23]FF Detailed information on website content is available from the ACARA submission.FF[24]FF ACARA reported that as at 25 June 2010 the website had received 2,445,308 visitors and 3,368,847 visits.FF[25]

Further development of the My School website

3.16      The committee majority notes the intention to develop the My School website in stages.FF[26]FF ACARA advised that it was asked by education ministers in a series of meetings in 2009-10 to investigate proposals for enhancing the website by:

• showing school financial data;
• including nationally comparable senior secondary information;
• including satisfaction with schooling;
• showing student population indicators;
• including growth data on literacy and numeracy achievement;
• listing teaching staff and levels of expertise;
• using student-level data to compute ICSEA;
• making other enhancements to ICSEA;
• providing increased contextual information; and
• taking action to minimise misuse of My School data.FF[27]

3.17      However, the committee majority believes the response so far is inadequate to deal with the level of concern in the community and raised during the inquiry, as discussed below.

Issues raised during the inquiry regarding the My School website

3.18      The committee majority notes the large number of issues raised regarding the My School website.FF[28]FF Many of these focused on misuse of data obtained from the website.FF[29]FF Many also questioned the reliability of school comparisons based on ICSEA values.FF[30]FF Other submissions stated that My School might be placing excessive emphasis on 'point in time measures of student achievement', thereby undermining what they saw as the intended diagnostic aims of NAPLAN.FF[31]FF

3.19      Some, such as the South Australian branch of the Australian Education Union, raised serious questions about the My School website's role as the foundation of the government's 'transparency agenda', explaining that the government already had direct access to a wide range of information on school performance and socio-economic status from existing sources.FF[32]

3.20      Teachers also expressed concern, one stating that NAPLAN tests '...are taking on a life of their own, all because of the unethical way they are being used'.FF[33]FF These concerns are fundamental and call into question whether the website in its current form is presenting useful information.

3.21      The Australian Education Union (AEU), whilst reiterating its in principle support for effective assessment, stated:

Our concerns stem from the fact that we believe that the policy mix advanced by the government has the potential for perverse consequences, perverse consequences which undermine our capacity to deliver sound educational programs for children... [W]e are concerned by a series of policy statements and announcements of government that allocate or attach to NAPLAN something for which it was never designed and something for which it cannot be used—that is, school performance measurement. That remains our central point of contention and concern, because stemming from that is the misuse of that student data for purposes never intended. Therein begin the perverse consequences.FF[34]FF

The AEU also expressed its belief that problems have emerged as a result of the creation of the My School website being '...driven by political imperatives and political timetables'.FF[35]

3.22      The section below describes in more detail some of the issues raised in submissions and then outlines the committee majority's suggestions for improving the presentation and usefulness of the data presented on My School.

Using ICSEA values to group 'like schools'

3.23      A large number of submissions expressed deep dissatisfaction with the use of ICSEA values to inform school comparisons on the My School website.FF[36]FF The issues focused on the validity of groupings of statistically similar schools on the My School website.

3.24      The committee majority notes that the ICSEA value is currently calculated by looking at the community a school is located in, not data pertaining to the actual socio-economic status of a school's enrolled students.FF[37]FF This means that schools located in the same community are considered to be at the same level of advantage or disadvantage, irrespective of the socio-economic backgrounds or other factors specific to their students. Given that private and selective schools attract and can pick students who may not always live near the school, whereas public schools tend to enrol students from their surrounding communities, the ICSEA-based method of measuring socio-economic status has understandably attracted substantial criticism.

3.25      The AEU argued that the way in which the ICSEA measure is calculated is the crux of the problem, saying that the measure does not take into account the fact that '...some higher income families live in lower SES regions and vice versa'.FF[38]FF If higher income families live in areas surrounding public schools, their presence would impact on census data collected in the area, which in turn '...causes an over estimate of the SES of government schools and an underestimate of non-government schools...[and impact on] their placement in the so called statistically similar schools'.FF[39]FF

3.26      The AEU submitted a number of examples of anomalies in 'like school' comparisons, including:

• The Kings' School, a wealthy private school in Parramatta, and Gundaroo Public School, in a small town a short distance from Canberra;
• Blacktown Boys High, Western Sydney, and Alice Springs School of the Air;
• Terrigal High School, with a student population of 1300, and Cameron Downs Public School, with only six students in the Queensland outback; and
• Haileybury College, a private school in Melbourne, and Cleveland Street Intensive English High School, a NSW public school which focuses on teaching students who are from a non-English speaking background.FF[40]

3.27      The Department of Education Tasmania explained the importance of applying appropriate safeguards around the presentation of test results and argued that:

The perceived weakness of the current ICSEA measure is that it is more an index of community socio-economic status, than an accurate measure of the socio-economic status of the students who attend a particular school.FF[41]

3.28      Other submissions raised similar concerns about ICSEA being '...systematically and substantially biased,' and argued that such area-based indexes, although useful in other contexts, can be '...vulnerable to the ecological fallacy'.FF[42]FF The ecological fallacy occurs when individual-level relationships are inferred from aggregate-level ones.FF[43]

3.29      Dr Mark Drummond, a researcher with a focus on mathematics and statistics, submitted that only around three per cent of the data used to calculate ICSEA scores is

...meaningful and valid data based on the actual families of the actual students at the actual schools. The other 97 per cent or so of data is meaningless "noise", based on families and households with no substantive connection at all to the schools whose ICSEA scores are being determined.FF[44]

3.30      Alternatives to area-based measures of advantage were outlined in submissions. These included using individual-level measuresFF[45]FF and conducting data matching between government agencies such as the Taxation Office and Centrelink, as already occurs for parents who receive benefits.FF[46]FF Such measures would go some of the way towards remedying statistical problems currently affecting 'like school' comparisons.FF[47]

3.31      Over the course of a series of meetings in the year to June 2010, education ministers asked ACARA to investigate making use of student-level socio-economic status (SES). This information is currently available from some states and territories but not all. ACARA is considering either obtaining family-level information in all jurisdictions or using the data for those jurisdictions where it is available.FF[48]FF Education ministers have also asked ACARA to look into:

• obtaining updated and comprehensive home address data for all students to improve the accuracy of ICSEA in cases where CCD data is used;
• including within the ICSEA formula a variable to take account of the effect of language background other than English;
• improving the process for quality assuring ICSEA values for individual schools and, for those for which CCD data is used, identifying instances where the initial estimates is inappropriate.FF[49]

3.32      The committee majority notes that education ministers have how endorsed the proposals outlined above and the new model outlined below:FF[50]FF

An analysis undertaken by ACARA compared the current ICSEA formula with a new formula based on student level-measures of parent education and occupation status, as well as considerations for LBOTE and the proportion of Indigenous students. The result of this modelling indicates that the new formula will improve ICSEA’s ability to predict school NAPLAN performance, in addition to having greater face validity. Analyses indicate that there is a 7% increase in the explanatory power of ICSEA when direct student-level indicators of parent education and occupation are used.

Under the methodology endorsed by Ministers, data on parent occupation and education collected directly by schools from parents will be used (where available) in preference to census data. This will ensure that the ICSEA value assigned to a school closely reflects the socioeconomic backgrounds of the students (SES) actually enrolled in that school. ACARA is currently collecting updated direct parent data from jurisdictions and sectors. At the same time, ACARA is collecting updated student address data to enhance the quality and completeness of the indirect parent data (census data).

The new model utilises ‘direct parent data’; however, to obtain an accurate indication of the backgrounds of students in each school, it is necessary that a certain percentage of data in each school be available. Where the available direct parent data do not meet this threshold, or where updated student address data are unavailable, the school’s ICSEA calculation will revert to the current calculation based on 2007 CCD information.

Once updated student information is collected for 2010, the ICSEA will be recalculated using the recommended approaches. All data included in the revised ICSEA will be tested prior to broader distribution and will involve extensive consultation with the ICSEA Expert Panel.FF[51]

Committee majority view

3.33      The committee majority notes the importance of accurately measuring the relative level of school advantage to ensure that 'like school' comparisons are much more reliable. The committee majority is alarmed by the evidence it has received outlining severe shortcomings of present methods of calculating ICSEA and is disappointed that ACARA launched My School without first anticipating and addressing at least some of these shortcomings.

3.34      The committee majority notes the approved enhancements outlined above, but retains some serious concerns about the enhancements' capacity to adequately address the issues raised. Of particular concern is the fact that student-level data will only be used 'where available'. The committee majority would prefer to see a more tangible commitment to replacing the current ICSEA calculation method for all schools.

3.35      In addition, the committee majority is concerned that—given the fact that some schools will in future have their ICSEA values calculated on the basis of student-level data and others on the basis of community-level data as is currently the case—there is potential for further inconsistency and distortion if NAPLAN test results for schools from the two groups are ever compared.

Recommendation 8

3.36             The committee majority recommends that ACARA prioritises the improvement of the method used to develop like school comparisons and commits to the introduction of a method based on student-level SES data for all schools prior to the reporting of 2011 NAPLAN test results.

League tables

3.37      In the absence of proper contextual information and transparent, professional interpretation of data, there is a concerning potential for schools to be crudely and unreliably ranked on the basis of raw NAPLAN data. This was one of the most frequent concerns raised in submissions—that is, that the publication of raw school performance data on the My School website could result in league tables of questionable accuracy being published.FF[52]FF

3.38      League tables are assembled in order to rank schools according to student performance in NAPLAN tests, and were created by a number of media publications across the country '...within days of the student data going online.'FF[53]FF

3.39      Private companies also sought to profit from disseminating NAPLAN results.

In February [2010] a private company Australia School Ranking established a website from which it was selling for $97 a 854 page report containing the rankings and league tables of all kinds of all Australian schools. To its credit, the threat of legal action by ACARA forced the company to withdraw its report from sale.FF[54]

3.40      More recently another website, Better Education Australia, was established. The website is funded through advertising and operated to

...[provide] informative and comparative NAPLAN results and information including private school scholarships and selective schools to parents wanting to make choices about schooling for their children. We also provide free service to the community in Australia and abroad by answering questions via emails. The website is just a hobby, and not for commercial purposes.FF[55]FF

3.41      The AEU believes that there is no substantive difference between Better Education Australia, which offers league tables free of charge, and websites which charge for their reports.FF[56]

3.42      In its submission the AEU also drew attention to what it claimed was inaccurate information published by media outlets such as the Sydney Morning Herald, which produced its own state performance rankings by averaging school mean scores for literacy and numeracy.FF[57]

3.43      The AEU further highlighted the unreliability of using league tables to rank schools by highlighting the dramatic effects of shifting student cohorts on particular school results, such as that of Mount Blowhard Primary School, where top rankings for some Year 3 students '...were over 100 points higher in 2009 than 2008 due to the changing cohort of students'.FF[58]FF

3.44      A submission from the South Australian branch of the AEU pointed to repeated assurances from former education minister the Hon. Julia Gillard MP, who categorically defended My School and maintained that it was not possible to compile league tables using information from the website. According to the South Australian AEU branch, given the ease with which league tables were compiled almost immediately following the publication of results on My School, this '...is either stunning technical incompetence, political naivety or deliberate misinformation – or perhaps a mixture of all three'.FF[59]

3.45      The Australian Primary Principals Association pointed to league tables being a consequence of publicly reporting test results without adequately taking into consideration factors beyond a school's control.FF[60]FF The NSW Primary Principals' Association argued that results available on the My School website 'should not exist in a form that allows comparisons between schools' because league tables are 'educationally indefensible'.FF[61]FF

3.46      The Department of Education in Tasmania advised that:

...the 'use by third parties of NAPLAN results to create simplistic league tables often maligns the excellent work of schools in enriching the lives of their students and working towards improved outcomes, and raises the stakes of the tests from their original purpose to one of 'high stakes'.FF[62]

3.47      It is evident that these concerns are present in the community from the following examples provided to the committee:

League tables are only harmful to schools and students by labelling them.FF[63]

The regrettable outcome [of NAPLAN] is the excessive importance given to it by the media and selective interest groups including the teachers' unions.FF[64]

3.48      Submissions also highlighted the argument over whether student performance data belongs only to teachers, parents and students and as such should not be released publicly in light of the potential for unintended harm. The AEU emphasised the distinction between parents' rights to information relating to their own children and that of the broader community, arguing that parents do not have any inherent right to information relating to the progress of other people's children.FF[65]FF

3.49      A range of submissions offered suggestions for minimising the potential for harm, including by removing raw school averages from the My School page. The ACT Council of Parents and Citizens Association would like to see raw averages replaced with student results in bands, thereby making it more difficult for the media or anyone else to devise simplistic league tables.FF[66]FF The AEU among others argued for the introduction of legislation which would prohibit the publishing of league tables.FF[67]FF The NSW Primary Principals Association proposed a number of ways in which adverse effects could be minimised, including the incorporation of an 'Acceptable Use' page which would require users to agree to conditions of use before accessing My School data.FF[68]

3.50      ACARA reiterated its firm position against league tables at a hearing held in Canberra:

One thing that I think every educator would agree with is that we do not want league tables. A league table, to my mind, is where you rank schools without regard to the nature of the students within the school. My School explicitly does not do that, and all of us in Australia are very much against having league tables.FF[69]

3.51      The committee majority notes that ACARA has taken action to minimise misuse of data on the My School website and will implement changes prior to the release of the new version of the website in December 2010. Enhancements will include:

• a 'click-wrap' requiring users to indicate their agreement up-front to terms and conditions of use of My School data;
• a tool to deter automatic scraping of data from the website.FF[70]

3.52      However, there are limits to the action ACARA can take in response to the misuse of data, particularly where newspapers, which are covered by their own legislation and not ACARA's copyright clause, are concerned. Federal legislation would have to be enacted, and enforced, in order to prosecute newspapers which printed school league tables using data from the My School website.FF[71]

Committee majority view

3.53      The committee majority makes a clear distinction between the My School website, which provides contextual information (to be enhanced as a priority, as per Recommendation 12 later in this chapter), and the publication of crudely designed school rankings by the media and other third parties.

3.54      The committee majority supports submissions calling for more rigorous protocols on reporting, accessing and using student data in order to prevent the media and other third parties from publishing crude league tables.

Effect of publishing comparative data and league tables on school, teacher and student morale

3.55      A range of views were expressed indicating that judging and comparing teachers on the basis of NAPLAN results was damaging school morale, sometimes even having a divisive influence among teaching staff.FF[72]FF A November 2009 letter from peak parent, principal and union organisations to the then Minister for Education, the Hon. Julia Gillard MP, stated that:

There is considerable evidence that the inappropriate use of data to compare schools can have serious negative impacts, both on the testing itself and on the very schools and children it was intended to help. Allowing student data to be inappropriately or mischievously used for the creation and publication of league tables could exacerbate the difficulties of the communities concerned, narrow the school curriculum and risk the testing process itself becoming corrupted.FF[73]

3.56      The Queensland Council for Parents and Citizens' Associations cited reports from parents of children being physically sick before tests, and being pressured to do well in order to avoid making '...their teacher look bad.'FF[74]FF Moggill State School Queensland Teachers Union argued that teachers now had their reputations at stake and had been given an incentive to teach strong performers and gifted students, who are often clustered in classes, instead of being judged on the performance of lower achievers.FF[75]FF The Australian College of Educators added that teachers in the most disadvantaged schools are already '...more likely to be our less experienced teachers [who] need clear standards and support, not more pressure'.FF[76]

3.57      A significant number of teachers reported feeling frustrated and demoralised. One of these, Marianne Scholem, pointed out that '...not every school can be at the top of the pile...teachers like myself will become disillusioned and add to the burnout statistics.'FF[77]FF Some also pointed to overseas experience which suggests '...that league tables lead to a climate of trepidation, incrimination and blame in schools'.FF[78]

3.58      Others were displeased with what they saw as forced competition at the expense of teacher and school collaboration, calling My School '...a veneer of action' and questioning '...how telling people that they work at the worst school will lead to them improving the school'.FF[79]

3.59      The Australian College of Educators advised that:

...[educational] excellence occurs and improvement agendas are most effective when respectful, relational cultures are built through schools in partnerships with parents, caregivers and their communities.FF[80]FF

The College also expressed its view that 'school against school data presentation', in its current form on the My School website, does not promote the '...development of networks, communities of practice, and communities of interest,' and suggested that MCEECDYA and ACARA might explore renaming the website from My School to OurSchools.FF[81]

3.60      Other submissions drew the committee's attention to the importance of morale at more disadvantaged schools which do not perform well when ranked. One teacher recalled her own schooling in Fairfield, Sydney—by no means an affluent suburb—where dedicated teachers imbued her with educational aspirations despite the circumstances.FF[82]FF 

3.61      A parent of a child attending a school with below average NAPLAN scores commented on what she saw as 'labelling' students as low achievers, expressing a fear that her daughter and others like her would simply accept the label and stop trying to do better.FF[83]FF Another submission lamented the harm done to parent, student and teacher self-esteem in low socio-economic communities which work hard and do their best to improve outcomes.FF[84]

3.62      Other submissions echoed this, such as that from Christine Turner on behalf of Chatswood Hills State School, who noted that some schools only appear to be underperforming when compared to others because they cater to particular groups of students or operate under particular circumstances of disadvantage. They are nonetheless proud of their record and afraid that their '...hard earned reputation is at risk'.FF[85]FF

3.63      The ACT Council of Parents and Citizens Associations referred the committee to a comment from a parent who had looked up her child's school on the My School website:

‘All it did was leave me with a bad taste in my mouth,’ because given where she lives within the school boundaries her child could not go to another school and her child’s school was being compared against private schools that she could not afford. So she felt that the school was being stigmatised with this bad reputation and she could not do anything about that. When she went to the website it left her with this bad taste.FF[86]

The Council cited the above as an example of how the current approach to publishing NAPLAN results:

...punishes, humiliates and demoralizes students, teachers and schools who have been singled out by the crude and at times inaccurate comparisons made between apparently "similar" schools as well as from the creation of simplistic league tables by the media and other organisations.FF[87]FF

Committee majority view

3.64      The committee majority notes the potential for harm to be caused by simplistic and crude league tables constructed using information from the My School website and takes very seriously any reports of adverse effects, however small in number. Until more is done to protect My School information from misuse the website and league tables will be inexorably linked by association, as will responsibility for any resulting harm or distress.

3.65      The committee majority endorses recent initiatives from ACARA and MCEECDYA regarding the responsible use of My School data but remains concerned about the harmful impact of irresponsible and unchecked conduct by third parties, such as the media, which are still able to misuse NAPLAN data available on the government's My School website.

Recommendation 9

3.66      The committee majority recommends that ACARA and MCEECDYA examine and publicly report on ways to mitigate the harm caused by simplistic and often distorted information published in newspaper league tables.

Turning NAPLAN into a 'high stakes' test

3.67      Professor Robin Alexander from the United Kingdom, who has written extensively on education policy and was the director of the Cambridge Primary Review of English Primary Education, recently gave a number of lectures in Australia in which he captured the advent of high stakes testing thus:

Of all the so-called 'levers' of systemic reform, tests seem to be the instrument of choice in policy-makers' efforts to do the two things which they believe they must always be seen to do: raise educational standards and call teachers and schools to account. This means that tests are high stakes not just for children and teachers but also for politicians, and that they may be as much about political capital as educational progress.FF[88]

3.68      Other submissions also conveyed the sense that the reputational consequences of publicly reporting and comparing school NAPLAN results have increased the stakes for schools to do well to unacceptable levels.FF[89]FF For example, the Association of Heads of Independent Schools of Australia (AHISA) argued:

The use of NAPLAN test results for the purposes of comparative measurement of school performance and distribution of some federal grants to the states and territories and to identified 'disadvantaged' schools has served to morph NAPLAN diagnostic tests into 'high stakes' tests, where money and/or reputation rides on their outcome. Media reporting and exploitation of the data through the creation of 'league tables' have exacerbated this misuse of NAPLAN.FF[90]

3.69      The Australian Primary Principals Association echoed this concern about turning NAPLAN into a high stakes test, citing two potential factors which contribute to this outcome: the My School website and '...the $350 million of reward money that has been offered for an improvement in NAPLAN scores':FF[91]

We think it is fantastic that there is money for schools in need. That is a big tick. But when reward money is used to threaten principals or set targets—you must improve by five per cent or 10 per cent before you can get the reward money—I think that has a perverse effect of what the reward money is intended to do. When I moved around the country talking to principals, I think what alarmed me most was a discussion saying yes we have been given targets that we have to reach; we now are going to choose the group of children in our school that we will put most our energy into because they are just below the national benchmark; we are not going to focus on those children that probably, with all the resources in the world, will struggle to get to the benchmark and improve our scores. This was a real ethical dilemma for principals that I sat with in three states. Some were saying you can’t do that, and others were saying they had been told they had to improve; this is a business proposition and that is how we have to look at it.

They are the sorts of things that we should be aware were happening. They only happened this year. That is what we have to keep in perspective—it was not happening before that [the advent of My School]. We do not have a problem with students doing the NAPLAN test; we have always supported the NAPLAN test. The test itself is not the problem.FF[92]

3.70      The creation of this 'high stakes' environment has a number of potential negative consequences including 'teaching to the test' and narrowing the curriculum, as outlined below.

Teaching to the test and narrowing the curriculum

3.71      Teaching to the test involves repeated practice of test format as opposed to merely instruction on test content, often at the expense of other, possibly more educationally valuable, curriculum content.FF[93]FF Many teachers and schools made submissions to the inquiry which expressed a sense that they were being forced to teach to the test, that is, to rearrange their teaching plans to focus on NAPLAN tests.FF[94]FF 

3.72      This extended to sacrificing 'desirable pedagogies' such as inquiry-based learning due to time constraints and instead applying a teacher-directed style more suited to extracting better NAPLAN results.FF[95]FF

3.73      The Junee Teachers Association stated that intensive preparation for a single point-in-time 'snapshot' of student achievement levels could actually compromise the diagnostic value of NAPLAN.FF[96]FF

3.74      The AEU argued that the high reputational stakes attached to NAPLAN have forced excessive emphasis to be placed on the tests, which '...has had a profound effect on schools, curriculum, teaching and students,'FF[97]FF and means that the '...primacy of the educational needs of students is subjugated to the requirements of schools to achieve in testing regimes.'FF[98]FF The AEU attributes some of the pressure on teachers to teach to the test to competition-induced pressure between jurisdictions to perform in NAPLAN tests, citing as an example Victoria, where the state education department set out a ten week 'delivery strategy' ahead of NAPLAN 2010:

Principals were directed to appoint a NAPLAN coordinator, to "facilitate a sample testing benchmarking process which may require further resourcing", to "provide additional assistance to students identified as capable of making a significant improvement" and to "privilege the testing as an event of significance". Teachers were directed to, "explicitly teach for NAPLAN by including the genre of NAPLAN, commonly used terms and a daily NAPLAN item in the program of instruction".FF[99]FF 

3.75      Submissions highlighted that the curriculum can be narrowed or distorted if teachers feel they must focus on a particular aspect of teaching, such as teaching for a specific test. Helen Stearman, a teacher, argued that NAPLAN is having precisely this effect on teaching practices around the country, threatening to become '...the de facto curriculum.'FF[100]FF This concern that schools are increasingly sacrificing the broader curriculum in pursuit of better NAPLAN results was echoed by Lutheran Education Australia, among others.FF[101]

3.76      The inquiry received submissions suggesting ways to prevent the curriculum being narrowed by NAPLAN. David Andrich, Chapple Professor of Education at the University of Western Australia, offered that:

...not every student in a particular year needs to sit a test in which every student responds to exactly the same items. The technology exists in test construction, administration, analysis and interpretation that the results of students and of schools can be placed on the same metric even if all students in the same year do not respond to exactly the same items. This is the same technology that currently permits the results of students from different grades, who do not respond to exactly the same items, to be placed on the same scale.FF[102]

3.77      Teachers also reminded the committee that it is important for NAPLAN tests and the wider curriculum to work together rather than compete for attention.FF[103]

3.78      This is echoed by international academics such as Professor Robin Alexander, from Cambridge University, who said in a recent keynote address at the University of Melbourne:

Over-concentration on the practice of basic skills in literacy and numeracy unrelated to a context in which they are needed means that those skills are insufficiently extended and applied.FF[104]

3.79      Others question whether teaching to the test is an entirely negative proposition. Dr Ben Jensen stated that teaching to the test ultimately results in an increased focus on literacy and numeracy, which is not necessarily negative.FF[105]

3.80      Addressing a similar point, Mr Angelo Gavrielatos, federal president of the AEU, explained:

Please do not misunderstand what I am saying as an argument against mastering, and proficiency in, literacy and numeracy. They are the foundation blocks, the building blocks, of learning. There is a significant difference between teaching to ensure children can attain the best literacy and numeracy skills and teaching to the test. Schools are being directed by bureaucrats to teach to the test, to teach NAPLAN, to teach ‘the genre of NAPLAN’. With all due respect, NAPLAN is not a genre; it is a test.FF[106]

Committee majority view

3.81      The committee majority believes that these examples of teacher dissatisfaction and concern show the importance of increasing teacher engagement in education policy development and rollout. The committee majority also believes that more must be done to encourage a complementary relationship between the tests and the wider curriculum.

3.82      The committee majority notes that ACARA has expressed a desire to look at ways in which to align NAPLAN tests with the wider curriculum in the future.FF[107]FF However, given community scepticism of the national curriculum currently being developed by ACARA, the committee majority cannot support this intention until concerns about the curriculum have been adequately addressed.FF[108]FF

Recommendation 10

3.83      The committee majority recommends that ACARA identify, analyse and report publicly on possible means of strengthening the relationship between NAPLAN tests and the wider curriculum. The committee majority reserves its support for any alignment between the tests and the new national curriculum until the quality of, and community support for, the curriculum become clearer.

Do parents really have a choice about where to send their children to school?

3.84      One of the key arguments underpinning the publication of NAPLAN results on My School is that the website provides parents with information and choice when assessing which school their children will attend. A number of submissions questioned this assertion.

3.85      The Tasmanian Education Department considers the choice argument to be misleading for the following reasons:

• It implies that schools are entirely responsible for a student's results, without taking into account the personal, political and social context in which they operate;
• Most parents do not have choice. In many parts of Australia there is no other school and in other areas many parents so not have the resources to move their child to another school;
• Where there is choice, parents choose schools for a variety of reasons, not only test results. They choose because of the needs of their children, the school culture, the social or extra-curricular programs the school has, because their child has friends who attend that school, because it is close to where they live or merely because they went there themselves;
• If the argument is accepted that parents will choose schools based on the public reporting of NAPLAN result, the parents who take this option could be changing schools each year as results change due to the different cohorts being tests.FF[109]FF

3.86      The Australian College of Educators, too, emphasised that parents with children in disadvantaged schools rely on the government to provide a sound standard of education for their children and cannot 'vote with their feet'.FF[110]FF Their submission argued that for schools serving economically and socially marginalised parents and children '...the logic of parent power and school choice, as a response to NAPLAN comparative information, does not apply'.FF[111]

3.87      Other submissions, such as that from Junee Teachers Association, informed the committee that parents were in fact withdrawing their children from particular schools in order to send them to better performing schools nearby, resulting in

...dramatic effects on the staffing, resourcing, and programming of schools...[including]...a significant narrowing of curriculum offerings, particularly in the senior school.FF[112]

3.88      This was echoed in by other submissions which also reported that talented students were leaving because their school had been 'branded a failure'.FF[113]

3.89      Mrs Sharyn Lidster, Acting General Manager, Strategic Policy and Performance, Department of Education, Tasmania, informed the committee the department had done a statistical analysis of student movement across schools in response to principals' concerns. The findings indicated that no significant movement has occurred; in fact, the '...movement variation is what we saw from year to year before we did public reporting, so it has not impacted'.FF[114]

3.90      The Australian Primary Principals Association (APPA) confirmed that some parents have decided to change the school their children attend as a result of NAPLAN reports, but that this only appeared to be happening in 'tiny numbers'.FF[115]FF

Inadequate contextual information

3.91      Submissions questioned the value of the comparisons between schools available on the My School website for reasons beyond those to do with ICSEA values and often in connection with a perceived lack of adequate contextual information about schools. For example, the NSW Teachers Federation argued that '[t]he prominence given to quantitative data presented in colourful graphics projects an air of scientific authenticity that is essentially populist.'FF[116]FF The submission quotes a NSW principal who says that the coloured comparison graphs used by My School will always get more attention than any other information on the website.FF[117]FF

3.92      A substantial number of submissions lamented the lack, or at least lack of prominence, of contextual information on the My School website.FF[118]FF Information on parents' views presented to the committee by organisations such as the ACT Council of Parents and Citizens Associations indicates that parents want more contextual information about the schools available to their children.FF[119]FF 

3.93      The committee majority notes that, as stated by the Australian Council for Educational Research, My School '...is best viewed as the first version of the website,' which will be '...refined and further developed into the future'.FF[120]

3.94      ACARA has notified the committee of its plans to increase reporting of school contextual information by publishing the percentages of enrolled students who come from a language background other than English, as well as by expanding the text field available for principals to describe school profiles. The latter are in the process of being collected from schools.FF[121]

3.95      The AEU expressed the following hopes for the new version of the My School website:

Our view would be that in the next iteration of the My School website, which will emerge, there needs to be a very serious and honest communication strategy in order to put front and centre what the data is about and what the limitation is of the data and contextualise it so that people understand that this is not the be-all and end-all of schools. We cannot lead with information about schools with the use of NAPLAN data. There is a lot more to a school and the need to establish context of a school before you even start discussing what is data—which ostensibly is a snapshot of student skills in the area of literacy and numeracy, only one at a particular time and only one narrow slither, important as it is—of the educational wellbeing of a child.FF[122]

The improvement that is required with the federal system of the My School website is that at the very least any information presented on that website should lead with contextual information about that school so that it can present a picture of itself which deals with, on a daily basis, its successes, its challenges and the like. Any subsequent information should be just that, subsequent information, and it should not be privileged, given the fact that it is not the be all and end all; in fact it is a very small part of a school’s life.FF[123]

Committee majority view

3.96      Providing parents with information to make empowered decisions about which school their child attends is a strong argument in favour of reporting NAPLAN results on the My School website. The committee majority acknowledges, however, the evidence received that not all parents have options about where to send their children to school.

3.97      The committee majority also notes the evidence received suggesting that too much can be mistakenly inferred about a school on the basis of student performance data. Therefore, the committee majority does not support publishing raw test results devoid of context or acknowledgement of the fact that schools are not solely responsible for student performance.

3.98      The committee majority notes and supports ACARA's plans to include information on the percentages of students from a language background other than English, but believes this only goes some of the way toward providing adequate contextual information about schools.

Recommendation 11

3.99      The committee majority recommends that ACARA and MCEECDYA move to include more contextual information about schools on the My School website, reflecting the complex range of factors that affect schools, and acknowledge to users of the website their awareness of the limitations of comparisons based on raw performance data due to extrinsic factors. The committee majority further recommends that ACARA commit to ensuring this contextual information is available ahead of the reporting of 2011 NAPLAN results.

Can we learn from the international experience?

3.100         Whilst standardised testing is used by a number of countries, student performance in international tests suggests that the existence of such testing does not lead directly to improved performance. Neither does the international experience with standardised testing necessarily translate directly into the Australian context. As stated by Dr Peter Hill of ACARA:

...[T]hat history of high-stakes testing leading to high-stakes consequences for the staff within schools has, of course, scared other people in other jurisdictions, and there has been a thought around that Australia is simply going down this path. What I would say is that we are not. Educators come and visit us from the UK and tell us all the bad news about what happened there. We know what happened there, we have learnt from that and we are not going down that path.FF[124]

3.101         Certain countries which outperform most others on international assessments of literacy and numeracy without relying on national standardised testing are frequently cited as arguments against standardised testing. On this, the committee received considerable information on Finland in particular.FF[125]FF

3.102         Finland achieves indisputably impressive results, consistently coming at or near the top of international rankings, with very narrow achievement gaps between the highest and lowest performers within individual schools and between schools, as well as between students from different socio-economic backgrounds and regions of the country.FF[126]

3.103         However, the fact that Finland does not base policy decisions on national testing in the same way as Australia does not of itself explain Finland's success. There are a number of important additional factors at play which this report can only touch on. In Finland every teacher:

...[has] a masters degree and is an expert in how to assess, test and act on the results of student assessment, about a seventh of all professional teachers in the school are people who are trained to ensure that no student will fall behind by more than 48 hours. That is the kind of strategy that is going to lift the performance of students and close the gap between our high and low-performing students.FF[127]

3.104         The Finnish education system enjoys what Professor Brian Caldwell of Educational Transformations calls a 'cultural advantage' which means that the system is not directly comparable to Australia's:

It is not necessarily salaries, because when you adjust for purchasing power of currency our teaching salaries are probably a little above those of Finland. There is a cultural advantage that Finland enjoys, and that is that for many decades teaching has been a very highly valued profession. The initiatives of the Finnish government, in saying that the level of professional knowledge and skill that you now require if you are really going to make a difference for each and every child requires five years of university preparation—and then making a master’s degree a requirement for beginning teachers—lifted the status of the profession quite significantly and, as Geoff Masters points out, and as the McKinsey report also points out, education is one of the top three preferences for those entering university.FF[128]

3.105         The English and American school systems do administer standardised tests and use the data to compare schools. Neither country performs well in international literacy and numeracy assessments.FF[129]FF Both countries' approaches to the publication of comparative reporting of test results, which focus on punitive measures for schools which do not meet targets, are controversial and the consequences of their respective policies disputed.FF[130]FF 

3.106         Dr Peter Hill of ACARA elaborated on these international examples:

I do understand the feelings of many people who have seen what has happened in the USA and the UK, which was not, in my mind, very intelligent in terms of accountability. To be honest, it happened a long time ago in the UK and more recently in the USA.

In the UK they certainly published lists of schools in terms of raw percentages meeting a standard, and they ranked the schools. The schools at the top, of course, were the schools that had all the smartest kids. They were independent schools, and typically girls’ independent schools. The schools at the bottom tended to be the schools in boroughs like the inner London borough of Hackney, where I have done quite a bit of work recently. Of course, the demographic composition there means that the schools are really struggling to do the right thing with their students. So schools were unfairly compared.

The other thing that happened was in the USA following the passing in January 2000 of the No Child Left Behind legislation, which incidentally was supported by both the Republicans and the Democrats; there was complete agreement on it. It meant that each state within the USA enacted its testing program but that there would be serious consequences for schools that did not meet annual targets. These serious consequences meant that you were pretty well placed on notice after one year. After two years there was some small action, but after three years the action got very serious, even down to taking money from the school so that it could be given out to parents to get private tutors. In other words, the school was deemed not to be able to deliver this, so the money was taken from them. Indeed, the school could be closed down or staff removed and so on. In other words, there were very high-stakes consequences, often unfairly, because the schools that were the subject of this found it very hard to improve the students for reasons which were partly outside their control.

So that history of high-stakes testing leading to high-stakes consequences for the staff within schools has, of course, scared other people in other jurisdictions, and there has been a thought around that Australia is simply going down this path. What I would say is that we are not. Educators come and visit us from the UK and tell us all the bad news about what happened there. We know what happened there, we have learnt from that and we are not going down that path.FF[131]

3.107         The committee majority notes that, despite similarities to the Australian system, the context and outcomes are quite different. The US and English education systems both pursue a policy of withdrawing resources from underperforming schools. Conversely, in Australia, the focus of the NAPLAN testing regime is improvement, and poor performance is identified in order to better direct additional resources.

3.108         The Australian College of Educators supported making school performance information available to the public and pointed to '...international literature on transparency and accountability for governments, [which] provides strong evidence that public access to this sort of information is a potentially powerful force that can contribute  to keeping governments focussed on their policy promises.'FF[132]FF The College of Educators also pointed out the problem with applying this to the Australian context, where '...the way in which the information has been organised for public consumption is not focussed around "keeping the government honest"'.FF[133]

What are the alternatives?

3.109         The committee majority notes that some submissions to the inquiry support replacing current school performance data on the My School website with information on the value a school adds to student performance.FF[134]FF

3.110         Value added to student performance by schools can be calculated by comparing the progress made by individual students between one test and the next, controlling for extrinsic factors such as student background information. Once the background factors are controlled for, what is left is a measure of the contribution a school has made to individual student progress,FF[135]FF which is calculated by:

...using a statistical model that compares the progress made by each student with the same initial level of attainment, controlling for background factors.FF[136]FF

3.111         Dr Ben Jensen informed the committee that value-added measures of school performance are preferable because they are more accurate than using raw NAPLAN test scores or similar 'contextualised attainment models' which attempt to control for background factors in less precise ways:

Australian and international research consistently shows that value added measures provide more accurate measures of school performance than the use of raw test scores or what have previously been termed contextualised attainment models. These models attempt to control for the socioeconomic background of students, usually through some form of multivariant modelling. These models are similar to the methods used to measure school performance on the My School website that use raw NAPLAN scores and then attempt to control for the socioeconomic background of the school... research consistently shows that this method produces less accurate measures of school performance than value added modelling.FF[137]

Even with data far more comprehensive than anything available in Australia, and modelling far more complex than that utilised for the My School website, the predictive power, the accuracy, of these types of models were roughly half that of a simple value added model. These findings are echoed around the world. Value added measures of school performance are not 100 per cent accurate—no estimate of school performance ever will be—but they are a substantial improvement on current measures and widely considered to be the most accurate measures available, particularly for schools serving disadvantaged communities. That is why they have been supported by stakeholders such as unions and school associations in numerous countries.FF[138]

3.112         Dr Jensen elaborated on what appears to be a widespread misunderstanding of what value-added measures of school performance are, explaining that they are not '...simple measures of student progress' but are instead relative, because:

All students will progress. Even a student who in year 3 performs at an average level but in year 5 performs at a below average level will have progressed in absolute terms, but we should be interested in the contributions schools make to that progression relative to the progress made by students at each initial level of attainment.FF[139]FF

3.113         An additional benefit of using value-added measures of school performance is that their introduction would remove the need to use the ICSEA index and to group schools into 'like schools', because both would be contained in the estimation used to calculate the value added.FF[140]FF This would address one of the most contentious aspects of the My School website.

3.114         The committee majority notes that Dr Jensen advocates including both raw test result data and value-added measures on the My School website.FF[141]

Committee majority view

3.115         Given the large number of concerns expressed in submissions about the My School website, the committee majority believes it is necessary to increase the accuracy of student performance measurements on the website in order to provide better quality information on how schools and students are progressing.

3.116         The committee majority has considerable reservations about the My School website in its current form, and believes that shutting down the website to prevent further harm may be necessary unless steps are taken to improve the quality of the information presented. However, the committee majority does not believe that taking the My School website down is the best way forward and instead sees substantial potential for improvement in methods of publishing school performance data on the website by incorporating value-added measures as outlined above.

3.117         The committee majority emphasises the importance of following up low NAPLAN test results, or low school value-added scores in future, with immediate intervention aimed at assisting individual schools and students. The committee majority does not support putting punitive measures in place for schools with low value-added scores, as is the case in England.FF[142]

Recommendation 12

3.118         The committee majority recommends that ACARA and MCEECDYA comprehensively revise the type of information available on the My School website to shift the focus from raw school performance data to value-added measurement of school performance.

Conclusion

3.119         On the weight of evidence received outlining numerous community concerns about the My School website, the committee majority cannot support the website in its current form.

3.120         The problems outlined in this chapter are wide-ranging and a cause for serious concern. They point to a substantial and justified lack of confidence in the website among the parent, teacher and wider community. The potential for this lack of confidence in My School to engulf the entire national literacy and numeracy assessment system, thereby compromising the benefits of NAPLAN tests themselves, is considerable.

3.121         For this reason and in the interests of progressing the central aim of national literacy and numeracy testing—that is, improving educational outcomes for students—the committee majority concludes that the best way forward is to instigate, as a matter of priority, a comprehensive revision of the My School website.

3.122         The committee majority is firmly of the view that a more measured and evidence-based approach must be taken. Instead of holding schools and teachers accountable on the basis of opaque and often distorted comparisons between raw student test results, the government must instead turn its attention to developing an optimally accurate, reliable and verifiable measure of school performance. Teachers and schools must then be allowed the autonomy and given the necessary support to design and implement frameworks aimed at improving student performance across the board, but in particular where students are performing below national benchmarks.

3.123         The committee majority concludes that the focus of the My School website must shift to include information on the value added by schools to student performance, taking into consideration background and extrinsic factors in a much more reliable fashion than is currently being achieved by the use of ICSEA values as a basis for comparison. The committee majority cannot support any future version of the website which fails to do this.

Senator Chris Back

Chair

Navigation: Previous Page | Contents | Next Page