Data collection about customer satisfaction and network performance

Introduction

6.1        The picture painted in evidence to this committee and in the broader public discussion of the NBN is that the project is beset with issues relating to poor customer experiences. It is difficult to ascertain however, just how representative this is of the overall experience for customers. Some submitters and witnesses argued that notwithstanding the issues experienced by a significant number of customers, the overall satisfaction level of people connected to the NBN is high.[1]

6.2        There are several sources of information that were brought to the committee's attention that provide some level of data on these issues, namely: data collected by nbn; complaints data from the TIO; and research data from the ACMA.

Customer satisfaction data captured by nbn

6.3        Mr Bill Morrow, CEO of nbn, stated in May 2017 that the overall rate of satisfaction for customers remains positive, with over 85 per cent of end users surveyed by nbn stating that their NBN service meets or exceeds their expectations.[2] Mr Morrow stated:

[T]he vast majority of people using the NBN are satisfied but they are also a silent majority. Nobody calls their local MP or talkback radio to say, 'Everything is wonderful' or, 'Everything is working fine.'

...The people who call are having problems and we acknowledge there are too many. We are working hard on this but remember this is a small percentage of a very large number. Across all of our technologies we average 21 faults for every 10,000 active premises.[3]

6.4        Representatives from nbn explained to the committee that nbn surveys customer satisfaction by asking its end users to rate their satisfaction with their NBN service on an 11 point scale from zero to ten; where response scores of eight to 10 represent customers being satisfied or extremely satisfied, scores of zero to four represent dissatisfied customers, and scores of five to seven represent customers who feel neutral about their service.[4]

6.5        An overall 'net promoter score' for customers is calculated by subtracting the number of dissatisfied customers from the number of satisfied customers to provide a rough idea as to whether the overall response is positive or negative.[5]

6.6        On the issue of nbn's customer satisfaction data and net promoter score,  ACCAN submitted that this information should be published on a regular basis:

nbn surveys all customers on their experience of switching and using the network. They previously released this 'Net Promoter Score' information on a technology basis, but do not release it regularly. If nbn named areas, technologies or RSPs which scored below 7 on the Net Promoter Score it would provide visibility about those that are not having a good experience connecting and using services.[6]

Monthly reports to the Department of Communications and the Arts

6.7        At the public hearing in Canberra in June, the committee questioned officers from the Department of Communications and the Arts (Department) about the information that nbn provides to the Department. However, the Department does receive monthly reports from nbn on the 'consumer experience':

The sorts of metrics [the Department] receive relate to customer satisfaction and how that changes over time ... [the Department] receive periodic reports from the company on other aspects of performance. For example, you have had a range of evidence that goes to the performance experienced on the Sky Muster service. It is well documented that that service has provided an inadequate quality of service, particularly towards the end of 2016 and the early parts of this year. So in relation to that, the government sought a number of briefings from [nbn] to understand the nature of those issues and what steps the company was taking to improve service quality. So through those briefings, there was information provided about the number of faults, the connection time frames and other factors.[7]

6.8        On notice, the Department provided the following summary of the information that it receives from nbn and the frequency of reporting:

NBN Co Limited (nbn) is required to maintain a high degree of transparency and engage closely with Shareholder Ministers and Departments, including by providing monthly progress reports, publishing online weekly progress reporting of network deployment and active services, and delivering quarterly management briefings to the public, including detailed financial and operational information. Additionally, nbn is also required to publish information that details how it has addressed the Government's objectives in the Statement of Expectations, in its Corporate Plan and Annual Report.[8]

6.9        During hearings the committee was informed that the Department did not receive information about faults within the NBN on a regular basis. This evidence was at odds with the Department's response to questions on notice where it advised:

As part of a monthly reporting regime, the Department receives data from nbn on network fault restoration, service fault restoration, connection performance such as right first time activations, and activities undertaken in accordance with service level agreement.

nbn also regularly provides the Department with information on consumer satisfaction which includes data collected regarding connection processes, the network usage experience, and issue resolution.[9]

6.10      Mr Andrew Madsen, Assistant Secretary of the Broadband Implementation Branch of the Department, was not able to confirm whether this reporting by nbn was required, instead stating:

It has been the practice of the company to provide those reports consistently.

...

I think it would be the expectation of the shareholder ministers that the company [nbn] provides those regular reports.[10]

6.11      The committee was advised that the source of the information in the reports that the Department received is 'the company itself, from its own tracking of its performance and reporting from its delivery partners'.[11]

6.12      The committee continued this line of questioning with officers from the Department at the public hearing in Sydney. The committee requested a copy of the most recent monthly report and was informed that it was a commercial-in-confidence document.[12] The committee also asked Mr Bill Morrow, Chief Executive Officer, nbn, if he was able to provide that report to the committee. Mr Morrow deferred to the government to decide whether to make the information available.[13]

6.13      In terms of the information received from RSPs, the Department does not receive formal reports from RSPs, however:

...[the Department] talk to them about their experience of interacting with NBN, the work they are doing in the rollout their services as the NBN is deployed and the products they are developing.[14]

Complaints data collected by the TIO

6.14      In addition to information collected and reported to the Department by nbn, the committee also sought information from the Telecommunications Industry Ombudsman (TIO) about the complaints information collected by the TIO. In particular, the committee explored in some detail what information is collected by the TIO, and whether the TIO should collect more information from complainants in order to provide a better statistical snapshot of NBN complaints and inform the TIO's ability to raise systemic issues that arise with government and the industry.

Information collected by the TIO during complaints resolution process

6.15      The TIO noted that it categorises complaint issues relating to NBN services in broad first-tier categories (including 'connections', 'faults', 'customer service', and 'complaint handling'), and then in second tier categories. The TIO noted that for complaints about services delivered over the NBN in 2016, the highest ranked first tier issues were 'Connections' and 'Faults'.[15]

6.16      The TIO emphasised in its submission that its primary purpose in collecting information from customers is on ensuring there is sufficient information recorded to facilitate resolution of the complaint.[16] The TIO stated:

The primary role of the TIO is to facilitate the resolution of complaints. In accordance with Treasury's Benchmarks and Key Practices for Industry-based Customer Dispute Resolution, the TIO must be efficient, accessible to all Australian consumers, and the dispute resolution processes must be easy to use.

This requires engaging with consumers in a way that makes lodging a dispute easy and shows we are listening. The TIO must not put up barriers or require technical information that the consumer does not have or is unable to provide, for any purpose other than resolving their dispute.[17]

6.17      Ms Teresa Corbin of ACCAN suggested that the complaints data collected by the TIO should be viewed cautiously in terms of using them to draw conclusions about the overall level of issues being experienced on the NBN:

I think that the issue with the TIO statistics is that it is very difficult to gauge from them whether in fact there are issues with NBN. ...I do not think that it is adequate to say that the number of complaints only a small percentage of connections at the moment, because the truth of the matter is we have not reached the peak time for connections. They are likely to increase over the next six months, and then into next year will be the major time when we are connecting a lot of new people. Any one complaint could be a reflection of many other complaints; it is just that nobody spoke up or nobody knew about the TIO. We need to be quite careful with those complaint statistics and be aware that they are really just an indicative thing. They are not going to be comprehensive.[18]

6.18      When asked whether conclusions could be drawn about the performance of particular RSPs based on the number of complaints against each RSP lodged with the TIO, the current Ombudsman, Ms Judi Jones, commented that simply looking at the raw complaints numbers for each RSP would not paint an accurate picture, without also understanding the number of NBN customers each RSP has connected.[19] Ms Jones noted that the TIO does not have access to the data about the number of connections for each RSP that would allow for meaningful analysis of complaints statistics on an RSP basis.[20]

Data on the number of missed appointments

6.19      The Ombudsman also confirmed that the way the TIO records complaints means that the data also could not be used to calculate matters such as the total number of missed appointments by NBN technicians, as one customer's case would be logged as a single complaint relating to the connection of a service, even if the case involved multiple instances of missed appointments with NBN technicians or RSP representatives.[21]

6.20      When questioned whether the TIO should collect information so as to enable visibility of the number of missed or rescheduled appointments, Ms Jones argued that any data provided by the TIO on this issue would be incomplete, and that nbn would have access to the full number of missed appointments across the nbn rollout.[22] The TIO expanded on this issue in response to a question on notice:

Out of a total of 7,948 connection issues reported in the 2016 financial year about services delivered over the NBN, 1,066 involved missed appointments... [T]he TIO does not quantitatively report on the number of visits nor the number of missed appointments. This is because the TIO does not award punitive damages based on the number of missed appointments. Instead, the TIO's focus is on facilitating the connection to be established, in order to reduce the ongoing consumer detriment.[23]

Collecting information about the underlying causes of complaints and responsible parties

6.21      A question raised throughout the committee's inquiry was whether the TIO should collect more information relating to the underlying causes of complaints made to the TIO, including identifying where the issue has originated and who is responsible for addressing it.

6.22      Ms Kathleen Silleri, Assistant Secretary Consumer Safeguards Branch at the Department, stated that the TIO would ideally collect more information about the causes of complaints:

[W]hat we would like the TIO to do is to provide a very accurate picture of what exactly is occurring in the industry. If that means we need to determine exactly who is to blame for an issue that is being experienced by a consumer, and then incentivise that not to occur by either being very public about it or reporting exactly where the problems are, that would be a good thing.[24]

6.23      Ms Silleri noted ongoing discussions between the Department and the TIO on this issue, and explained the importance of getting clearer information on this point:

We are talking with the TIO about how we determine what is actually occurring in a situation where a consumer has lost service, has been unable to get a service or has issues with that service. The first issue that is encountered, generally, from our perspective...is the way that they record the initial contact with the consumer, and it becomes a route that is set from the initial contact. The threshold question then almost determines what the nature of the complaint is. There is also an issue that follows on from that. If the TIO is not understanding exactly what is occurring when a consumer is experiencing a fault, nobody is ever going to get to the bottom of it and be able to solve it for a consumer.[25]

6.24      Ms Silleri elaborated on this point further at a later public hearing in Sydney:

We think the fundamental issue is that, when somebody rings with a complaint, they are ringing to advise what isn't working. From that, the TIO complaints officer goes down a path set by various keywords. It would be useful to think about what is most illustrative of the actual problem in identifying what those keywords would be. For example, I think the TIO at the moment asks whether it's a problem with your landline, mobile or internet. Your internet problem could be caused by your landline or mobile, so that doesn't take you anywhere. It would be very useful to understand whether it's a problem with an existing connection or a new connection and whether or not that problem has occurred as a result of transition to a new network. Those are the sorts of things we would've encouraged the TIO to consider.[26]

6.25      On the issue of identifying who is ultimately responsible for the problems being raised in a complaint, the current Ombudsman, Ms Judi Jones, outlined the complexity in some cases of the TIO trying to attribute responsibility for a complaint to a single party:

[I]n the vast number of complaints we do not find out what the problem was. It is not just the distinction between what the RSP is contributing and what NBN Co is contributing. There may be things related to the wholesaler or aggregator or things on the consumer's premises.[27]

6.26      Ms Jones cited the example of a specific case to highlight the difficulties involved in classifying complex complaints:

A consumer had an appointment to connect to the NBN, and an NBN Co technician arrived. When they arrived they found there were problems with the consumer's internal wiring and so could not connect them. The consumer went away and fixed the wiring problems, and later in that same month an NBN technician attended again. That time the technician identified there were problems with aerial cabling and he did not have the necessary equipment to fix that, so he had to go away. After three more appointments with NBN Co technicians, the work was unable to be completed. There was another technician that missed an appointment, without any information to the consumer about why. So six scheduled appointments and the consumer was still not connected to the national broadband network. In that there is a combination of problems: a problem on the consumer's premises and a problem with the appointment-keeping or work to be done by NBN Co. It is not always easy, even when we do look at those complaints, to say that it was a single fault, that it was the fault of either the RSP or NBN Co.[28]

6.27      Ms Jones reiterated in evidence to the committee that because most complaints to the TIO are resolved simply by referral back to the RSP, the TIO does not generally know the root cause of these complaints.[29] Ms Jones explained that in the majority of cases, while customers do describe the problem that has occurred to the TIO, there will not necessarily be an explanation or understanding in this initial phase of what has caused the problem.[30]

6.28      When questioned whether RSPs should be required after the fact to advise the TIO what the cause of the problem was and how it was resolved, Ms Jones contended that this would add a significant cost to the TIO's operations:

[When] you are looking at 112,000 complaints [in the 2015-16 financial year], to have every retail service provider ring us and get us to record the outcome would be an extraordinary burden on cost. At the end of the day—let's be real—consumers pay the cost even though the members fund the scheme. That would be an enormous impost, I think.[31]

Collecting information by technology type

6.29      The committee discussed with the TIO the issue of whether it could record NBN complaint data by technology type, to enable analysis about any specific issues affecting customers on the different NBN technologies.

6.30      The TIO stated in its submission that it does not routinely record the technology type for the consumer's connection to the NBN. It stated there are a number of reasons for this practice, arguing as follows:

• consumers do not generally know the technology type;
• where the complaint is being made by a referral agency (e.g. financial counsellor), the referral agency is unlikely to know the technology type;
• requiring a consumer to identify the technology type before accepting the complaint would introduce an unreasonable barrier to making a complaint to the TIO – and not align with the TIO's requirement to provide an accessible service;
• it is not necessary to know the technology type to effectively refer complaints to the member for resolution – the vast majority of complaints (90 per cent) are resolved by referral back to the member, without the TIO providing the technology type;
• identifying and recording the technology type would take additional resources, without a clear dispute resolution benefit; and
• it is currently not possible to automate the collection of this information.[32]

6.31      Representatives from the ACMA expressed the view that the TIO should ideally collect information on the technology type underlying complaints made to the TIO, but acknowledged that the TIO's primary purpose was to resolve issues rather than analyse their causes, and that consumers often were not even aware of the technology type in place at their premises.[33]

6.32      Ms Silleri from the Department informed the committee that it would be supportive of the TIO capturing complaints by technology type.[34]

Publication of data collected by the TIO

6.33      Representatives from the ACMA informed the committee that it is provided with raw complaints statistics each month by the TIO on a confidential basis, to allow it to get a sense of emerging issues in a timely manner.[35]

6.34      Ms Jones noted at a public hearing in March 2017 that the TIO only currently publically reports complaints statistics annually, but stated that the TIO is looking at potentially publishing data six-monthly, depending on the availability of rollout data from nbn:

I think it is always important with complaints about services delivered over the National Broadband Network that we do it in context to the rollout. We are just reviewing our reporting across the organisation, trying to look at the effort that is required for the reporting. It might sound like it is just a matter of pushing a button, but there is a lot of checking and quality assurance and giving people guidance on how to interpret the data as well—so effort and value. We expect to have finished that work by the end of March and then we will go back into more regular reporting. With services delivered over the NBN you really need the number of premises connected, and that data is only available publicly from NBN Co in six-monthly batches. We certainly would not be reporting about services delivered over the NBN more frequently than six monthly, unless the data becomes available more frequently.[36]

Ability of the TIO to raise systemic issues relating to the NBN rollout

6.35      Another question discussed in relation to the TIO's data collection was whether it is sufficient to enable the TIO to identity systemic issues arising in the NBN rollout and raise these issues with government and industry.

6.36      The TIO informed the committee that in 2016, seven systemic issues were finalised that related to the actions of RSPs and the NBN rollout, and stated that as at 28 April 2017, there were three possible systemic issues relating to the NBN currently under consideration by the TIO.[37]

6.37      The TIO stated that it does already identify NBN-related issues and work with industry stakeholders to address them. It submitted:

The TIO monitors complaint trends and becomes involved in a range of formal and informal discussions, including on issues about the NBN. The TIO facilitates meetings and discussions between retail service providers and nbn to highlight issues and to exchange information. Members are encouraged to take prompt action to resolve wider issues across the industry.

The TIO also engages with consumer organisations (including ACCAN), regulators (the ACMA and ACCC), representatives of the Department of Communications and the Arts, retail service providers and nbn to support the resolution of issues. The TIO does this by sharing insights to highlight the issues consumers identify and experience in the rollout of the NBN.[38]

6.38      The TIO stated that examples of issues it had raised through these processes include: technician appointment systems; infrastructure difficulties; sales and marketing practices; early termination fees; and the NBN interim satellite service and SkyMuster.[39]

Research on the customer experience by the ACMA

6.39      The ACMA's 2016 research paper Migrating to the NBN—The experience of Australian consumers included survey data in relation to the satisfaction of NBN users in FTTP premises. Nine hundred residents and 304 businesses were surveyed about their experience migrating to the NBN, and further qualitative research was also undertaken for the study. In relation to overall consumer satisfaction with the connection process, the ACMA's findings were as follows:

Most consumers were satisfied (rating of five or above out of 10) with the whole process of connecting to the NBN; however, one in five residents and more than a third of businesses were dissatisfied with the process (rating of four or below out of 10). A key positive factor influencing consumers' satisfaction ratings was the absence of any service disruptions.[40]

6.40      In relation to customers' experience once connected to the NBN, the findings were summarised as follows:

It was evident from the research that, for most consumers, migrating to the NBN met their expectations of having access to a faster, more reliable internet service. For connected consumers, satisfaction with internet speeds was higher than for those not connected. Nearly half of residents (48 per cent) and two-thirds of businesses (65 per cent) expected the NBN to be faster than their current service. Residents were reasonably satisfied with their fixed-internet and landline phone services since connecting to the NBN, with 33 per cent reporting that their internet was more reliable now than before and 51 per cent reporting that their internet service was comparable to their service prior to connecting. Satisfaction levels for fixed-internet services were similar for businesses but slightly lower for landline phone services. A quarter of businesses reported that their internet was more reliable now than before.

The findings suggest, however, that some consumers continued to experience concerns with service reliability, with around one in five reporting that their fixed-internet and landline phone services were less reliable now than before connecting to the NBN.[41]

6.41      At the public hearing in Sydney, Ms Jennifer McNeill, General Manager, Content Consumer and Citizen Division, ACMA, noted that the authority was moving to a new phase of evidence gathering in relation to the NBN:

We will have a new consumer-focused research piece in the field in a few months time. Again, that will be looking at the consumer experience of Australians across a range of technologies as the network's rolled out and, also, smaller and medium-sized businesses. That's the consumer-facing piece. We're also moving to collect more granular information from companies involved in the NBN supply chain so that we can get a better grasp of the extent, the scope and the nature of problems that people are encountering as the network rolls out. Then, we can reflect on that information, together with industry and others involved in government, to see what might be done to make the experience as positive a one as it can be for consumers.[42]

6.42      In answer to a question on notice from the 23 June hearing in Canberra, the Department provided some further information on how the research by ACMA would be carried out:

....[the] Australian Communications and Media Authority (ACMA) will conduct research and collect data from businesses across the NBN supply chain using its powers under the Telecommunications Act 1997. Twenty-one industry participants including retailers, wholesale providers and nbn will receive notices seeking a range of data on issues such as fault handling, connection timeframes, and appointment keeping.[43]

Committee view

6.43      In the committee's view there is clearly significant data being collected in relation to the customer experience in relation to the rollout of the NBN. However, the committee considers that the information made publically available relating to customer satisfaction on the NBN network does not enable adequate analysis and evaluation of the overall customer experience.

6.44      nbn's customer satisfaction metrics provide a much greater sample size than any other available data set. While a headline figure on end user satisfaction ratings is generally included in nbn's Annual Reports and Corporate Plan, this level of disclosure does not allow for any rigorous analysis of the data. The committee considers that the regular publication of nbn's end user satisfaction metrics would provide much needed transparency about the overall level of satisfaction in the NBN, and would provide context for the broader public discussion around the success of the rollout from a customer perspective.

Recommendation 19

6.45      The committee recommends that nbn publish prominently on its website, monthly information relating to its end user satisfaction metrics, including:

• its overall net promoter score as measured each month;
• the overall net promoter score for each technology type as measured each month;
• relevant disaggregated information about end user satisfaction metrics in relation to each RSP; and
• any relevant disaggregated information about end user satisfaction metrics in specific geographic areas, such as:
• data broken down by state and territory; and
• data relating to each fixed-line area in the rollout footprint, as areas are designated Ready for Service.

Data collection by the TIO

6.46      The committee considers that there is considerable scope for the TIO's data collection activities to be enhanced in order to enable its statistics to become a much richer source of information in evaluating the performance of the NBN. The committee acknowledges that it is not possible in every instance for the TIO to collect information such as the NBN technology type at the complainant's residence, however it is important that such data be collected wherever possible.

6.47      The committee notes that both the Department and the ACMA expressed the view that the TIO could collect data in a more robust fashion to provide a greater level of information on the NBN. The committee awaits to see how these issues are dealt with by the current independent review of the TIO, and the committee will provide further monitoring of this issue in its future work.

Recommendation 20

6.48      The committee recommends that the scope, function, and operation of the Telecommunications Industry Ombudsman (TIO) be expanded so that, among other improvements determined through the current review process, the TIO should keep data according to technology type, and should record and report multiple issues as separate items, especially where nbn and an RSP are both involved.

6.49      The committee welcomes the additional research work that will be undertaken by the ACMA, examining the NBN consumer experience across all stages and technology types. This will provide valuable, in-depth qualitative data on these issues.

Provision of information to the committee

6.50      The NBN rollout is the largest public infrastructure project in Australia which touches every premises in the country. The significant conjecture about the speed, efficiency and effectiveness of the rollout underscores the importance of transparency and disclosure.

6.51      The committee notes the refusal of the Department to provide the committee with the monthly reports provided to the Department by nbn, despite the committee's offer to accept this material in confidence. The committee believes that the information contained in nbn's monthly reports would greatly assist the committee in its inquiry and strongly encourages the government to reconsider the publication of this information.

Recommendation 21

6.52      The committee recommends that the Department of Communications and the Arts publish the data it receives from nbn as part of its monthly reporting regime, including data relating to:

• network fault restoration;
• service fault restoration;
• connection performance, such as right first time activations; and
• activities undertaken in accordance with service level agreement.

Navigation: Previous Page | Contents | Next Page