This Research Note is a companion to a series comparing the results of the major polling organisations: Roy Morgan Research [Morgan], Newspoll and ACNielsen on voting intention and, where available, the importance of issues. The most recent editions can be found at: http://www.aph.gov.au/library/pubs/rn/2003-04/04rn25.pdf and http://www.aph.gov.au/library/pubs/rn/2003-04/04rn24.pdf. This Note provides background information on polling methods and highlights potential pitfalls in interpreting results. Such information may be especially useful in what is expected to be an election year, when polls are seen as particularly salient.

Lies, damned lies and polls

Today, the public is polled from many different angles on a wide range of issues. Results are highlighted in newspaper, magazine and television reports. Polling methods vary from questioning randomly selected respondents in telephone interviews to tallying the numbers of self-selected respondents who call in, or click a response button on a web page. Poll results are widely regarded as an accurate gauge of the publics mood. Apparent alterations to policy after results are published have led to accusations that todays politicians are opinion poll-driven rather than policy-driven.

Some of the most keenly watched polls, especially in the months before an election, are those on party support, leadership and political issues. In fact, the concern about the impact of such opinion polls on voting behaviour has led some countries to ban the publication of polls immediately before elections, despite evidence of the impact of polls on vote choice being slim.⁽¹⁾ Between elections, opinion polls are used to assess party leadership and policy proposals. Parties may remove leaders who are unpopular in the polls, even if the leaders are popular with their party colleagues.

However, despite the emphasis that the media and, arguably, politicians place on poll results, an important question is whether opinion polls, in fact, tell us anything useful.

An American academic who specialises in polling, James Fishkin, criticises ordinary polls on the grounds that they measure only off the top of the head responses to questions to which respondents have given little thought. He claims the only useful poll is a deliberative poll, in which respondents are taken aside (often for a day or a weekend), exposed to the complexity of an issue, and then asked for their considered opinions.⁽²⁾

Deliberative polls may answer criticisms such as those eloquently summarised by a New York Times commentator who wrote of being polled on the 2003 war in Iraq:

Regardless of criticisms as to their utility, opinion polls have become staples of contemporary political reporting.⁽⁴⁾ That said, there are important methodological aspects of polls that are seldom reported in detail.

Reporting essential details

The Australian Press Council has guidelines outlining the details that should be published in opinion poll reports. These include: the identity of the poll sponsor (if any) and the name of the polling organisation, the question wording, the sample size and method, the population from which the sample was drawn, and which of the results are based on only part of the sample (for example, male respondents).⁽⁵⁾ The council also suggests that reports include how and where the interviews were held as well as the date of the interviews.

It notes that space reasons may restrict the number of details that are published, but it argues that, where a poll has a marked political content, more information is needed. It adds: The public needs to be able to judge properly the value of the poll being reported.⁽⁶⁾

Macquarie University poll analyst Professor Murray Goot notes that, although no complaint under these guidelines has been registered with the Council since 1996, some studies point to a media performance that leaves much to be desired.⁽⁷⁾

Polling methods and pitfalls

Some of the details listed above are essential to understanding a poll, especially whether its results are useful. The margin of error (or sampling error) is an oft-overlooked part of polling that can have significant effects on the utility of results, especially those that are within a few percentage points of one another. [Note the difference between per cent and percentage point. An increase from 40 per cent to 50 per cent, for example, is not an increase of 10 per cent (10 per cent of 40 is four, which would take the initial figure to 44 per cent); it is an increase of 10 percentage points. This is a common reporting error.] Generally, if respondents are selected at random and are sufficiently numerous, then their answers will deviate only slightly from those that would have been given if every eligible voter had been polled. The margin of error is the maximum likely difference between the poll result and that of the voting population at large.

Australias major polls are of randomly selected samples large enough to have a relatively low margin of errorabout plus or minus () 3 percentage points or lessand a high confidence level. As one columnist has summarised:

It is the possible variation in the resultsthe spread of 48.5 to 53.5that highlights the need for caution when interpreting poll results. For example, where a result is given as 51 per cent support for a party in a poll with a 3 percentage points margin of error, it is not accurate to claim that more than 50 per cent of voters support that particular party because the actual support result ranges from 48 per cent support (that is, minus 3 percentage points) to 54 per cent support (plus 3 points).⁽⁹⁾ Unfortunately, only rarely do the media highlight this limitation. As one observer notes: Editors dont let the statistics get in the way of a good headline.⁽¹⁰⁾

Other questions about the validity of voting intention polls relate to disparate results. That is, if polls are a fairly accurate map of broader public opinion, and the polls are of similar-sized groups of randomly selected people, then we could expect both that the results of polls would be roughly the same and that they would roughly match election outcomes. Yet the major polls do not always produce the same results, despite similarities in their methods. Nor do polls always accurately predict election outcomes.

Explaining disparate results

Several factors may contribute to different poll results. It may be that the pollsters use different calculations to weight their samples to reflect the population.⁽¹¹⁾ In addition, the pollsters may take different approaches to dealing with dont know (uncommitted) and non responses. That is, one polling organisation may exclude from its calculations the responses of those who do not answer or who say they dont know while another may allocate them according to the respondents political leaning (for example, those dont know respondents who identify with Labor are assigned to Labor).

Both exclusion and allocation have problems in terms of creating potential differences between poll results and election results. Discarding the uncommitted responses and recomputing the percentages based on definite answers assumes that the undecided will cast their votes as the more committed voters do. In systems where voting is not compulsory, excluding such voters also may assume that they do not vote. In both cases, the assumption is that these voters make no difference to the result. Urging uncommitted respondents to select a party, or assigning these responses on the basis of party identification or leaning, assumes that undecided voters will come home and vote for that party at an election.

The major pollsters take different approaches on this issue, with Newspoll and Morgan excluding those who do not name a party and ACNielsen redistributing them.⁽¹²⁾ Both Newspoll and Morgan try to limit the number of those excluded by asking those who say they are uncommitted to name the party they lean towards. Thus, those who effectively dont know are urged towards choosing a response even if they are not sure.

A problem with urging those who dont know to nominate a party is the assumption that all voters identify with a party strongly enough to vote for it at an election. However, election specialist Professor Ian McAllister has shown that, although most voters still identify with a party, more now have no party attachment or are less attached to their party than previously.⁽¹³⁾

McAllisters figures show a threefold increase since 1987 in the number of voters who do not identify with a major party (from 5 per cent of respondents in 1987 to 15 per cent of respondents in 2001) and a substantial decline in the strength of party identification (in 1979, 34 per cent of respondents had very strong identification with their party; in 2001, only 18 per cent had such a strong attachment).

Thus, it cannot be assumed that dont know respondents will vote for the party they lean towards or the party for which they voted previously. Assigning them on this basis could distort poll results vis--vis election results.

Some additional pitfalls

Other factors that may affect poll results can be discussed in the context of the following example in which two of the major pollsters had significantly different results in their pre-election polls. A week before the 2001 election, Newspoll and Morgan gave the following first-preference results:

Newspolls figures were close to the election outcome (Coalition 42.7 per cent; Labor 37.8 per cent), but the Morgan poll was incorrect by a considerable margin. This is not to argue that Newspoll is necessarily better at polling than Morgan. In fact, a recent academic comparison found that, historically, Morgan outperformed Newspoll, and that, at the 2001 election, ACNielsen outperformed both heavyweights.⁽¹⁴⁾ The same article noted that, over the longer term, election betting was a better predictor of election results than opinion polls.

Putting that aside, how can we account for the disparity between polls on the same issue? Ultimately, it is impossible to explain with certainty, although several factors may contribute. One factor may be that discussed above: urging those who dont know to nominate the party they lean towards. These leaners may have responded differently to each pollster. Another factor may be margin of error (roughly 3 percentage points in each poll). This does not help Morgan because subtracting 3 points from Morgans result for Labor still shows Labor winning (and adding 3 points makes it a Labor landslide!).

Other factors that may explain disparate results, such as different timing of interviews or different question wording, do not help here because both polls asked the same thing at the same time. The polls use different methods to gather data, which may have some effectNewspoll uses telephone interviews while Morgan uses face-to-face interviews. Those in the polling industry disagree as to how the different techniques affect results, with advocates of face-to-face interviews arguing that these are more accurate because people find it harder to lie or evade when asked directly. In addition, some argue that:

Morgan Researchs executive chairman, Gary Morgan, argues that his organisations telephone interviews gave dramatically different results to the face-to-face polls conducted at the same time, with the telephone polls having the Coalition winning by a landslide.⁽¹⁵⁾ He says Morgan Research relies on face-to-face polls because telephone polls in Australia and overseas have a bad record of being biased toward the party or candidate that electors believe will win.⁽¹⁶⁾

However, Newspolls Sol Lebovic and ACNielsens John Stirton dispute the claim that telephone polling is less accurate, arguing that there is evidence that the manner and appearance of face-to-face interviewers can have an adverse effect on respondents.⁽¹⁷⁾ In addition, some researchers argue that:

Academics are undecided on this issue, with Goot noting there is little evidence either way.⁽¹⁸⁾

Morgans own explanation for the results, put forward immediately after the election, was that the electorate changed its mind in the last week as to how it would vote. Morgans executive chairman, Gary Morgan, notes that re-interviews after the election of nearly 400 voters who had intended to vote Labor before the election showed that 20 per cent had changed their minds in the last days of the campaign. This argument fits with that made above about the decline in strength of party identification; that is, fewer voters today follow the precept, my party right or wrong. Rather, they are more prepared to vote for any party that matches their own views on a particular issue.

Morgan claims that this is what happened with those Labor supporters who had intended to vote for Labor but who then voted for the Greens or the Democrats. These voters were clear in their change being due to the major parties response to boat people.⁽¹⁹⁾ He highlights the Tampa crisis and the September 11 terrorist attacks in the United States as turning points in the fortunes of the Coalition government. Whereas the goods and services tax had been a continuing problem for the Howard government, in the midst of the war on terrorism, the GST was no longer top of mind for the uncommitted voters, he says. That is, although voters were concerned about this issue, they were more concerned about other issues, such as security.

Election analyst Antony Green had predicted that asylum seekers and security would decide the election, despite polls showing that health, education and the economy were the top issues in voters minds. As Green noted:

Asylum seekers and defence may not rate highly in polls, but in terms of the space they occupy in the election campaign spotlight, they crowd out other issues and become bound up in the leadership perceptions of Mr Howard and Mr Beazley.⁽²⁰⁾

This means that the important poll results were not those highlighted in Newspolls poll on the importance of issues at the time; rather, the important results were those on the party preferred to handle those issues at the forefront of the campaignimmigration and defenceon which the Coalition was scoring higher than Labor.

Thus, the prominence of an issue at the time, as well as the perceived party differential on that issue, may have more effect on how people cast their votes on polling day.

The nations pulse

None of the factors mentioned above offers a definitive explanation of the significantly different poll results before the 2001 election. For example, Morgans claim that voters changed their minds in the last week does not account for Newspolls accuracy a week before the election. In fact, Goot notes that the Newspoll and ACNielsen polls in the last week showed little sign of movement, and concludes that Morgans explanation is not plausible and is not supported by other polls.⁽²¹⁾

The arguments in this Note highlight perhaps the most important point about opinion polls: that polling is not an exact science. In the words of American humorist E. B. White:

Ultimately, as many have observed, the only political poll that matters is the one taken on election day.

Endnotes

1. France bans the publication of polls one day before an election, Canada two days and Italy 15 days. For discussions of this debate, see W. Donsbach, Public opinion polls: legal regulation, in R. Rose, (ed.), International Encyclopedia of Elections, Macmillan Reference, Ltd, London, 2000, pp. 24647, and F. Spangenberg, The Freedom to Publish Opinion Poll Results, Freedom for Information, Amsterdam, 2003, http://www.unl.edu/WAPOR/Opinion%20polls%202003%20final%20version.pdf. For a brief discussion of the impact of polls on voters, see R. Worcester, Public opinion polls: how they work, in Rose, ibid., pp. 24546.
2. In Australia, deliberative polls have been held on whether Australia should become a republic, on reconciliation, and on an ACT Bill of Rights.
3. W. Kirn, Dont count me in, New York Times, 6 April 2003.
4. R. Welch, Polls, polls and more polls, Press/Politics, vol. 7, no. 1, Winter 2002, p. 102.
5. For the full guidelines, see Australian Press Council, Reporting guidelines, Press Release, no. 246 (iv), July 2001, http://www.presscouncil.org.au/pcsite/activities/guides/gpr246_4.html.
6. ibid.
7. M. Goot, Reporting the polls, in S. Tanner (ed.), Journalism: Investigation and Research, Longman/Pearson Education Pty Limited, 2002, p. 243. He observes that some media outlets are better than others in the details they report.
8. See P. Brent, Lies and statistics, Australian Financial Review, 24 May 2003.
9. In a two-party result of 4951 with a 3 point error, the results range from 46 per cent to 54 per cent, meaning there is a potential 8 percentage points between the parties.
10. Brent, op. cit.
11. Weighting is the adjustment made to results to ensure that the survey represents the distribution of the characteristics of the population, such as age or gender.
12. The Australians reports of Newspoll note in small print beneath the primary vote table the percentages of uncommitted and refused that have been excluded. For the poll reported on 23 March 2004, these figures were: 4 per cent uncommitted and 4 per cent refused. See D. Shanahan, Latham on equal footing with PM, The Australian, 23 March 2004. Roy Morgan Research notes in its web-site reports the percentage of those polled who did not name a party. For the poll reported on 27 March 2004, the figure was 5 per cent. See Roy Morgan Research, Finding No. 3722, 27 March 2004. Note the level of fluctuation: 6.5 per cent on 11 March 2004, but 4 per cent on 11 October 2003.
13. I. McAllister, Political parties in Australia, in P. Webb, D. Farrell and I.  Holliday, Political Parties in Advanced Industrial Democracies, Oxford University Press, Oxford, 2002, p. 388.
14. J. Wolfers and A. Leigh, Three tools for forecasting federal elections: lessons from 2001, Australian Journal of Political Science, vol. 37, no. 2, p. 237.
15. Gary Morgan says: Our interpretation has always been that telephone polls measure the mood or the emotional response to an issue, whereas face-to-face polls measure the more considered responseand that voting is best measured by capturing the electorates considered responseas their vote on the day will be a considered one. See Roy Morgan Research, Finding No. 3472, 13 November 2001.
16. ibid.
17. P. Clark, Whose figures are right depends on your opinion, Sydney Morning Herald, 11 October 2001.
18. ibid.
19. Roy Morgan Research, Finding No. 3476, 27 November 2001.
20. A. Green, Flawed polls create a smokescreen, Sydney Morning Herald, 8 November 2001.
21. M. Goot, Turning points: for whom the polls tolled, in J. Warhurst and M. Simms (eds), 2001: The Centenary Election, University of Queensland Press, St Lucia, 2001, p. 86.
22. E. B. White, in the New Yorker, 13 November 1948, in E. Knowles (ed.), The Oxford Dictionary of 20^th Century Quotations, Oxford University Press, Oxford, 1998, p. 326.