Please support our coverage of democratic movements and become a supporter of rabble.ca.
“Will you come with me and linger? And discourse with me of those, secret things the mystic finger, points to, but will not disclose?” — Mervyn Peake
There is an old saw that says that the only poll that matters is the one conducted on election day. Furthermore, former Canadian Prime Minister, John Diefenbaker, famously quipped (with a deliberate homonymic pun) that “dogs know best what to do with polls.” Both of these are not uncommon sentiments that come to the fore during election campaigns — particularly when the polls indicate something that we would rather not hear. However, beyond the dismissive and derisive, is there anything useful to be gleaned from the mysterious art of political polling?
Polling: The case against
Polls are typically criticized for two principle reasons:
1) They are inaccurate and/or unrepresentative; and
2) They distract from more important matters (such as consideration of policy issues) and/or turn political campaigns into popularity contests, inclining those with short political attention spans to vote for a “popular” party/leader/candidate rather than on the basis of some more substantive consideration.
The concept behind opinion polls is that a representative number of people are asked something and the ratio of their replies is an indicator of the prevalence of those views in society at large. In this respect, the principle differs little from many empirical scientific pursuits, where a limited number of samples are taken (or trials conducted) and the results are assumed to — more or less — represent the population at large.
There are a host of procedural issues that determine if this is a reasonable assumption. Have a sufficient number of people been polled so as to rule out statistical variation? Are the questions fair and impartial or do they lead a respondent in a particular direction? Is the subgroup that has been polled demographically representative of the population at large? Are there methodological issues that would tend to favour some constituencies at the expense of others? Are people responding honestly to the question(s) or could they be motivated to conceal their true inclinations?
All these are non-inconsequential issues. Without going into mind-numbing detail, suffice it to say that all credible national pollsters (i.e., Nanos, Abacus, Ekos, Léger, Forum, Ipsos, Environics, Angus Reid, Innovative, etc., as well as poll aggregation analysts such as CBC’s Poll Tracker and Threehundredeight.com) are aware of and have grappled with these issues and make credible efforts to address them. There is no perfect methodology or technology and there are strengths and weaknesses in various approaches. What is important is to be clear and precise about how polling is conducted and to assess its margin of error of error (MOE) and clearly state it. The MOE is typically given as being accurate to within “X” percentage points 19 times out of 20. Again, in this, polling is little different from other empirical scientific research that also produces results that are subject to a margin of error.
Note that in opinion polling a random sample of 1,000 people is apt to produce a result with an MOE of approximately three percentage points 19 times out of 20. One can improve this to approximately one percentage point 19 times out of 20 but this requires polling approximately 10,000 people, typically far beyond the effort of what pollsters are in a position to undertake. Results with large margins of error or in which small numbers of people have been polled should be regarded with considerable skepticism.
That said, such accuracy applies only 19 time out of 20 and inevitably there will be one occasion out of 20 when the variance is greater. This is simply a consequence of variability in the world, and there is no way to avoid such occasional stochastic “outliers.” Don’t expect perfection from opinion polls. Perfection, however, is not what we should be expecting from anything in the world, and so the imperfect and that qualified by margin of error does not mean that polling is without merit or use.
Even more skepticism should be applied to polls that are “self selecting,” in other words where the participants themselves determine who is included (for example, most Internet polls that ask those interested to “vote”). These do not represent a random selection of people and can easily be dramatically skewed.
Are polls democratically harmful distractions?
There is a school of thought that argues that polls, particularly the obsessive focus on the “horserace” component of political polling, are harmful distractions foisted upon the electorate by a media incapable of critical policy examination and hence reduced to simplistic “who is leading” metrics. And there is no doubt that this is sometimes true. There are all manner of media organizations, and some of them are of a remarkably shallow disposition. It is simpler to focus on the horserace rather than delve more deeply into the merits of the policy propositions that are put forward by the parities. By focusing on “numbers” it can also convey the false impression of empirical certainty on a subject that is clearly malleable.
And there is no doubt that this critique has validity and that such considerations may incline those of feeble political cognition into climbing on board the wagon of whoever is in the lead at the expense of more substantive political thinking.
That said, for many people, political polls may be largely irrelevant. If you have been paying attention you will likely know the policies that you support and which political party best represents those. Once the election is called you can either enjoy the midway sideshow, or else tune out, go to the ballot box, and indicate your preference, paying no attention whatsoever to what the polls prognosticate.
Those most frequently opposed to paying attention to polls are those who are concerned about their political impact, i.e., that they may motivate strategic decisions on the part of the electorate (for more information on strategic voting see: The art and science of strategic voting). Suffice it to say that, certainly under a first-past-the-post electoral system, there are legitimate reasons to think strategically in terms of voting. Strategic decisions play an important part adjudicating many decisions in our lives and politics is no exception to the rule. To dismiss strategy as a legitimate political consideration in voting is to express a political view in and of itself — one that not everyone subscribes to.
That said, polls are not the only elements of herd mentality that are employed in election campaigns. Take the ubiquitous election signs that pop up all over lawns during election campaigns. Extolling us to vote for “candidate X” representing “party X” they have so policy substance; their intention is to try and outshout by sheer volume the signs of other parties and hence create the impression that more people support the aforementioned “party X.” Political advertising frequently has marginal substance but instead relies on endless repetition on television or other media to hammer home a simplistic message bereft of meaningful political content, i.e., “Justin Trudeau is just not ready.”
What do we wish to learn?
Polling is predicated on asking questions, but before we can find satisfactory answers we need to be clear on what information we are seeking. A superb study that illustrates why this is important is Frank Graves’ masterful analysis for Ekos Politics, Accurate Polling, Flawed Forecast , an analysis of Ekos own polling and forecasting during the 2011 federal election. I recommend this to anyone interested in understanding the complex of motivations and methodology that combine in opinion polling.
In short, Graves was seeking to understand the disparity between Ekos voter intention polling numbers in the 2011 election campaign and the actual election results. Graves found that, to an important degree, the difference arose because younger voters, who were preponderantly more progressive in their political inclinations, were also markedly less inclined to actually vote. So, while Ekos’ polling took great pains to try and be maximally representative of the Canadian public’s age demographics in charting voter intentions, actual voter turnout differed from this quite markedly. Not having taken this fully into account therefore lead to a flawed forecast.
This finding has very significant political dimensions, some of which are indicated in Graves’ commentary:
The focus on the gap between the final polls and the election outcome reduces the role of the pollster to the monitor of the horserace. In some respects, the final polls are the least important for anything other than assigning the vanity points to pollsters who had the smallest gaps to final results. As we have argued, in cases where there is a profound systematic difference between the actual and eligible voters, this may be a very poor assessment of polling accuracy in any traditional scientific sense. If we find that we come closer by ignoring large subpopulations then is the role of the pollster only to forecast the final result?
In addition, if there is a widening gulf between the voting and non-voting populations, where do we hear the voice of the non-voter? Can we just say that this large group — nearly 40 per cent of all voters and over half of under 50 Canada can be safely ignored? What if the factors producing non-voting are not simply “laziness and apathy”? What if the lack of voting is linked to alienation and conscious political strategies designed to suppress the interest of those voters? What if there is a mutually reinforcing tendency to further weaken “next Canada’s” interest in federal government by virtue of a federal agenda which systematically undervalues and de-emphasizes their interests and values and emphasizes the interests and values of its constituency? Does this become a particularly troubling problem at a point where our highly unusual demographics have produced a voter whose median age is around 60? Are we fashioning the future Canada in the image of those who are disengaging or those poised to inherit the positions of authority and influence in short order?
… Elections are rare periods of collective public judgement which should serve to elucidate important things about the societies in which they unfold. These are important roles for the pollster which are not merely connected to the closeness of final results.
Are we simply interested in forecasting election results as accurately as possible, or are we interested in knowing the views of the citizenry as completely as possible? Graves’ analysis shows us that these are not the same questions, and how polling is approached and the results are reported differ depending on what it is that we wish to learn.
Polling: The case for
A poll result represents a point in time. Individual poll results are much less useful than a series that represents how attitudes are changing. Polls are like barometers. They can provide useful indications of how the political weather is changing.
For political activists such information can be useful in understanding what is working and what is not in terms of how the political message is being received by the electorate. If a barometer indicates that a storm is approaching this is useful knowledge; one can take attempt action accordingly. So it is with opinion polls. If, for example, poll results indicate that another Harper majority government may be in the offing, it is useful for opposition party activists to be alerted to this possibility.
However, like barometers, they are not infallible. The weather they forecast may — or may not — come to pass. Under no circumstances should one regard poll results with fatalism. “It ain’t over till its over,” muttered the recently and dearly departed master of the inscrutable, Yogi Barra. Ergo, unfavorable poll results should neither dissuade one, nor should favourable ones make one complacent.
The most useful elements of polls are most frequently those that receive much less attention in comparison to the up-to-the-minute horserace stats. Examine the reports of most pollsters and you will find a wealth of information on second choice voting preferences and hence on theoretical party ceilings; on age, gender, educational level, and other demographic characteristics of partisan support; on perceived competence (approval versus disapproval) of various prime ministerial candidates; on the fluidity of voting intentions; satisfaction with the direction of the government and country; the comparative importance of various political issues; regional support of parties; attention to debates, and many other questions that pollsters elect to ask citizens.
These may or may not be of interest to an average voter but to party campaigners, social, political, and environmental activists, writers, journalists, pundits, and political scientists, such polling data is a goldmine of information about the Canadian body politic. If one wants to understand what motivates or concerns people, what they pay attention to, how various aspects of their demographics and geographics factor into their beliefs — all of this is of immense interest and utility. How is Canada evolving as a nation in terms of the concerns of its citizens? What political approaches resonate or do not? What sorts of things might be achievable politically? Polling data sheds light on these and many other such similar questions.
If politics is about choice — the choices we are prepared to make that reflect what we believe is of value in a society — then polling information helps us discern the political lay of the human landscape. In my perusal of such data I am sometimes excited by the possibilities of, say, a younger demographic making social, economic, and environmental choices in the future different than their predecessors. At times, it is to despair, when the fickleness of the electorate becomes apparent, or how seemingly trivial issues come to assume great importance. But love it or hate it — this is it. As activists, if we want to create change we need to understand where we stand. Such knowledge is power.
Christopher Majka is an ecologist, environmentalist, policy analyst, and writer. He is the director of Natural History Resources and Democracy: Vox Populi.
Please support our coverage of democratic movements and become a supporter of rabble.ca.