Spinning the Election

David Skillicorn and Ayron Little, School of Computing, Queen's University.

With the Canadian election coming up, we thought we'd use the techniques we've developed to analyse how mental states leak into speech and text to look at how the political party leaders are doing. Can we trust them? Here's a look at how software analysis can help you decide.

James Pennebaker, from the University of Texas at Austin, developed a model for deception in text. The model is based on the relative frequency of certain kinds of words.

The Pennebaker model predicts that deceptive text will be marked by:

• A decreased frequency of first-person pronouns, perhaps because of a speaker's attempts to distance himself or herself from what's being said;
• A decreased frequency of exception words, such as `however' and `unless', perhaps to keep the story simple;
• An increased frequency of negative emotion words, perhaps because of some instinctive distaste about deceiving; and
• An increased frequency of action words, perhaps to keep the story going so that inconsistencies might not be noticed.

We applied this model to a large collection of email to and from Enron employees in the three and a half years before the collapse of the company. We discovered that, while the model detects deception, it also detects other kinds of unusual text. The best way we know to categorise what the model detects is spin -- text whose apparent meaning isn't the true beliefs of the person saying or writing it.

Politicians, above all, serve multiple masters and may find themselves saying things that may not quite be what they would say if they could speak their minds. We've applied this model to speeches given by the three English-speaking party leaders: Paul Martin, Stephen Harper, and Jack Layton. We rank their speeches by how well they fit the model -- how much spin the speeches contain.

We collected 20 speeches by Stephen Harper (numbered 1 to 20); 20 speeches by Paul Martin (numbered 21 to 40); and 10 speeches by Jack Layton (numbered 41-50). Since the model was developed for English texts, we could not apply it to speeches by the Bloc.

The Table below is a list of the speeches in increasing order of spin. Speeches are colour coded: blue for Conservatives, red for Liberal, and orange for NDP. You can click on the links to see the text of the speech -- see if you agree with the spin rating.

Text	Spin Score
Speech 36	294
Speech 35	247
Speech 37	201
Speech 28	190
Speech 27	185
Speech 34	180
Speech 45	173
Speech 30	136
Speech 23	132
Speech 48	128
Speech 31	123
Speech 22	118
Speech 21	116
Speech 12	113
Speech 18	108
Speech 7	107
Speech 2	104
Speech 46	104
Speech 10	100
Speech 1	99
Speech 17	96
Speech 50	95
Speech 32	94
Speech 33	93
Speech 9	93
Speech 15	84
Speech 19	83
Speech 47	82
Speech 3	81
Speech 29	79
Speech 13	75
Speech 8	74
Speech 44	73
Speech 16	73
Speech 39	73
Speech 20	72
Speech 25	71
Speech 43	70
Speech 4	67
Speech 26	66
Speech 24	64
Speech 41	58
Speech 11	56
Speech 5	56
Speech 42	55
Speech 14	47
Speech 49	38
Speech 6	31
Speech 40	30
Speech 38	27

Average spin score per speech: Liberals: 124; Conservatives: 73; NDP: 88.

Technical Stuff

To produce these results, we counted the frequency of 88 words derived from Pennebaker's deception model, and divided the counted frequencies by the length of the speech in which they appeared (so that long speeches did not appear more spinful just because they were longer). The columns of the matrix were zero-centred, and those columns corresponding to words where a reduced frequency is significant were negated (reversed around the origin). A singular value decomposition was used to create a perceptual space for both speeches and words.

This figure shows the perceptual space for speeches.

The further a point is from the centre point (origin) of this plot, the more spinful it is. The spin scores in the table above are based on these distances. Notice, though, the directions are important too. Most of the most spinful speeches are in the left hand region of the plot.

We can learn about the role of the words in the deception model by considering a perceptual space of words, like this:

Words that appear close together play a similar role. Words that are plotted in different directions play a different kind of role. Notice that the four different kinds of words in the deception model don't appear in four different directions, indicating that the model is describing something more than just four distinct aspects of deception. As before, the further a word is plotted from the centre of the plot, the more important it is.

We've used these ideas in areas such as organizational behaviour, fraud detection, detecting deception, and detecting potential terrorist communication. You can find out more at David Skillicorn's web page

David Skillicorn is a Professor at Queen's. Ayron Little did this work as a fourth year student, and went on to look at deception in the testimony to the Gomery Commission in her Master's research. We do not yet completely understand these models of word use, so the results here should be taken with a grain of salt. An earlier version of this page had the speeches in the reverse order.