Public Policy Polling: "Do you have enough Santorum in your life, or not?"

November 14, 2012 3:28 PM

How Polling Firm PPP Won The Election With Its Hilarious And Infuriating Questions: "Public Policy Polling, the firm that correctly predicted all 50 states in the presidential election, is known for asking some weird, quirky and, sometimes, controversial questions in its polls... Here are some of the firm's best questions of the election cycle."

More from Business Insider - How A Three-Man Polling Team Completely Nailed Their Election Prediction:

While Nate Silver got a lot of attention for predicting the election through poll aggregation and statistics, Public Policy Polling nailed every state through first-hand polling... The firm conducted about 255 polls this election season — including 19 polls in the final four days. It went 50 for 50, and its 50-47 popular vote prediction is coming closer to fruition as all the ballots are tallied. Last week, PPP was rated the election's most accurate pollster. "It feels great," PPP director Tom Jensen told Business Insider. "It vindicates that we are making the right assumptions about the electorate." They didn't do it without receiving their fair share of criticism — because, like Silver, they did not try to hide the fact that they were openly rooting for President Barack Obama to be re-elected.

The Hill - Study finds PPP was the most accurate pollster in 2012:

A study conducted overnight found liberal-leaning pollster Public Policy Polling was the most accurate predictor of the 2012 presidential election... The analysis by Costas Panagopoulos, Associate Professor of Political Science at Fordham University, also found that none of the 28 major polling outlets he analyzed showed bias toward either candidate in the national polls. “For all the derision directed toward pre-election polling, the final poll estimates were not far off from the actual nationwide vote shares for the two candidates,” he wrote.

NPR - North Carolina Polling Firm Spotlighted During Primaries:

Just a few years ago, not many people outside North Carolina knew of a small start-up called Public Policy Polling. It was founded by a Democratic businessman with no political experience. These days, PPP is one of the most prolific polling companies in the country, along with Gallup and Rasmussen. From North Carolina Public Radio, Jessica Jones has the story of a political start-up turned success story...

JONES: Debnam is a soft-spoken, jeans-clad entrepreneur who runs two other successful companies. He started PPP after he got tired of seeing local Republican polls he didn't think were accurate.

DEBNAM: I decided that the highest and best use of my time was to invest in something that made Democrats more competitive in races. And one of the things to do was to make polling more affordable.

JONES: Traditional pollsters like Gallup still use real people to call prospective voters, but that's expensive. So Debnam set aside company phone lines for automated calls. PPP's polls cost as little as $1,500 for clients who range from local school board candidates to big Democratic interest groups...

JONES: Jensen enjoys putting together wacky polls too, including a survey last year asking independent voters whether they'd support Sarah Palin or Charlie Sheen for president. Charlie Sheen won. But PPP hasn't neglected its original client base of Tar Heel Democrats. Alan Norman won his first election as a rural sheriff two years ago. He says Tom Jensen's polls helped him identify crucial voters.

ALAN NORMAN: He said, you know, here's your weak area and here's where you need to go. And he gave us the numbers. He said you need to zero in and work harder on 65 and older, retired individuals, such as my parents.

JONES: Norman says, by following PPP's advice, he won the public office he'd always coveted. And Public Policy Polling got what it wanted too: another Democrat in public office.

Criticism of PPP's "push poll" questions:

• The Daily Beast - Obama’s a Muslim? PPP Polls in South Appear Designed to Bring Out Bias:

  ...every state has its not-so-admirable biases. And asking Republicans in these particular states—and evidently only these states—about these particular issues smacks not so much of political research as cultural profiling.

• Betsy's Page - PPP's Clown Question:

  I was polled on Sunday by PPP about politics in NC and, for the last question, they asked "Who was more responsible for killing Osama bin Laden, Obama or Romney?" I was just flummoxed. I couldn't figure out what it was doing in a poll of standard questions on whom I was going to vote for or if I approved or disapproved of certain politicians. Well, now it's clear what the purpose of the goofy question was - to generate stats so they could make fun of Republicans. Well, I'll confess - I answered "not sure." That wasn't because I wasn't sure but it was my way of registering how ticked off the question made me. As I tell my students, one problem that we can't account for in polling is that people lied. So I lied. Now my answer is part of a Comedy Central gag.

• Washington Post's "Right Turn" blog - Exclusive: PPP caught doing advocacy polling on race:

  Whit Ayres of the conservative Resurgent Republic (who also polls for National Public Radio) was taken aback as well. He said, “Particularly for liberals it is a very leading question.” He said, “Suppose a pollster asked, ‘Do you think the conservative media want voters to think Obama is anti-Semitic?’ People would come out of the frame.” Pollsters acknowledged that putting hot-button questions at the end at least did not taint the favorable/unfavorable result. But none of the pollsters I spoke with would have asked the question or thought it was appropriate. As one said, “It goes pretty far over the edge.”

Public Policy Polling say many of their questions are user-driven: they ask for suggestions on where to poll and what questions to ask - on Twitter as well as on their weblog.

posted by flex (37 comments total)

More pollsters should end their questionnaires with the query "ARE YOU FUCKIN' NUTS?"
posted by oneswellfoop at 3:34 PM on November 14, 2012 [7 favorites]

Loved the analysis re: the Rapture and how it would affect voting patterns
posted by DoctorFedora at 3:39 PM on November 14, 2012 [1 favorite]

Also known as Adam Curtis's "Century of the Self", Chapter 4. Still relevant.
posted by daq at 3:40 PM on November 14, 2012 [1 favorite]

This totally reminds me of the viral marketing command prompt that built up hype for Portal 1 by running an aptitude test on the player.

I can't seem to find the site or a mirror anymore, so I guess I'll have to link to this video.
posted by mccarty.tim at 3:43 PM on November 14, 2012 [1 favorite]

Most people want to think they are rational but react emotionally when the time comes. Forcing an emotional response with some obviously biased questions probably makes it more likely that their responses on the real questions will correlate better with their election day behavior.

Though there is an 80% chance this analysis is full of shit.
posted by chundo at 3:49 PM on November 14, 2012 [1 favorite]

Correction:

Q12 Do you have enough Santorum in your life, or not?
Have enough Santorum 1%
Do not 1%
Not sure 1%
OMG That's disgusting! 97%
posted by brina at 3:52 PM on November 14, 2012 [1 favorite]

Forcing an emotional response with some obviously biased questions probably makes it more likely that their responses on the real questions will correlate better with their election day behavior.

They're just questions, Leon. In answer to your query, they're written down for me. It's a test, designed to provoke an emotional response... Shall we continue?
posted by zsazsa at 3:56 PM on November 14, 2012 [38 favorites]

Which candidate's father would win in a fight with all the other candidates' fathers?
posted by mccarty.tim at 4:03 PM on November 14, 2012

I really like how Tom Jensen explained his "likely voter" screen - "If they hang up, we assume they're not likely voters. It seemed to work." In other words, those who DID participate were considered likely voters, and PPP didn't monkey with "skewing" the polls beyond that. Funny how the simplest approach was the most accurate.
posted by (Arsenio) Hall and (Warren) Oates at 4:17 PM on November 14, 2012 [6 favorites]

Well, now it's clear what the purpose of the goofy question was - to generate stats so they could make fun of Republicans.

If your political group is able to generate significant pro-Romney-killed-bin-Laden stats, it deserves to be mocked.
posted by DU at 4:34 PM on November 14, 2012 [9 favorites]

If they hang up, we assume they're not likely voters. It seemed to work.

lol
posted by DU at 4:35 PM on November 14, 2012

(obligatory)

I have been thinking that I would make a proposition to my Republican friends... that if they will stop telling lies about the Democrats, we will stop telling the truth about them.
- Adlai Stevenson

posted by ethansr at 4:56 PM on November 14, 2012 [18 favorites]

Mike or Joel?
posted by robocop is bleeding at 4:56 PM on November 14, 2012 [6 favorites]

Canadians also ask Americans weird questions.
posted by weapons-grade pandemonium at 4:56 PM on November 14, 2012 [2 favorites]

Although a lot of that was (hilarious) push polling, I wonder if there's at least a nugget of wisdom in their methods.

Does anyone know of any research on the reliability... unconventional polling questions?
posted by graphnerd at 4:58 PM on November 14, 2012

I love their twitter account.
posted by drezdn at 5:02 PM on November 14, 2012 [1 favorite]

I don't think it's push polling if it's only at the end.
posted by (Arsenio) Hall and (Warren) Oates at 5:02 PM on November 14, 2012 [1 favorite]

As has already been stated, if all it takes to generate legitimate cause for mockery is to ask you a simple question, it might just be you that has the bigger issue.

Mike or Joel?

Always Joel.
posted by Dysk at 5:03 PM on November 14, 2012 [3 favorites]

People seem scandalized by the bias and content of the questions, but it isn't a new thing. I worked he phones conducting political polls in the early 90s, and this doesn't seem dissimilar to some of the polls we conducted. "Opinion shaping" polls, they call them. People reacted strongly to the loaded questions, but those reactions and the answers they gave helped candidates decide how to use their resources and tailor their message.


Those polls suuuuuuhhuuucked to conduct, by the way. Everyone knew some of the questions were obnoxious, but we had to ask them, and the people answering got pissed.
posted by louche mustachio at 5:05 PM on November 14, 2012 [4 favorites]

That wasn't because I wasn't sure but it was my way of registering how ticked off the question made me. As I tell my students, one problem that we can't account for in polling is that people lied.

Betsy doesn't know a goddamn thing about surveys or social science.
posted by clockzero at 5:11 PM on November 14, 2012 [3 favorites]

Polling, and reliance on polling is one of the reasons the political landscape is so f$@ked up.

An election is the only poll that matters. I seriously think everyone should tell pollsters to fuck off when they call. Never mind that polls are invasions of privacy, an opportunity to push-poll and to plant disinformation, and are used to identify your household with a party or viewpoint. The information gleaned has even been used to misdirect or discourage voters who aren't likely to support a given candidate.

Without the constant feedback, candidates are less able to manipulate public sentiment, and would have to rely more on -gasp- having a position and presenting a coherent plan and message.

I do believe that Obama's reelection represents the success of substance over mirage, but the fact that Romney, with his undisguised pandering and self-contradictions, and with Ayn Rand's love-child as running-mate, still came so close, shows that with sufficient skill, electing anyone can be as managed as promoting a breakfast cereal. Next time, they may pull it off.

I'm serious - stop responding to polls during elections. It won't make anything better. Or, if you must, lie lie lie.
posted by Artful Codger at 6:02 PM on November 14, 2012 [1 favorite]

Mike or Joel?

Mike for the riffs, Joel for the host segments.

...I still can't believe I had to watch that documentary about horror hosts before I twigged to the fact that MST3K is fundamentally a horror host show.
posted by Pope Guilty at 6:18 PM on November 14, 2012 [1 favorite]

Either PPP didn’t help us here in North Carolina or, if they did, it wasn’t enough. We had the election from hell here, and now the Republican’s have completely taken over state government for the first time in living memory.

Our new governor had a career with Duke Power, since merged with Progress Energy to become one of the largest power companies in the world. He’s declared his intentions to bring off-shore oil drilling and fracking to the state, and more industrial hog farms are a likely prospect.

I came here in 1973 and for a long time the place looked full of promise. Now I can’t decide wether to stick around for the inevitable environmental disasters, to see if people are able to learn from their mistakes, or move back to Maryland which seems headed more in the right direction.
posted by Huplescat at 6:22 PM on November 14, 2012 [1 favorite]

Were these the SOB's that were calling my house every night?
posted by incandissonance at 7:21 PM on November 14, 2012

Mike or Joel?

Who?
posted by This, of course, alludes to you at 7:29 PM on November 14, 2012

This, of course, alludes to you: "

Mike or Joel?

Who"

Bay or Schmaker, obvs.
posted by DoctorFedora at 7:45 PM on November 14, 2012 [2 favorites]

Well I don't like either of those guys! What a terrible question.
posted by This, of course, alludes to you at 7:47 PM on November 14, 2012

Mike or Joel?

JOIKE, of course.
posted by fifteen schnitzengruben is my limit at 8:37 PM on November 14, 2012 [4 favorites]

All polls are push polls.

What makes the whole franchise ludicrous is the market demands for daily data, which flies in the face of any decent attempt at survey design. The market doesn't want good polls. Full stop. Nate Silver realized this early and normalized polls against historical data to contextualize them. But ultimately we have pollsters who are slightly dumber than reality TV actors writing leading questions to respondents who are slightly dumber than reality TV watchers.
posted by allen.spaulding at 8:48 PM on November 14, 2012 [1 favorite]

Or, if you must, lie lie lie.
posted by Artful Codger at 8:02 PM on November 14

A lot of people do. I used to do polling, briefly, for a living. I was assigned to the least biased political poll possible - after the demographic questions, the only questions were, "If a federal election were held tomorrow, for which party would you vote?" and "If a provincial election were held tomorrow, for which party would you vote?" People lied like crazy, and they told you they were lying, but there was nowhere on the computer screen for me to enter anything like, "respondent only said they'd vote communist party to 'put a scare in them'." Presumably the lies sort of cancel out, since the political parties commissioning the polls weren't interested in absolute numbers, but in how they changed over time.
posted by joannemerriam at 8:58 PM on November 14, 2012

I think the lies do cancel out.

My experience was that, really, very few people lied. Well within our acceptable margin of error. People like that stick out in your mind, true, but that doesn't mean there is enough of them to make much of a difference statistically.
posted by louche mustachio at 9:05 PM on November 14, 2012

Also, Joel is cuter.
posted by louche mustachio at 9:05 PM on November 14, 2012 [3 favorites]

I'm sure I'll regret this, because I was a much better journalist back in the day than I am a shitty cable access TV host, but here's a 26-minute interview I did in July with Tom Jensen, PPP's young and brilliant director who deserves at least three quarters of the fame Nate Silver has:

Jensen on what PPP does differently and why that makes them more accurate *and* much more affordable

Jensen on what journalists do that "drives us crazy as a polling company"

Landline decline and the future of internet polling, including news that polling companies are finding people who don't use the internet and giving them free internet
posted by mediareport at 9:11 PM on November 14, 2012 [6 favorites]

every state has its not-so-admirable biases.

What biases, exactly, does each state have? Inquiring minds want to know.
posted by dunkadunc at 10:20 PM on November 14, 2012

Interesting that more Democrats are willing to spend a night in a haunted house than Republicans; I wonder whether that measures neophilia and openness to the unknown (inversely correlated with conservatism) or scientific/materialist vs. religious/supernatural worldviews.
posted by acb at 4:58 AM on November 15, 2012

clockzero: That wasn't because I wasn't sure but it was my way of registering how ticked off the question made me. As I tell my students, one problem that we can't account for in polling is that people lied.

Betsy doesn't know a goddamn thing about surveys or social science.

Surveys of self-reported homosexuality and penis size, both seriously affected by social factors which can encourage lying, are notoriously inconsistent. The incidence of those who admit to homosexuality in particular has increased over the decades.

Either we're getting a lot gayer year by year (and male peak erections grow and shrink by about an inch), or lying is difficult to account for in polling. If you have evidence to the contrary, I'd like to see it.
posted by IAmBroom at 10:38 AM on November 15, 2012

As far as it applies to state-level polls of Presidential elections, lying is not much of a factor. I believe the averages have correctly predicted the state outcomes in 48 or more of the 51 contests since at least 1996. (IIRC, I'm not looking at the data right at this very second)
posted by wierdo at 6:41 PM on November 15, 2012 [1 favorite]