Following polls has become our vision of what political participation is
September 30, 2024 12:11 PM   Subscribe

That error opens up onto the myriad conceptual fallacies built into the entire enterprise, if something so unavoidable can be called an “error.” Past performance is no guarantee of future results; but past performance is all a pollster has to go on. That’s why much of the process of choosing and weighting samples is … well, you can call it “more art than science.” Or you can call it “intuitive.” Or you can call it “trial and error.” But you can also call it “made up.” from The Polling Imperilment, an essay by Rick Perlstein built around Joseph Campbell's 2020 book Lost in a Gallup [The American Prospect; ungated]
posted by chavenet (29 comments total) 5 users marked this as a favorite
 
I'll add this brief piece by Jay Kaspian Kang into the mix.

I have long been quite critical about the amount of space the polls take up in our political discourse. The reasons are pretty simple: although polls certainly have a place in assessing the state of the election, they’ve inspired a type of sophistry in which the pundit or the politician flashes the results of some fallible poll and treats it as irrefutable proof of the will of the electorate. The result is a tower of bad takes, built upon a foundation of solid polls and good pollsters. The question is not whether we should “trust the polls.” It’s whether the onslaught of analysis and extrapolation that invariably follows them actually holds any predictive or explanatory power.
posted by kensington314 at 12:34 PM on September 30 [11 favorites]


I get so many terrible polls where I'm presented with a list of twenty issues and asked to pick five and rank them in the order of importance.

Maybe there's some ten-thousand foot view that makes the exercise worthwhile, but I honestly can't see that there's much meaning in determining whether "Combating Systemic Racism" is slightly more or less important than "Protecting Democracy" while measuring how either of those compare with "Ending Childhood Hunger" as a priority.

Fuck pollsters.
posted by RonButNotStupid at 12:40 PM on September 30 [6 favorites]


Atwood said. “Not every best candidate wins. That’s just how it is.”

Or maybe God answered your prayers, you ignorant *!($*#$&)%(%

I love that a literal fucking plague while their guy is an office couldn't possibly be a sign to these folks that maybe they're on the wrong path.
posted by Abehammerb Lincoln at 12:46 PM on September 30 [4 favorites]


There's the problem with polls and how politics/elections are covered in the media.

And then there's the difficulty of designing surveys that are clear, not unintentionally leading, and actually interesting. Regardless of topic! A lot of my students want to conduct surveys (Pew Research link) as part of their research and I always groan a little bit because survey design is SO HARD.
posted by spamandkimchi at 12:57 PM on September 30 [5 favorites]


Every poll that has my preferred candidate ahead is an infallible predictor of the election. Every poll that has them behind is partisan propaganda from some hack that's using bad data.
posted by fortitude25 at 12:58 PM on September 30 [1 favorite]


So what's the alternative? "Vibes?" Whatever the NYT decides it is?

Polls are deeply imperfect. But they beat the pants off of everything else.
posted by Frayed Knot at 1:01 PM on September 30 [4 favorites]


So what's the alternative? "Vibes?" Whatever the NYT decides it is?

Asking this with no snark, but alternative or "vibes" as regards what? What's the "it" in, "Whatever the NYT decides it is?" The election is the ultimate "it," but what are polls thought to be addressing, in this question? Just an overall need for political punditry? Is your question, "What is the alternative to polls for people who want political punditry and horse race coverage?" Or is it something else?
posted by kensington314 at 1:05 PM on September 30 [5 favorites]


You can’t ungate something that doesn’t have a paywall!
posted by Going To Maine at 1:06 PM on September 30 [1 favorite]


I actually bought a domain to post info about how off the polls have been in the past few years but I haven't had time to do anything with it. Argh.

I linked to some of the data I'd seen about polls being off in a comment back in June (I should just put THAT up at my incomplete website, it'd be better than nothing). There's also the way the polls were way off for the latest elections in France and the UK.

There are so many other indicators we should be folding in to the conversation:

* volunteers
* campaign contributions
* number of donors
* % and $ totals of donations from small donors

I wish we got more and better reporting on those indicators.

Thanks for this, chavenet. I appreciate the opportunity for us to talk about the many problems with polls these days.
posted by kristi at 1:09 PM on September 30 [3 favorites]


The article posted goes on at some length about how the polls are as inexact as they were 100 years ago, and should not be trusted to tell us how people will vote come election day.

So what will? Campaign's need to use something to decide where to put limited resources. I need to use something to decide where I ought to donate or volunteer.

And news outlets will inevitably use something to frame stories and develop narratives.

I asked it in a snarky way, because I'm frustrated by the article (I think it's basic assertion is dead wrong), but it's a serious question. These needs must be met. If not via polls, then what?
posted by Frayed Knot at 1:12 PM on September 30 [2 favorites]


I'd appreciate polls and pollsters much more if they stopped obfuscating voter sentiment behind vague concepts like "enthusiasm" and "double-haters" and just honestly tried to represent why voters support a given candidate. It frustrates me that I have to withhold honest criticism and always give five stars lest my preferred candidates chances be jeopardized because the polls show them to be "weak".

And news outlets will inevitably use something to frame stories and develop narratives.

Maybe they should try using candidates' policies to frame news stories? Like how one candidate is literally spreading malicious lies about immigrants?
posted by RonButNotStupid at 1:22 PM on September 30 [6 favorites]


Instead of polls (and especially candidate approval polls) being the media bread-and-butter for election season, I think getting a snapshot of what people want to know / worried about and then doing a series of actually reported articles on those topics would be amazing.

For example, taking the results from this Pew Research survey on voters' top issues where the options are already preset, and then actually digging into what aspects of the economy are stressful, and how that might differ for different age groups or economic classes etc, and if the candidates have addressed those in any meaningful way.

Issues will land differently because of ideology and (ahem) media coverage -- see crime/immigration/climate change. But also will land differently because how an individual is situated. Stock markets are up! I am, thanks to being in a public pension, one of those 62% of Americans who is categorized as an investor in the stock market. But on a gut level, record high stock markets give me no joy, only frustration in the hyperfinancialization of the economy and the dominance of the FIRE industries (finance, insurance, real estate) in economic coverage. Health care costs continue to rise. A concept of a plan isn't going to cut it.
posted by spamandkimchi at 1:23 PM on September 30 [7 favorites]


Maybe we should just go with hat sales (NYT).

“We saw virtually no Biden sales all year,” he said.
But when Ms. Harris became the nominee and Mr. Walz joined the ticket, enthusiasm for their campaign — and for the style of hat that came to be associated with it — led to a huge increase in Unionwear’s production.
The factory usually produces 2,500 hats a day, maxing out at around 4,000. Lately, Mr. Cahn said, it’s churning out about 5,000 Harris-Walz hats a day.Since 2008, he said, the number of hats ordered by a presidential campaign has foretold the winner of that year’s election.
In 2016, Mr. Cahn said, the Hillary Clinton campaign hadn’t moved many baseball hats.
“That was fascinating to us because she did not sell any hats,” Mr. Cahn said. “We were just trying to reconcile that with the presumption that she was going to win.”

posted by jenfullmoon at 1:25 PM on September 30 [12 favorites]


Surely the needs of campaigns to target resources must be met, and they rely heavily on internal polling that never gets into the punditry blender. Granted.

When 538 was a lot more popular, I always really resented the site because I felt it treated political coverage like sports betting. (Still, did I obsessively click through the page everyday? At times, I certainly did.) They modeled elections based on polling instead of, say, hiring a journalist to go cover what was going on with Michigan voters. (My ambivalence about this approach to political journalism showed up here in an AskMe, though my objection wasn't about polls, per se, and was more about election modeling a la 538.)

I use the Michigan example because in 2016 I remember going to the Clinton campaign office in East LA to call Michigan voters and having the creeping realization that the reason all the phone numbers were bad is because I was tapped into some list of people that was so economically vulnerable, they couldn't keep a steady phone number. That moment, days before the election, felt more telling than months of reading poll aggregations. I learned something, aside from the fact that polls with a 4% margin of error showed a tight race.

I think the reason this article resonated with me is that for the general public (as opposed to the campaign), I don't really understand the value of polling coverage. It fulfills a certain compulsive urge, and it allows many of us to feel informed. But what have we learned?

Anyway, I don't say this with much stridency. I look at polls. I read some polling punditry. I bank my hopes on what I take to be a polling trend toward Kamala Harris. I just don't think polling coverage is journalism and I always find myself wishing I could click through to rich information about the electorate in a given state. But I understand that as expensive as polling is, doing journalism well on a national scale is even more so.
posted by kensington314 at 1:27 PM on September 30 [4 favorites]


There's also this, from my current daily read, Electoral-vote.com:

"Who do you think will win?" may be the better question:
Going back 60 years and using state polling data, the leaders in the polls won the state 69% of the time. In contrast, the candidate the voters predicted would win actually won 81% of the time. In other words, asking people "Who do you think will win your state?" is a better predictor than asking people how they will vote and seeing who got the most votes.
posted by kristi at 1:36 PM on September 30 [8 favorites]


From the kristi link, I thought this was interesting:

The reason is not clear, but it could be that although a voter wants candidate X to win, he or she hears all the neighbors raving about candidate Y. The question about what the voter wants is just one data point, but the question about who will win the state collects information from many voters. That might actually be a better way to do forecasting.

Maybe tough to give this a very science-y valence, but it has a certain logic working in its favor!
posted by kensington314 at 1:39 PM on September 30 [3 favorites]


If you look back at the relatively modern era of U.S. presidential polling, starting in 1952 (after the famous "Dewey Defeats Truman" kerfluffle of 48), you see that national polling averages picked the popular vote winner every time, and in most cases to within a couple of percentage points.

Moderate outliers would be 1980, 1992 (shakes fist at Perot), and really 2020, when Biden underperformed the polling averages. 2016 was weird, but the polling predicted Clinton would win the popular vote--which she did.

Polling has had a couple of rough cycles, with polling orgs not adjusting for the Trump effect...and then overadjusting for it in 2022 in turn.

There's less polling being released to the public than ever, it seems like. Only about one in six U.S. House races have had any sort of polling released at all. Senate polling is thin outside of a few marquee races in swing states; it's been months since I've seen anything on the New Jersey race for Senate, as one example.

In 2022 there was a set of right-wing "polls" released that I'm pretty sure were flat-out fraudulent, made up out of thin air. Those got averaged into the polling trends with everything else, which polluted a lot of the state-by-state polling and made Republicans in those races look stronger than they really were.

Oh, and the NYT doesn't even really care about its own polls anyway. I remember at least one 2022 article where they released some Siena College polls that they had commissioned for Congressional races, the polls made Democrats look fairly good, the NYT was basically all "here's why you shouldn't believe these polls we just paid for".
posted by gimonca at 1:48 PM on September 30 [6 favorites]


The pollsters can likewise be arbitrary once the numbers come in. Lost in a Gallup notes a fascinating experiment carried out by Nate Cohn for The New York Times. He had four pollsters interpret the same raw data from a 2016 poll of Florida. Their choices in how to weight ranged from Clinton winning by four percentage points to Trump winning by one.

Is this supposed to be an argument against polling? It sounds like all the methods agree that the race is close and he variation induced by methodological choices is in a pretty narrow band.

They’re cocky about it; that’s a pattern, too.

I don't know what's meant by this. If you want to hear that poll numbers are probabilistic, fallible, and depend on a lot of unverifiable internal assumptions.... ask a pollster. They will also probably tell you that, for all this, polls are still a better indicator of where the race stands than the vibes Bret Stephens perceives in his neighborhood or what Thomas Friedman's cab driver told him. And they're right.
posted by escabeche at 1:56 PM on September 30 [6 favorites]


The article starts with "In 2016, I experienced the desolation of my candidate for president losing after the most respected polling experts told me she had a 71.4 percent, 85 percent, 98.2 percent, and even 99 percent chance of winning." That's not polling. That's overly clever people taking the second or third derivative of a tangent of a trend and presenting it as hard data. I remember those "99 percent" style predictions, they really didn't have any rationale for existing. And I don't see people are doing much of those since.
posted by gimonca at 2:16 PM on September 30 [5 favorites]


Two problems I see with polls are Nate Silver and Rasmussen.
posted by nofundy at 2:18 PM on September 30 [2 favorites]


I don't do polling but my work sometimes involves survey research. I've been in the field 25 years now and sampling design has gone from an interesting problem we must consider to a functional impossibility in a lot of our research. Phone numbers could be anywhere and people don't answer them anyway, panelists are fake, if you pay respondents you're swamped by bots which are now nearly undetectable, response rates are ridiculously low, etc. A lot of clients are moving to focus groups, which can't provide the same information but at least can tell you SOMETHING.
posted by metasarah at 3:26 PM on September 30 [5 favorites]


The media also capitalizes on the general public's ignorance of statistics. Tell someone that the polls show that Trump only has a 17% chance of winning and people figure he's as good as lost already. Tell them they have the same odds of getting a bullet playing Russian Roulette and all of a sudden no one wants to play (unless they've got a death wish).
posted by VTX at 3:29 PM on September 30 [1 favorite]


Now now now, that's not correct. Nate Cohn (NYT's lead polling entrails predictor) is a total POS tool as well!

Today he led with 'its been a quiet week on the campaign' story, you know, the week Trump went all in on immigrants as animals, the VP as having mental health problems, the magas vile Haitian immigrant hate-fest - now with bomb and death threats for anyone who also doesnt hate Haitians, and to top it off, the big orange rapist calling for a day of cop violence - hey remember with the NYT went bananas on the VP for saying Trump predicted a bloodbath is he didnt win?

In sum, the NYT politicos like Cohn are awful, and want Trump to 'win'.
posted by WatTylerJr at 3:32 PM on September 30 [1 favorite]


Rick Perlstein is a terrific historian and also a sweet person to boot.
posted by doctornemo at 3:53 PM on September 30 [1 favorite]


I'm a huge fan of Rick Perlstein's books, so I'm probably bringing a pretty generous reading ito this essay, but, for me, he doesn't quite make his case.

Like, he's criticizing polling, but he really ought to be criticizing journalism, or maybe he ought to be criticizing capitalism.

Past performance is no guarantee of future results; but past performance is all a pollster has to go on. That’s why much of the process of choosing and weighting samples is … well, you can call it “more art than science.” Or you can call it “intuitive.” Or you can call it “trial and error.” But you can also call it “made up.”

This sounds like someone complaining about meteorology.
posted by box at 3:55 PM on September 30 [2 favorites]


The preoccupation with polls seems like one facet of a larger tendency of our time, which is that the most politically sophisticated and engaged people I know spend a lot more time engaged in meta-political thinking about what the “average voter” wants than thinking about what they want. I worry that this constrains the political imagination severely.

On the other hand when I see the Nate Silver “sports betting” approach to politics contrasted with narrative exploration of issues and voter sentiment, there’s a part of me that wants to say - at least what Nate made his name doing is (sort of, if at unfortunately infrequent intervals) testable and accountable for its misses. The worst version of Nate Silver is clearly the version that strays from polls and odds into narrative punditry!
posted by atoxyl at 4:53 PM on September 30 [2 favorites]


I wouldn't have needed to read all the issues with polling he enumerated to get to where he ended up if he'd had atoxyl as his editor — "the most politically sophisticated and engaged people I know spend a lot more time engaged in meta-political thinking about what the 'average voter' wants than thinking about what they want" — which pulls forward what I thought the piece was promising going in.

The Clinton team coined "triangulation" in the '90s. I don't think we were meant to triangulate for them.
posted by mph at 6:03 PM on September 30


“Polls aren’t politics” - a very good point

“Polls aren’t accurate” - I dunno man, they are and they aren’t, that’s statistics
posted by atoxyl at 6:12 PM on September 30


I'm shocked people here have been polled! I don't know a single person who has been. Like...it really happens, I guess! Huh!
posted by tiny frying pan at 6:29 PM on September 30


« Older "A mic on the street that Shazams 24/7"   |   Playing Chicken Newer »


You are not currently logged in. Log in or create a new account to post comments.