Another blow for the TED talk industrial complex

July 24, 2022 6:01 PM

On Nudge Theory: No evidence for nudging after adjusting for publication bias.
posted by latkes (31 comments total)

Which category does the idea that if you make something the default (e.g. being an organ donor, choosing renewable energy, etc.) more people will choose it? That was a nudge thing wasn't it? Because while it was an intuitive finding it always made me feel a little depressed that people were so easy to manipulate, so if it turns out to be not true that's a silver lining for me.
posted by Wretch729 at 6:31 PM on July 24, 2022

I feel like it's moving in a direction where the proper heuristic is that if something seems like an idea your boss would like than it's probably empirically dubious.
posted by Reasonably Everything Happens at 6:32 PM on July 24, 2022 [36 favorites]

Oh boy, can't wait to dig into the paper. In the meantime, via the UK govt (that nudge dept Cameron set up), mine can't be the only country whose economic planning bodies got super committed to setting up their own Nudge units, thanks to their (the British) tireless evangelism. I mean, advocacy.
posted by cendawanita at 6:34 PM on July 24, 2022 [1 favorite]

in 2022 we should know enough by now that any kind of brand new behavioral psych breakthrough (esp of the kind that seems to reify the existing capitalist structure of society) is just total bullshit

in 2022 the people who trust this kind of shit the most are CEOs and their ilk, heavy consumers of total self-help bullshit that pulls singular studies like these and spins grand conclusions about it

in 2022 I'd hope we'd have realized that the people making top level decisions in society are so actively clueless that the Duning-Krueger effect, itself a phenomena lacking replication/any kind of real evidence (RCTs), at least is able to describe an all-too-common trait of those in leadership positions (rather than describing some kind of catch-all effect governing all human behavior) and we'd have come around to maybe not having strict hierarchies
posted by paimapi at 6:46 PM on July 24, 2022 [22 favorites]

My experience over many years in the tech corporate world, is that CEOs seem to respond to and promote whatever the latest how to run your business bestseller is. In Search of Excellence led the charge. I had to go through frequent shifts of how to do things whenever a “new approach” came out. Given the shifts, made it obvious to me that this was all bullshit. How to run a business, I would assume, would not be so malleable that any new idea would so quickly displace it. Conclusion? They don’t know what the fuck they are doing. Lower level people just kept everything churning along by nodding their heads at company wide meetings on the new paradigm and then going back to their office to continue keeping the machine going. Academic researchers getting into this racket makes perfect sense, as long as they can turn their “peer reviewed” research papers into NYT bestsellers.
posted by njohnson23 at 7:07 PM on July 24, 2022 [11 favorites]

Never were maciej's words better applied to credulous promulgators of pop-sci psychology

'The whole genre reminds me of the the wooly business books one comes across at airports ("Management secrets of Gengis Khan", the "Lexus and the Olive Tree") that milk a bad analogy for two hundred pages to arrive at the conclusion that people just like the author are pretty great.'
posted by lalochezia at 7:11 PM on July 24, 2022 [15 favorites]

Yet another failed Big Idea in psychology/psychiatry.

At what point does this field start accepting that they have a persistent systemic failure of quality control?*

They seem completely incapable of understanding basic methodology, and why it is not a luxury optional extra.

(*There are, of course, some within the profession trying hard to get this problem acknowledged and corrected, and who deserve our gratitude and support.)
posted by Pouteria at 7:27 PM on July 24, 2022

I had to go look up publication bias so here it is for anyone who needs:
In published academic research, publication bias occurs when the outcome of an experiment or research study biases the decision to publish or otherwise distribute it. Publishing only results that show a significant finding disturbs the balance of findings in favor of positive results.[1] The study of publication bias is an important topic in metascience.

If I understood the paper correctly (and I admit I didn't understand it at all) what they're saying is that when they looked at a lot of papers trying to measure if there's such thing as "a nudge effect" they did some math on the numbers and found that it seems likely that some papers were only published because they had positive results, which implies an existence of unknown negative results. Therefore there's no statistical evidence of a measurable "effect" as such. Is that right?

Because I find it hard to believe that encouraging people to do the thing you want them to do.... doesn't work. I mean. Someone tell me what they're trying to say.
posted by bleep at 7:29 PM on July 24, 2022 [10 favorites]

My experience over many years in the tech corporate world, is that CEOs seem to respond to and promote whatever the latest how to run your business bestseller is.

OK. So we need someone to write "Make your business succesful by paying your employees well and leaving them alone to do their jobs." The rest of us have to by eight copies each.
posted by stevis23 at 7:52 PM on July 24, 2022 [24 favorites]

The other reason I find this confusing is because in my line of work I have to find ways of getting people to do one thing instead of another thing by finding subtle visual treatments that speak the universal language that I use to suggest the optimal course of action. So given that I'm pretty familiar with the language of doing this I'd be interested in learning what it could mean for this to... not exist?
posted by bleep at 8:00 PM on July 24, 2022 [2 favorites]

Why did we even think we COULD measure something like this at all? Just like, one way out of thousands that people have been manipulated? If we can't make it into a good pattern of numbers then what good is it eh?
posted by bleep at 8:12 PM on July 24, 2022

Which category does the idea that if you make something the default (e.g. being an organ donor, choosing renewable energy, etc.) more people will choose it?

according to the paper the study in the OP is responding to, a 'default' choice is a decision structure manipulation. In both the linked paper and the previous paper, structural 'nudges' are the most effective/least bullshit. When the linked paper says that there is 'no evidence' for nudge effectiveness, it is considering all types of manipulations together, not just structural manipulations. When considering structure alone, the paper says the evidence is 'undecided.'

Yet another failed Big Idea in psychology/psychiatry.

not sure you can lay this on psychology. Thaler is an economist, and poking through the list of studies (pdf file) that went into the meta-analysis, a lot of those look to be published in economics journals. I'm sure there are some very fine empiricists in economics who do quality behavioral work, but, well, i haven't met many of them.

Because I find it hard to believe that encouraging people to do the thing you want them to do.... doesn't work. I mean. Someone tell me what they're trying to say.

it seems like the authors of the paper are not providing a lot of nuance in their big takeaway. They basically took the meta-analysis someone else did and gave a hot Bayesian take on it. Wagenmakers, the last author on the linked paper and therefore probably the professor overseeing all the work, is a big methods wonk who basically goes around looking for existing questions to apply methods to... like his research topic is 'bayes theorem and the various things you can do with it.' So the linked paper is not really geared to say that some kinds of 'nudges' (god i hate that term... it basically reeks of an economist with ambition) work and some others don't - it more or less accepts the data that someone else curated as being 'nudge-related,' applies some bayesian-flavored analysis, and comes back with a yes/no answer. For them to get into the details about why some means of persuasion work and others don't kind of defeats their entire purpose in life.

In this sense, this paper probably fits in with a lot of other overblown methodological criticisms of psychology/neuroscience (voodoo correlations, brain activity in a dead salmon, and so forth): technically correct, but not practically useful. Probably the more sober and useful takeaway here is that some forms of persuasion are useful, but there is a lot of trash out there that makes it difficult to say what is good and what is bad.
posted by logicpunk at 8:27 PM on July 24, 2022 [22 favorites]

Ok, so more bullshit from the bullshit factory that is economics. I would not have read it if I realized that. My mistake.
posted by bleep at 8:32 PM on July 24, 2022 [3 favorites]

Yes but did they try favorites and a highly moderated flagging system ?
posted by NoThisIsPatrick at 8:53 PM on July 24, 2022 [10 favorites]

Governments like “nudge” because it seems like a cheap way of doing something. And the whole “personal responsibility” bit fits in with the prejudices of right wing administrations.

The problem is that there is no such thing as “nudge theory”. It’s a mishmash of traditional advertising/con artist/magician tools like priming mixed up with woo.

We added a “Become an organ donor” option at the end of of the online process for applying for/renewing your driving licence. It has driven up the numbers but that’s as likely to be down to people being conditioned to keep filling in forms until the questions stop!
posted by fallingbadgers at 12:55 AM on July 25, 2022 [6 favorites]

I taught little ms flabdablet a bit of nudge theory out on the road this evening.

We were on the highway, she driving, L plates clearly displayed, about 2km from the spot where we were eventually going to need to turn across the centre. The highway has a 100km/h speed limit at that point and without instruction from me she was maintaining about 90km/h because it was dusk and there are wallabies and wombats.

Asshole zoomed up behind and sat right on our tail with his brights beaming in through the back window. Little ms flabdablet did exactly the right thing by maintaining course and speed and swearing at him in fluent Motorist, but it had no effect and instead of either falling back to two seconds behind us or overtaking, he just kept sitting back there blinding her.

So I pointed out that in another 30 seconds or so we'd be on a 1km section with double white lines down the middle, and predicted that if she were to drop her speed by a mere 5km/h when we got there then (a) it would make almost no difference to the amount of time it took to transit that section but (b) by the time we'd got to the end of it the guy would be fuming and would instantly pull out and overtake and no longer be an issue by the time we needed to make our turn.

Worked perfectly. He gave us the finger on the way past and everything.

Passive aggression is skilled aggression.
posted by flabdablet at 4:12 AM on July 25, 2022 [16 favorites]

A tap on the break with exactly three emergency blinks does wonders for increasing the tailgaters blood pressure. Oddly satisfying.
posted by sammyo at 6:15 AM on July 25, 2022 [1 favorite]

njohnson23: "My experience over many years in the tech corporate world, is that CEOs seem to respond to and promote whatever the latest how to run your business bestseller is."

When the Steve Jobs' biography came out, I had to put up with a few years of MBAs explaining to me why every corner in the websites I made for them had to be round…
posted by signal at 6:48 AM on July 25, 2022 [4 favorites]

From what I've been able to see, over the past decade or so, is economics seems to be a field of study where the floodgates have opened for researchers attempting to disprove prior work. Normally I would say, that's great, lets get some concrete results. But of course, when combining humans who tend to lie to themselves as much or more as to anyone else, and really complex questions, getting any results, let alone something definitive feels like an impossible task.

And secondly, it seems so...personal? Like, I have some faith in the belief that the crime waves in the US died down in the 80's-90's because, in part, of a combination of the phase out of leaded gasoline and paints, and the legalization of abortion. I've heard of researchers going HARD against these theories to disprove them. As if it personally offends them. I'm baffled. Honestly, a combination of those two things might be a contributor to the era of lower crime rates, but that cant' be the whole story? Why does every theory have to be either a silver bullet or totally wrong?

Perhaps nudging doesn't work in X out of 100 types of interactions, but obviously it does work in a small sub-sample. Shouldn't we be drilling down to figure out the why of that instead of just tossing our hands up and saying the whole thing is wrong, wrong, wrong?
posted by sharp pointy objects at 7:53 AM on July 25, 2022 [2 favorites]

So I'm not surprised that a meta analysis (especially in the social sciences) found no effect. That being said, the meta analysis format (even the Bayesian subvariety) is not without criticism. I do wish Maier et al went into more more depth about what assumptions they made in the RoBMA technique since a) the Bayesian approach is really cool b) their assumptions impact the study's outcome. All in all though, this study's results are only one (or several) data point(s) and I wouldn't call it the "final nail in the coffin" of the "choice architecture" field. No one approach is going to be suitable for all situations. Contrary to "popular belief" there's no "Big Hammer" conspiracy, but sometimes hammers can be remarkably useful in specific contexts, like driving in nails.
posted by oceano at 7:54 AM on July 25, 2022

Can we dial back the dancing on the graves of psychology and behavioral economics? This is a 1-page letter that responds to a meta-analysis published in PNAS. It is unclear whether it is peer reviewed or not, but letters often are not. As logicpunk notes, this is a drive-by reanalysis of a piece of original research with a catchy title and a big takeaway that lacks any kind of nuance.

If you want a little bit more nuance, there is another critical letter in the same issue: 'No reason to expect large and consistent effects of nudge interventions.' They conclude: 'We argue that, as a scientific field, instead of focusing on average effects, we need to understand when and where some nudges have huge positive effects and why others are not able to repeat those successes (2, 4, 5). Until then, with a few exceptions [e.g., defaults (6)], we see no reason to expect large and consistent effects when designing nudge experiments or running interventions.' In other words, even if on the whole nudges aren't that effective, some 'nudges' might indeed work.

Which, BTW, is exactly what the authors of the linked letter also say: 'However, all intervention categories and domains apart from “finance” show evidence for heterogeneity, which implies that some nudges might be effective, even when there is evidence against the mean effect.'

The original meta-analysis is interesting, but asking 'do nudges work?' is a bit like asking 'do pharmaceuticals work?' Nudges, like pharmaceuticals, are tools: some kinds of nudges (like some kinds of drugs) that may be very effective for some conditions and totally ineffective for others.

There is certainly a lot of hype and over-inflated claims in the nudge literature, and there is publication bias across many fields. But that letter mounts over-inflated claims too.
posted by googly at 7:57 AM on July 25, 2022 [12 favorites]

brain activity in a dead salmon [...] technically correct, but not practically useful

Oh, I don't know--that specific critique came neatly packaged with a suggested methodological correction: incorporating multiple comparison corrections. The problem the salmon critique points to is that measuring blood flow changes in thousands of voxels across time is vulnerable to false positives generated by random noise, and it suggests using more stringent statistical tests in response. And I mean, the field response is as far as I understand it to... use more stringent statistical tests now. As far as methodological changes go, adding more post-hoc statistical tests to the analysis pipeline is honestly pretty cheap and easy.

It's maybe worth considering that that particular critique was leveled at the field by fMRI researchers who have continued to work in fMRI work, though, and who provided an obvious solution to the identified methodological problem.
posted by sciatrix at 8:11 AM on July 25, 2022 [3 favorites]

A tap on the break with exactly three emergency blinks

is also illegal where we live: the emergency blinkers are only to be used when the vehicle is stationary and positioned so as to present a potential hazard to other road users. Tapping the brakes, especially when done by a less experienced driver, risks causing sudden changes to the vehicle's state of motion and its relationship with a closely following vehicle that would reduce safety for all.

Driving to the prevailing conditions, though, is unarguably appropriate. And if the prevailing conditions include reduced visibility caused by blinding lights coming from behind, a small and non-sudden speed reduction seems completely warranted. I doubt you'd find a magistrate who'd disagree.

The reason I think the obviously correct and obviously infuriating-only-to-assholes behaviour that I recommended to little ms flabdablet counts as nudge theory is that what we wanted the asshole to do was make a fucking choice instead of carrying on driving like a dipshit. The action we took was a successful attempt to assert such control as we had over the options available to him such that he'd make the choice we preferred he made: take his asshole driving style far away from us as soon as possible.

Near as I can tell, you really wouldn't get a ciggie paper between most of the practices I've seen described as nudging and passive aggression. Both proceed from a sense of frustration at seeing people repeatedly doing the wrong thing, and both rely heavily on a solid understanding of how the prevailing power dynamic actually operates and a preference for working within it rather than seeking to change it.
posted by flabdablet at 8:35 AM on July 25, 2022 [4 favorites]

asking 'do nudges work?' is a bit like asking 'do pharmaceuticals work?' Nudges, like pharmaceuticals, are tools: some kinds of nudges (like some kinds of drugs) that may be very effective for some conditions and totally ineffective for others.

QFT. (Also, everything that logicpunk says).

If one is going to posit that nudges don't work, then one likely also has to accept the corollary that dark patterns don't work. It might require a few more assumptions to show that if "nudges don't work" is true, then then there are no benefits of applying the outcomes of A/B testing, but I think we get there with a little work, despite literal piles of political and commercial money suggesting otherwise.

Corrections on overuse of a theory are certainly important! But - as fellow commenters point out - the story here is mainly that the dose is the poison.

And honestly, the absolutist rhetoric of those touting a correction is just as irritatingly unscientific as the rhetoric of the nudge evangelists. Expressing the value of behavioral interventions via the logical "for all" seems extremely dangerous.
posted by BlueBlueElectricBlue at 9:26 AM on July 25, 2022 [3 favorites]

Several years ago I went to a “mentalist” event staged as a dinner party. The performance was very convincing and impressive, but I earned later it was a bog-standard mentalist routine after all. Anyway, most of what mentalists do, and do reliably, could be characterized as nudges.
posted by sjswitzer at 9:34 AM on July 25, 2022

re: dead salmon It's maybe worth considering that that particular critique was leveled at the field by fMRI researchers who have continued to work in fMRI work, though, and who provided an obvious solution to the identified methodological problem.

at the time the salmon poster came out, correcting for multiple comparisons was already a thing in fMRI, and had been for 15 years. Perhaps the framing ('lol neuroscientists') made the field more rigorous in taking out the trash, but the author acknowledged that 65-75% of fMRI studies they reviewed already applied corrections. All the nuance (a minority of fmri studies may be reporting inflated stats) got thrown out for lulz.
posted by logicpunk at 9:41 AM on July 25, 2022 [2 favorites]

It's the publication-bias bit that's most interesting to me personally. I'm working on a project in a completely different field that I believe is full of publication bias; I've heard secondhand stories of research projects that didn't achieve positive results not even being sent out to journals.

Appreciate the OP for acquainting me with a new testing method I hadn't known about.
posted by humbug at 10:26 AM on July 25, 2022

failed Big Idea in psychology/psychiatry.

psychology and psychiatry are two extremely different fields where evidence for psychiatric interventions (like CBT/SSRIs) are consistently tested with and pass dozens of double-blind RCTs with hundreds of thousands of samples with systemic reviews and meta-analyses concluding the same, whereas psychology faces basic replication issues for a single, non-RCT study with like 20 participants

which is also to say that economics is even worse than behavioral psych (though not much, and with so much collusion between the two), and is 'scientific' in a reductive philosophical sense and less in a practical, evidence-based sense
posted by paimapi at 9:35 AM on July 26, 2022

Psychiatry also has some problems.
posted by latkes at 1:22 PM on July 26, 2022 [1 favorite]

asking 'do nudges work?' is a bit like asking 'do pharmaceuticals work?' Nudges, like pharmaceuticals, are tools: some kinds of nudges (like some kinds of drugs) that may be very effective for some conditions and totally ineffective for others.

I little biased, but no. The point of the "nudge" branding was that they generally worked and had surprisingly large effect sizes. Restated as sometimes being able to persuade people of stuff, but also the effects are so variable you can't really tell the difference between doing it and not, is just something else entirely.

Or maybe I'm overstating things. Is there a specific small intervention, huge effect size situation that has been reproduced reliably (large samples, preferably RCT, preferably pre-registered, multiple independent studies) that you are thinking of when you assert nudge is "very effective?"
posted by mark k at 12:19 AM on July 27, 2022

Is there a specific small intervention, huge effect size situation that has been reproduced reliably (large samples, preferably RCT, preferably pre-registered, multiple independent studies) that you are thinking of when you assert nudge is "very effective?"

I didn't assert that nudges are effective, I said that some may be effective in some situations but not in others. The overall spirit of my comment was that the cited letter - as the letter's authors themselves acknowledge - should not be read as an argument that nudges don't work, full stop.

But since you asked: one of the most well-known, successful examples of a 'nudge' is changing the default option for organ donation from opt-in to opt-out. Here's an experimental study (n=270) in the US showing that it doubles donation rates. Here's a survey-based experimental study (n=2069) from the Netherlands showing an increase in donation rates from 50% to 62% with a switch from opt-in to opt-out. Here's a review article that surveys a ton of empirical studies; it finds 21 studies that show positive impacts of changing defaults, 4 inconclusive, and 1 negative.

Do all nudges work? No. Are they a perfect policy panacea? Of course not. Are they a form of intervention that might sometimes work to achieve particular goals? Possibly - which makes them very similar to every most other policy tools (taxes, prohibitions, incentives) at our disposal.
posted by googly at 11:04 AM on July 27, 2022 [3 favorites]