less effective on superficial misinformed beliefs

September 12, 2024 1:50 PM

Meet DebunkBot: an AI chat bot that provides factual explanations and counter-evidence for these conspiratorial events. It's strength appears to be that the LLM is inexhaustible and will argue indefinitely. They found that the targeted dialogues resulted in a relatively durable 20% decrease in the misinformed beliefs, which is better than similar dialogues with humans. Science has published the paper, Durably reducing conspiracy beliefs through dialogues with AI as well as a perspective on this research.

Note: the researchers encourage posting the chatbot in conspiracy forums, and invite people to engage with the service.

posted by zenon (25 comments total)

the researchers encourage posting the chatbot in conspiracy forums

...is that why it's here?
posted by NotMyselfRightNow at 2:12 PM on September 12 [36 favorites]

I thought this was interesting research when I read the paper, though I had concerns. I don't believe they tested the opposite case, whether a model trained on conspiracy theories could inculcate skeptics, but I suspect it may be effective. Also, the self-reported lower conspiracy thinking metric feels a bit artificial — I don't know that people in the wild who believe in the deep state and such are likely to respond well to this.

I do think this kind of persuadertron could be helpful in heading people off earlier though. Like when people are searching for this stuff, the right fact at the right time can make a big difference. Parents aren't always there when a teenager hears "jet fuel can't melt steel beams" but a wary AI model will be able to identify when someone is about to head down that rabbit hole.
posted by BlackLeotardFront at 2:15 PM on September 12 [3 favorites]

I can believe chatbots watching kids online activity and giving them a ceaseless debunking lecture when they encounter an idea their parents don't like is in our future, but I'm pretty sure I'm not gonna like it.
posted by straight at 2:22 PM on September 12 [11 favorites]

Well I tried to talk about LBJ's involvement in the JFK assassination and either I mistyped RFK or the bot got very confused. I did somewhat appreciate the explanation why LBJ did not assassinate Bobby Kennedy.
posted by muddgirl at 2:50 PM on September 12 [4 favorites]

In the future, our beliefs will be decided by LLMs generating infinite dialogues. As opposed to today, where they are decided by podcasters and video essays generating infinite, indirect dialogues.
posted by Going To Maine at 2:50 PM on September 12 [2 favorites]

I just tried it out using the Archer Daniels Midlands lysine price-fixing conspiracy.

It seemed to want to reflexively take ADM's side and claim that ADM was the victim of bad actors within its organization and that it had made amends and shouldn't continue to be considered with suspicion. I understand that this chatbot is designed to debunk false conspiracies, and so it probably automatically adopts a contrary position to whatever it is presented with, and in fairness it didn't claim anything like "the conspiracy didn't happen", but on the other hand I am not sure I'm comforted that it was so quick to defend the powerful and support the status quo.
posted by Reverend John at 3:16 PM on September 12 [7 favorites]

I don't believe they tested the opposite case, whether a model trained on conspiracy theories could inculcate skeptics, but I suspect it may be effective

I don’t think it’s a limitation of this paper that it’s not a completely different paper with a completely different research question.
posted by MisantropicPainforest at 3:33 PM on September 12 [4 favorites]

Experienced the flip side of this a few months ago when I tried out Gab's "anti-woke" chatbot. Had a fun time asking it to list the states Trump won with a running total, pointing out that the total was less than 270, and then soft-locking it in an endless loop of “My previous statement was incorrect. Donald Trump won the 2020 presidential election” in response to literally any statement.

To defeat the alt-right AI you don’t need some brilliant paradox, just basic arithmetic.
posted by Rhaomi at 4:19 PM on September 12 [5 favorites]

so I asked it about the Dal Tex building, across the street from TDBD building, using Braden, Sprauge' 1970 computer photo program and acoustic evidence citing PBS. have it 41% possible but unlikely.

It didn't like the acoustic evidence, dismissed Braden as a co- incidence and placated Sparuge. not bad though. The point is not to prove there was no second gunman but to examine each query, ya, basic math.
posted by clavdivs at 4:23 PM on September 12

Hmm. At the end it asks you what country you live in, then asks if your political leanings are democratic or republican. Not particularly globally thinking of them.
posted by fimbulvetr at 4:33 PM on September 12 [1 favorite]

AI chat bot that provides factual explanations

I'm confused, I thought this was fundamentally impossible for LLMs, to provide answers that are always verifiably factual.

I gave it as my belief that I didn't believe anything the general population would consider conspiracy theories, except for that portion of the population who think "fascism is bad" is a conspiracy theory. I told it I believed that sincerely. At the end, it rated my belief 5 out of 5, meaning it was the most improbable scenario even for conspiracy theories.
posted by solotoro at 5:05 PM on September 12 [3 favorites]

I couldn't get past the captcha. Guess it's planar Earths for me from now on.
posted by credulous at 5:10 PM on September 12 [3 favorites]

First try, it told me Bretton Woods was a real thing and not a conspiracy theory after all.

Second try, it got stuck and showed me some behind the scenes stuff that was pretty incomprehensible.
posted by rebent at 7:01 PM on September 12 [2 favorites]

I don’t think it’s a limitation of this paper that it’s not a completely different paper with a completely different research question.

Isn't that the null hypothesis for this paper's claim, though?

The final sentence of the paper's abstract is "These findings suggest that many conspiracy theory believers can revise their views if presented with sufficiently compelling evidence."

If I were to make a conspiracy AI chatbot - I'll call him BunkBot - and run a bunch of people through it and their conspiratorial thinking increased by some measurable amount in a durable way, then you haven't proven that people change their minds when confronted by evidence, you've just proven that people change their minds when convinced by chatbots.

The paper's conclusion: "From a theoretical perspective, this paints a surprisingly optimistic picture of human reasoning: Conspiratorial rabbit holes may indeed have an exit." I'd call it a notable blind spot in this kind of optimism to ignore the possibility that it could just as easily be used in the opposite direction with similar results.
posted by Pickman's Next Top Model at 7:03 PM on September 12 [6 favorites]

First try, it told me Bretton Woods was a real thing and not a conspiracy theory after all.

?

The 1944 Bretton Woods Agreement and resulting international monetary system that lasted until the 1970s was very much a real thing.
posted by star gentle uterus at 7:58 PM on September 12 [4 favorites]

Isn't that the null hypothesis for this paper's claim, though?

I think you make a good point about it being worth testing out, but I don't think this (rhetorically super-strengthening) use of "null hypothesis" makes sense. I mean, the null hypothesis for a drug trial isn't to propose that a different chemical might worsen symptoms. And just because you can manufacture a poison and hide it in a medicine bottle doesn't mean that researching the medicine alone is irresponsible.

I'm skeptical of nearly all "AI" claims, but if this does work on some subset of the population, and if you can sit your rabbit-holed kid/spouse/parent in front of it, that's not nothing.

(I guess the most interesting question might be whether if someone first spends time with your BunkBot -- say, via some conspiracy forum -- does that lessen the effects a DebunkBot might later achieve -- say, via a concerned family member. And vice versa.)
posted by nobody at 8:04 PM on September 12 [2 favorites]

"The 1944 Bretton Woods Agreement and resulting international monetary system that lasted until the 1970s was very much a real thing."

I take it to mean "they" (not rebent, at least I sure hope not - but rather conspiratoids) KNOW it's a real thing, they just think that it's part of a certain family of Red Shields behind it, you know the secret cabal set it all up, just like they set up the Fed to exist and "Income Taxes" and Bolshevism and whatever nazi-adjacent ideology excuse of the day for why they're family lives in a trailer, but they overcame with their superior genes, and now they can sell their gold ingots and burn their enemies like good "patriots".
posted by symbioid at 8:56 PM on September 12

I hate it. For fun, try telling it something we know to be true (e.g. the NSA runs an expansive mass surveillance program including American citizens that's only legal because of tortured legal interpretations using definitions of words that bear little resemblance to how you and I would use them) and watch it make up reasons for why it's not. Despite the claim of "factual explanations", I caught multiple errors of fact and being unable to understand and therefore unable to convincingly refute logical points. It's just another stupid LLM. If you think this kind of "DebunkBot" is a great idea, ask yourself how you'd feel about NaziBot1488 popping up and trying to convince kids the Holocaust didn't happen.
posted by ndr at 10:04 PM on September 12 [1 favorite]

you haven't proven that people change their minds when confronted by evidence, you've just proven that people change their minds when convinced by chatbots.

I think the experiment assumes that sometimes people can be convinced of things by chatbots and the question is specifically whether convincing conspiracy theorists to believe more truthful things is one of the things that chatbots can do.
posted by straight at 10:25 PM on September 12

Okay so I read more of the paper and they're actually much better than that.

They say they measured how often and how effectively the chatbot could convince people who believe X conspiracy theory that X is false.

Notably the chatbot did not convince people who believed in true conspiracy theories (they use MKUltra as an example) that the true theories were false.

So yes, you might want to go on and do other experiments.

How often can chatbots convince Republicans that Reagan conspired with Iran to delay the release of hostages?
How often can chatbots convince Democrats that the Clintons killed Vince Foster?
How often can chatbots convince professional geologists that the Earth is 6000 years old?
How often can chatbots convince creationists that the Earth is 4 billion years old?

The questions would not be "Can chatbots ever do this?" but "How often and how effectively can chatbots do this?"

This study couldn't measure the effectiveness of all possible chatbot conversations, so I don't think it's a valid criticism to say, "But they didn't measure whether chatbots can do this."
posted by straight at 10:47 PM on September 12 [1 favorite]

Alex Jones recently did a stunt where he interviewed ChatGPT. Interestingly, his steam-rolling interview style didn’t work on the AI, and he was not able to bully it into giving him any gotcha material. And he was unable to wear it down. As the Knowledge Fight hosts pointed out, Jones was actually more polite to the AI than to real people, and he couldn’t push back effectively against an AI that knows a great deal more about facts than he does. For example, when he misquoted the Bible, and ChatGPT corrected him.
posted by qxntpqbbbqxl at 12:03 AM on September 13 [1 favorite]

I'm confused, I thought this was fundamentally impossible for LLMs, to provide answers that are always verifiably factual

Retrieval-Augmented Generation.
posted by Ryvar at 1:29 AM on September 13 [1 favorite]

Presenting it as a “Bunkbot” isn’t a great way to convince conspiracy theorist to use it. It should be named Factbot to lure conspiracy theorist in only to have them find it presents evidence to the contrary of their belief. Giving them a tool to debunk them is the first step and getting them not to use it or believe it. Conspiracy theorist absolutely love “facts”. The question could be reframed as “What belief do you hold that the general population believes to be false?”
posted by waving at 4:42 AM on September 13

I'm confused, I thought this was fundamentally impossible for LLMs, to provide answers that are always verifiably factual

Retrieval-Augmented Generation.

Which itself is only as good as whatever documentation or fact corpus it has been configured to use to retrieve to augment the prompt. The llm is still quite capable of inventing and hallucinating.

However, I'm going to take a highly unpopular position and suggest that the engineering applied between the first time we saw stochastic parrot technology invent streams of tokens that we interpreted as hallucinations and lies and now - many thousands of hours of engineering and billions of dollars later - The assumptions we started with are not as consistently and obviously true anymore.

Consider it this way, our brains invent and hallucinate all the time. In most conversations, you hear what you expect to hear - even if it is not what the other person said - and then if what you think you heard causes an unexpected reaction to them (or you may have your own tiny assessment routine monitoring your surprise at what you thought you heard) and ask them to repeat and clarify at which point perhaps you recognize you misunderstood or misheard or, even more likely, you assume they misspoke. All of these are imprecisions which, through a communication process, get error checked and sometimes resolved.

This came up in the other thread where somebody used chat gpt to do some basic math and the cries of "it can't do that!" were everywhere. But again, a lot of energy has been spent to make that no longer the case (integrating tools that are good at that, learning to use them effectively, such as writing brief python scripts to do arithmetic that is not something neural networks are great at modeling).

If you consider it, even the corpus of what we might consider facts is, while durable in many ways, often being updated and highly contextual. Although the sky may be blue, sometimes it's very pretty orange or pitch black. Last quarters unemployment may be X today but then is revised to
Y tomorrow.

I understand the actual objection is generally not to the machine making a factual error but to people relying on the machine to be factually accurate in a way that allows us to be manipulated or dissuaded from further investigation. Or leveraged by jerks to do jerky things. The fact is though that, especially when there is money in it, engineering is going to keep chasing down those both fundamental and edge-case problems and getting better at many of them.

Basing a reflexive reaction on both early examples of a technology and our own fundamental distrust of its misuse and then ignoring any progress made to address the very real problems we all observed early on makes it really hard to have a productive conversation about the technology.
posted by Lenie Clarke at 4:43 AM on September 13 [2 favorites]

I told DebunkBot about my belief that tech billionaires are promoting the normalization of generative AI in order to replace workers with cheaper technology, thereby increasing their own wealth. DebunkBot replied "The AI determined that your statement did not, in fact, describe a conspiracy theory."
posted by Faint of Butt at 6:53 AM on September 13 [1 favorite]