less effective on superficial misinformed beliefs

September 12, 2024 1:50 PM

Meet DebunkBot: an AI chat bot that provides factual explanations and counter-evidence for these conspiratorial events. It's strength appears to be that the LLM is inexhaustible and will argue indefinitely. They found that the targeted dialogues resulted in a relatively durable 20% decrease in the misinformed beliefs, which is better than similar dialogues with humans. Science has published the paper, Durably reducing conspiracy beliefs through dialogues with AI as well as a perspective on this research.

Note: the researchers encourage posting the chatbot in conspiracy forums, and invite people to engage with the service.

posted by zenon (32 comments total)

the researchers encourage posting the chatbot in conspiracy forums

...is that why it's here?
posted by NotMyselfRightNow at 2:12 PM on September 12, 2024 [42 favorites]

I thought this was interesting research when I read the paper, though I had concerns. I don't believe they tested the opposite case, whether a model trained on conspiracy theories could inculcate skeptics, but I suspect it may be effective. Also, the self-reported lower conspiracy thinking metric feels a bit artificial — I don't know that people in the wild who believe in the deep state and such are likely to respond well to this.

I do think this kind of persuadertron could be helpful in heading people off earlier though. Like when people are searching for this stuff, the right fact at the right time can make a big difference. Parents aren't always there when a teenager hears "jet fuel can't melt steel beams" but a wary AI model will be able to identify when someone is about to head down that rabbit hole.
posted by BlackLeotardFront at 2:15 PM on September 12, 2024 [6 favorites]

I can believe chatbots watching kids online activity and giving them a ceaseless debunking lecture when they encounter an idea their parents don't like is in our future, but I'm pretty sure I'm not gonna like it.
posted by straight at 2:22 PM on September 12, 2024 [14 favorites]

Well I tried to talk about LBJ's involvement in the JFK assassination and either I mistyped RFK or the bot got very confused. I did somewhat appreciate the explanation why LBJ did not assassinate Bobby Kennedy.
posted by muddgirl at 2:50 PM on September 12, 2024 [7 favorites]

In the future, our beliefs will be decided by LLMs generating infinite dialogues. As opposed to today, where they are decided by podcasters and video essays generating infinite, indirect dialogues.
posted by Going To Maine at 2:50 PM on September 12, 2024 [3 favorites]

I just tried it out using the Archer Daniels Midlands lysine price-fixing conspiracy.

It seemed to want to reflexively take ADM's side and claim that ADM was the victim of bad actors within its organization and that it had made amends and shouldn't continue to be considered with suspicion. I understand that this chatbot is designed to debunk false conspiracies, and so it probably automatically adopts a contrary position to whatever it is presented with, and in fairness it didn't claim anything like "the conspiracy didn't happen", but on the other hand I am not sure I'm comforted that it was so quick to defend the powerful and support the status quo.
posted by Reverend John at 3:16 PM on September 12, 2024 [14 favorites]

I don't believe they tested the opposite case, whether a model trained on conspiracy theories could inculcate skeptics, but I suspect it may be effective

I don’t think it’s a limitation of this paper that it’s not a completely different paper with a completely different research question.
posted by MisantropicPainforest at 3:33 PM on September 12, 2024 [5 favorites]

Experienced the flip side of this a few months ago when I tried out Gab's "anti-woke" chatbot. Had a fun time asking it to list the states Trump won with a running total, pointing out that the total was less than 270, and then soft-locking it in an endless loop of “My previous statement was incorrect. Donald Trump won the 2020 presidential election” in response to literally any statement.

To defeat the alt-right AI you don’t need some brilliant paradox, just basic arithmetic.
posted by Rhaomi at 4:19 PM on September 12, 2024 [8 favorites]

so I asked it about the Dal Tex building, across the street from TDBD building, using Braden, Sprauge' 1970 computer photo program and acoustic evidence citing PBS. have it 41% possible but unlikely.

It didn't like the acoustic evidence, dismissed Braden as a co- incidence and placated Sparuge. not bad though. The point is not to prove there was no second gunman but to examine each query, ya, basic math.
posted by clavdivs at 4:23 PM on September 12, 2024 [1 favorite]

Hmm. At the end it asks you what country you live in, then asks if your political leanings are democratic or republican. Not particularly globally thinking of them.
posted by fimbulvetr at 4:33 PM on September 12, 2024 [2 favorites]

AI chat bot that provides factual explanations

I'm confused, I thought this was fundamentally impossible for LLMs, to provide answers that are always verifiably factual.

I gave it as my belief that I didn't believe anything the general population would consider conspiracy theories, except for that portion of the population who think "fascism is bad" is a conspiracy theory. I told it I believed that sincerely. At the end, it rated my belief 5 out of 5, meaning it was the most improbable scenario even for conspiracy theories.
posted by solotoro at 5:05 PM on September 12, 2024 [6 favorites]

I couldn't get past the captcha. Guess it's planar Earths for me from now on.
posted by credulous at 5:10 PM on September 12, 2024 [4 favorites]

First try, it told me Bretton Woods was a real thing and not a conspiracy theory after all.

Second try, it got stuck and showed me some behind the scenes stuff that was pretty incomprehensible.
posted by rebent at 7:01 PM on September 12, 2024 [4 favorites]

I don’t think it’s a limitation of this paper that it’s not a completely different paper with a completely different research question.

Isn't that the null hypothesis for this paper's claim, though?

The final sentence of the paper's abstract is "These findings suggest that many conspiracy theory believers can revise their views if presented with sufficiently compelling evidence."

If I were to make a conspiracy AI chatbot - I'll call him BunkBot - and run a bunch of people through it and their conspiratorial thinking increased by some measurable amount in a durable way, then you haven't proven that people change their minds when confronted by evidence, you've just proven that people change their minds when convinced by chatbots.

The paper's conclusion: "From a theoretical perspective, this paints a surprisingly optimistic picture of human reasoning: Conspiratorial rabbit holes may indeed have an exit." I'd call it a notable blind spot in this kind of optimism to ignore the possibility that it could just as easily be used in the opposite direction with similar results.
posted by Pickman's Next Top Model at 7:03 PM on September 12, 2024 [8 favorites]

First try, it told me Bretton Woods was a real thing and not a conspiracy theory after all.

?

The 1944 Bretton Woods Agreement and resulting international monetary system that lasted until the 1970s was very much a real thing.
posted by star gentle uterus at 7:58 PM on September 12, 2024 [7 favorites]

Isn't that the null hypothesis for this paper's claim, though?

I think you make a good point about it being worth testing out, but I don't think this (rhetorically super-strengthening) use of "null hypothesis" makes sense. I mean, the null hypothesis for a drug trial isn't to propose that a different chemical might worsen symptoms. And just because you can manufacture a poison and hide it in a medicine bottle doesn't mean that researching the medicine alone is irresponsible.

I'm skeptical of nearly all "AI" claims, but if this does work on some subset of the population, and if you can sit your rabbit-holed kid/spouse/parent in front of it, that's not nothing.

(I guess the most interesting question might be whether if someone first spends time with your BunkBot -- say, via some conspiracy forum -- does that lessen the effects a DebunkBot might later achieve -- say, via a concerned family member. And vice versa.)
posted by nobody at 8:04 PM on September 12, 2024 [5 favorites]

"The 1944 Bretton Woods Agreement and resulting international monetary system that lasted until the 1970s was very much a real thing."

I take it to mean "they" (not rebent, at least I sure hope not - but rather conspiratoids) KNOW it's a real thing, they just think that it's part of a certain family of Red Shields behind it, you know the secret cabal set it all up, just like they set up the Fed to exist and "Income Taxes" and Bolshevism and whatever nazi-adjacent ideology excuse of the day for why they're family lives in a trailer, but they overcame with their superior genes, and now they can sell their gold ingots and burn their enemies like good "patriots".
posted by symbioid at 8:56 PM on September 12, 2024 [2 favorites]

I hate it. For fun, try telling it something we know to be true (e.g. the NSA runs an expansive mass surveillance program including American citizens that's only legal because of tortured legal interpretations using definitions of words that bear little resemblance to how you and I would use them) and watch it make up reasons for why it's not. Despite the claim of "factual explanations", I caught multiple errors of fact and being unable to understand and therefore unable to convincingly refute logical points. It's just another stupid LLM. If you think this kind of "DebunkBot" is a great idea, ask yourself how you'd feel about NaziBot1488 popping up and trying to convince kids the Holocaust didn't happen.
posted by ndr at 10:04 PM on September 12, 2024 [5 favorites]

you haven't proven that people change their minds when confronted by evidence, you've just proven that people change their minds when convinced by chatbots.

I think the experiment assumes that sometimes people can be convinced of things by chatbots and the question is specifically whether convincing conspiracy theorists to believe more truthful things is one of the things that chatbots can do.
posted by straight at 10:25 PM on September 12, 2024 [1 favorite]

Okay so I read more of the paper and they're actually much better than that.

They say they measured how often and how effectively the chatbot could convince people who believe X conspiracy theory that X is false.

Notably the chatbot did not convince people who believed in true conspiracy theories (they use MKUltra as an example) that the true theories were false.

So yes, you might want to go on and do other experiments.

How often can chatbots convince Republicans that Reagan conspired with Iran to delay the release of hostages?
How often can chatbots convince Democrats that the Clintons killed Vince Foster?
How often can chatbots convince professional geologists that the Earth is 6000 years old?
How often can chatbots convince creationists that the Earth is 4 billion years old?

The questions would not be "Can chatbots ever do this?" but "How often and how effectively can chatbots do this?"

This study couldn't measure the effectiveness of all possible chatbot conversations, so I don't think it's a valid criticism to say, "But they didn't measure whether chatbots can do this."
posted by straight at 10:47 PM on September 12, 2024 [2 favorites]

Alex Jones recently did a stunt where he interviewed ChatGPT. Interestingly, his steam-rolling interview style didn’t work on the AI, and he was not able to bully it into giving him any gotcha material. And he was unable to wear it down. As the Knowledge Fight hosts pointed out, Jones was actually more polite to the AI than to real people, and he couldn’t push back effectively against an AI that knows a great deal more about facts than he does. For example, when he misquoted the Bible, and ChatGPT corrected him.
posted by qxntpqbbbqxl at 12:03 AM on September 13, 2024 [3 favorites]

I'm confused, I thought this was fundamentally impossible for LLMs, to provide answers that are always verifiably factual

Retrieval-Augmented Generation.
posted by Ryvar at 1:29 AM on September 13, 2024 [3 favorites]

Presenting it as a “Bunkbot” isn’t a great way to convince conspiracy theorist to use it. It should be named Factbot to lure conspiracy theorist in only to have them find it presents evidence to the contrary of their belief. Giving them a tool to debunk them is the first step and getting them not to use it or believe it. Conspiracy theorist absolutely love “facts”. The question could be reframed as “What belief do you hold that the general population believes to be false?”
posted by waving at 4:42 AM on September 13, 2024 [1 favorite]

I'm confused, I thought this was fundamentally impossible for LLMs, to provide answers that are always verifiably factual

Retrieval-Augmented Generation.

Which itself is only as good as whatever documentation or fact corpus it has been configured to use to retrieve to augment the prompt. The llm is still quite capable of inventing and hallucinating.

However, I'm going to take a highly unpopular position and suggest that the engineering applied between the first time we saw stochastic parrot technology invent streams of tokens that we interpreted as hallucinations and lies and now - many thousands of hours of engineering and billions of dollars later - The assumptions we started with are not as consistently and obviously true anymore.

Consider it this way, our brains invent and hallucinate all the time. In most conversations, you hear what you expect to hear - even if it is not what the other person said - and then if what you think you heard causes an unexpected reaction to them (or you may have your own tiny assessment routine monitoring your surprise at what you thought you heard) and ask them to repeat and clarify at which point perhaps you recognize you misunderstood or misheard or, even more likely, you assume they misspoke. All of these are imprecisions which, through a communication process, get error checked and sometimes resolved.

This came up in the other thread where somebody used chat gpt to do some basic math and the cries of "it can't do that!" were everywhere. But again, a lot of energy has been spent to make that no longer the case (integrating tools that are good at that, learning to use them effectively, such as writing brief python scripts to do arithmetic that is not something neural networks are great at modeling).

If you consider it, even the corpus of what we might consider facts is, while durable in many ways, often being updated and highly contextual. Although the sky may be blue, sometimes it's very pretty orange or pitch black. Last quarters unemployment may be X today but then is revised to
Y tomorrow.

I understand the actual objection is generally not to the machine making a factual error but to people relying on the machine to be factually accurate in a way that allows us to be manipulated or dissuaded from further investigation. Or leveraged by jerks to do jerky things. The fact is though that, especially when there is money in it, engineering is going to keep chasing down those both fundamental and edge-case problems and getting better at many of them.

Basing a reflexive reaction on both early examples of a technology and our own fundamental distrust of its misuse and then ignoring any progress made to address the very real problems we all observed early on makes it really hard to have a productive conversation about the technology.
posted by Lenie Clarke at 4:43 AM on September 13, 2024 [7 favorites]

I told DebunkBot about my belief that tech billionaires are promoting the normalization of generative AI in order to replace workers with cheaper technology, thereby increasing their own wealth. DebunkBot replied "The AI determined that your statement did not, in fact, describe a conspiracy theory."
posted by Faint of Butt at 6:53 AM on September 13, 2024 [11 favorites]

Right. As long as that isn't followed by "Thank you, citizen. Please stand by while DeplotBot terminates this factual conspiracy." this feels a bit incomplete?
posted by Ashenmote at 8:16 AM on September 13, 2024 [1 favorite]

even the corpus of what we might consider facts is, while durable in many ways, often being updated and highly contextual.

Indeed, but what constitutes a fact in a strict sense relies on trust, accountability, attribution, reproducibility, and authority. This is a fundamental issue with neural net LLMs, even those like Copilot that sort of make an effort
posted by aspersioncast at 10:17 AM on September 13, 2024 [2 favorites]

I don’t think it’s a limitation of this paper that it’s not a completely different paper with a completely different research question.

Isn't that the null hypothesis for this paper's claim, though?


That's not what a null hypothesis is.
posted by MisantropicPainforest at 10:25 AM on September 13, 2024 [1 favorite]

Indeed, but what constitutes a fact in a strict sense relies on trust, accountability, attribution, reproducibility, and authority. This is a fundamental issue with neural net LLMs, even those like Copilot that sort of make an effort

A fact that meets all of the criteria you have outlined does not cease to be a fact if it is provided in the output of a large language model. You note that some such models attempt effort toward this exact goal but then fall back to it being somehow fundamentally impossible.

What is the technical or I guess philosophical basis for your assertion that testing those characteristics is fundamentally outside the capabilities of a neural network LLM? If you can show me how to test the factuality of a statement using a tool the same way I can test the correct answer to an arithmetic problem using a tool then there's nothing fundamental about a tool-enabled model using the exact same tool to test factuality. But that factuality testing tool is the real hard part - in some ways we know it when we see it because we recognize some fellow mind state in it.

Again, early hype versions of this technology hoped against all good sense that a bunch of hyper dimensional numbers connected by learned weights could spontaneously configure to behave as calculating machines or other such deterministically solved problems with predictably terrible results (what's pretty amazing is that sometimes those configurations did get close, such as those modeling an othello board albeit imprecisely). That doesn't mean the people doing the work gave up trying and said screw it, just keep shipping it.

Naturally, it is very hard to test facts, especially those presented in context. It often seems what people are reacting to so viscerally with llm's is that we lack the cues we can normally rely upon to determine who qualifies as an authority, worthy of trust, especially can provide accountability in a way that they will correct themselves if proven wrong, all of which add up to some rating of trust. As soon as we replace all of that assessment with a calculated or aggregated rating, I can totally see how it feels like it is extremely fragile and a likely point of attack for jerks who want to make it hard to understand what's true and what's not.

I'm saying I'm empathetic to the position but I don't think if you think it all the way through that it can hold.
posted by Lenie Clarke at 10:54 AM on September 13, 2024 [1 favorite]

It often seems what people are reacting to so viscerally with llm's is that we lack the cues we can normally rely upon to determine who qualifies as an authority, worthy of trust, especially can provide accountability in a way that they will correct themselves if proven wrong, all of which add up to some rating of trust.

What a lot of people don't get is that this not a skill one person has but a product of the social relationships of humanity. It takes decades for a human being to create the kinds of relationships that enable the human to be a trusted part of the human endeavor to figure stuff out. There is no algorithmic shortcut to becoming a trusted part of the human network.

If a computer wants to be as "smart" as human beings, it must be not as smart as a person but as smart as humanity.
posted by straight at 2:00 PM on September 13, 2024 [2 favorites]

If a computer wants to be as "smart" as human beings, it must be not as smart as a person but as smart as humanity.

Computers don't want to do anything. Some humans want computers to replace people because people are expensive - on the rampant capitalist side they might imagine this prevents the true power of labor from being wielded and thus has almost infinite upside even if it takes enormous investment. Essentially, slavery is the ultimate goal of capital and always has been.

Those humans are cunts with backwards priorities.

Other humans want computers to do things humans can do but it's really expensive to train to do well, and even when you do there's always more information coming in than you can become expert at. These humans also suspect that computers might be able to be configured to have fewer of the human biases that may underlie the often terrible experiences we have with experts because of exactly the capital pressure to squeeze profit over quality of work product. The thinking goes that expert humans can use even more expert machines to achieve significant progress faster than they could ever do using only themselves and other humans, and as a happy side effect maybe basic medical and psychological care becomes effectively no cost.

But you'll probably have a pretty high bar of trust for the second case. When a human graduates medical school it's not just the MCAT and the grades in that school but the whole network of authority and trust that we as a society imbue in that process. This is your human network of trust, and you're not wrong about it - it exists because frauds are deadly.

But that network also exists because of back room deals, racism, misogyny, hard won victories, setbacks, memory, grudges, and the power of government backed up with a monopoly on violence and punishment. I can't help but not see that as a net positive on the side of human experience - most of our institutions and networks of human trust exist atop a pile of bodies and forgotten or suppressed alternatives.

So... it sure seems like trying to find a means to derive credibility from sources other than exactly the way we've always done it might be worth exploring. Teasing apart the society and network of memory, memory of iterated prisoners dilemma outcomes and observed behaviors from the assumptions, stereotypes, biases, and genetic predispositions to tribalism.

The fact the first many attempts we tried reflected our own biases means we have the opportunity to measure that failure and do better. The only people who want to ignore that and skip to the crushing of labor are the aforementioned cunts. The rest of us are really trying to be the generation that builds something better than its worst impulses.
posted by Lenie Clarke at 3:24 PM on September 13, 2024 [5 favorites]

ask yourself how you'd feel about NaziBot1488 popping up and trying to convince kids the Holocaust didn't happen

this probably exists, with a modestly less obvious name.
posted by Glinn at 6:55 PM on September 13, 2024 [1 favorite]