The Cyber-Ombudsman
January 29, 2013 8:24 AM Subscribe
TruthTeller is an ambitious new automated application built by the Washington Post, which fact checks political speeches, ads and interviews "in as close to real time as possible." The prototype is intended to be a complement to the paper's Fact Checker Blog. More on the project from TechCrunch and Poynter.
It's just a small shell script that responds "LIES LIES LIES," right?
Because if it isn't, they've made it way too complicated.
posted by entropicamericana at 8:33 AM on January 29, 2013 [4 favorites]
Because if it isn't, they've made it way too complicated.
posted by entropicamericana at 8:33 AM on January 29, 2013 [4 favorites]
Thinking that "facts" matter is so The Past.
posted by The Card Cheat at 8:35 AM on January 29, 2013 [2 favorites]
posted by The Card Cheat at 8:35 AM on January 29, 2013 [2 favorites]
From the automatic fact. check of Rep. Conolly's speech:
Transcript text
recovery act which i probably supported cut taxes but ninety five percent of all americans
Original fact
The Recovery Act saved or created millions of jobs and cut taxes for 95 percent of the American people.
Probability: false
The problem with NLP is that while it is often very effective, an embarrassing mistake is still an embarrassing mistake. Here's to its future iterations. Let's hope one of them is letting users mark up errors.
posted by Going To Maine at 8:35 AM on January 29, 2013 [1 favorite]
Transcript text
recovery act which i probably supported cut taxes but ninety five percent of all americans
Original fact
The Recovery Act saved or created millions of jobs and cut taxes for 95 percent of the American people.
Probability: false
The problem with NLP is that while it is often very effective, an embarrassing mistake is still an embarrassing mistake. Here's to its future iterations. Let's hope one of them is letting users mark up errors.
posted by Going To Maine at 8:35 AM on January 29, 2013 [1 favorite]
Only a matter of time before we have this running in the debate auditorium, connected to electric shock generators. THAT, I'd pay money to see.
If only I could remember the comedy routine that predicted this.... I remember seeing it once...
posted by JoeZydeco at 9:01 AM on January 29, 2013
If only I could remember the comedy routine that predicted this.... I remember seeing it once...
posted by JoeZydeco at 9:01 AM on January 29, 2013
"I don't deserve this kind of shabby treatment."
BZZT
posted by entropicamericana at 9:04 AM on January 29, 2013 [1 favorite]
BZZT
posted by entropicamericana at 9:04 AM on January 29, 2013 [1 favorite]
And now the way is cleared for automated NPC politicians that are elected on their ability to game the TruthTeller software...
posted by RobotVoodooPower at 9:11 AM on January 29, 2013 [1 favorite]
posted by RobotVoodooPower at 9:11 AM on January 29, 2013 [1 favorite]
Has anyone thought to ask the Cyber-Ombudsman its opinion of gold?
posted by Xenophon Fenderson at 9:30 AM on January 29, 2013
posted by Xenophon Fenderson at 9:30 AM on January 29, 2013
Yeah--that example above seems particularly egregious, because if you read the actual source for the determination that this fact is "false," it's basically a summary judgment that the claim is half true based on a bunch of weasel-words and handwaving and a cite to another fact-checker that ruled the claim was true.
This hyper-"evenhanded," comically literal minded approach to fact checking is too easily subject to political gamesmanship.
Evaluated by its own overly-literal-minded and reductionist logic (i.e., that if any finer point of a claim can by some convoluted reasoning be judged as not quite true, the entire claim is "False"), this one bad example invalidates the entire premise. The tool would be forced to give the validity of its own fact-checking claims an evaluation of "False."
tl;dr: At least needs middle ground classifications for facts that are mostly true or mostly false. Ordinary language claims are too imprecise and context-sensitive to be evaluated in the same way as rigorous logical propositions in propositional calculus without far more analytic unpacking of the terms of the claim than can effectively be automated.
posted by saulgoodman at 9:37 AM on January 29, 2013
This hyper-"evenhanded," comically literal minded approach to fact checking is too easily subject to political gamesmanship.
Evaluated by its own overly-literal-minded and reductionist logic (i.e., that if any finer point of a claim can by some convoluted reasoning be judged as not quite true, the entire claim is "False"), this one bad example invalidates the entire premise. The tool would be forced to give the validity of its own fact-checking claims an evaluation of "False."
tl;dr: At least needs middle ground classifications for facts that are mostly true or mostly false. Ordinary language claims are too imprecise and context-sensitive to be evaluated in the same way as rigorous logical propositions in propositional calculus without far more analytic unpacking of the terms of the claim than can effectively be automated.
posted by saulgoodman at 9:37 AM on January 29, 2013
And now the way is cleared for automated NPC politicians that are elected on their ability to game the TruthTeller software...
If the 2012 US Presidential election taught us nothing, it's that we're a long way from perfecting a workable version of that software. Romney-bot shouldn't have been released to the public, even for testing.
posted by MCMikeNamara at 9:48 AM on January 29, 2013 [1 favorite]
If the 2012 US Presidential election taught us nothing, it's that we're a long way from perfecting a workable version of that software. Romney-bot shouldn't have been released to the public, even for testing.
posted by MCMikeNamara at 9:48 AM on January 29, 2013 [1 favorite]
If the 2012 US Presidential election taught us nothing, it's that we're a long way from perfecting a workable version of that software. Romney-bot shouldn't have been released to the public, even for testing.
Romney-bot only looked bad in comparison to human beings and the Obamatron, a more sophisticated corporate ATM with slightly higher service charges, lower withdrawal ceilings and better rewards programs.
posted by srboisvert at 10:42 AM on January 29, 2013 [1 favorite]
Romney-bot only looked bad in comparison to human beings and the Obamatron, a more sophisticated corporate ATM with slightly higher service charges, lower withdrawal ceilings and better rewards programs.
posted by srboisvert at 10:42 AM on January 29, 2013 [1 favorite]
I'm pretty sure that the most perfectly-performing, rapid fact-checking software possible isn't going to have much of an effect on the US electorate. Most will continue to do what brain research tells us they already do: fervently believe evidence that supports their convictions, and discount anything contradictory.
One-fifth of the Republican party believes Obama may not be a US citizen. One-quarter believes Obama is Muslim. With this much of a disconnect from reality, who are they going to listen to? Some pill-addicted pundit fresh from a tour of a pedophile-friendly Asian nation, or some fancy, 'lectronic com-pew-ter?
Heck, a com-pew-ter turned me into a newt. Cain't trust'em.
posted by IAmBroom at 12:12 PM on January 29, 2013
One-fifth of the Republican party believes Obama may not be a US citizen. One-quarter believes Obama is Muslim. With this much of a disconnect from reality, who are they going to listen to? Some pill-addicted pundit fresh from a tour of a pedophile-friendly Asian nation, or some fancy, 'lectronic com-pew-ter?
Heck, a com-pew-ter turned me into a newt. Cain't trust'em.
posted by IAmBroom at 12:12 PM on January 29, 2013
I'd expect wikipedia must help enormously already, but obviously breaking it down by quotes should help differently.
posted by jeffburdges at 7:08 PM on January 29, 2013
posted by jeffburdges at 7:08 PM on January 29, 2013
« Older Gnome place like home | X-Mensch Newer »
This thread has been archived and is closed to new comments
posted by valkyryn at 8:33 AM on January 29, 2013