Twitter algorithm always chooses white faces for picture thumbnails
September 20, 2020 8:19 AM Subscribe
You can see a response from David Dantley, Chief Design Officer at Twitter. Unfortunately his first response is some made up technical explanation that turns out not to be true. But he is appropriately upset and says they have to fix it, so that's a little encouraging.
This whole mess was first noticed recently by Colin Madland. As he points out, this kind of racist algorithm literally ruins peoples lives by leading to false arrests among other things.
posted by Nelson at 8:50 AM on September 20, 2020 [8 favorites]
This whole mess was first noticed recently by Colin Madland. As he points out, this kind of racist algorithm literally ruins peoples lives by leading to false arrests among other things.
posted by Nelson at 8:50 AM on September 20, 2020 [8 favorites]
Ah, I was looking for that Zoom decapitation thread, thanks!
posted by Tom-B at 8:50 AM on September 20, 2020
posted by Tom-B at 8:50 AM on September 20, 2020
Thank you, I saw this the other day and was having a hard time understanding what I was looking at and now I do. Has anyone actually been able to describe what is happening in any way that isn't 100% "Twitter picks the White face" because, gosh, Twitter sure is picking the White faces there.
posted by jessamyn at 8:52 AM on September 20, 2020 [3 favorites]
posted by jessamyn at 8:52 AM on September 20, 2020 [3 favorites]
It happens with fictional characters, too.
posted by bitteschoen at 9:08 AM on September 20, 2020 [1 favorite]
posted by bitteschoen at 9:08 AM on September 20, 2020 [1 favorite]
It is "Twitter picks the white face", but in the format of "Twitter does A/B tests on its thumbnail crops to see which gets the most clickthroughs".
So it's reflecting user prejudice back at them. Which we know full well is not the way to do things. Machine learning needs to be explainable precisely in order to make sure that this doesn't happen. And it needs to be monitored specifically in order to remove implicit bias.
posted by ambrosen at 9:20 AM on September 20, 2020 [9 favorites]
So it's reflecting user prejudice back at them. Which we know full well is not the way to do things. Machine learning needs to be explainable precisely in order to make sure that this doesn't happen. And it needs to be monitored specifically in order to remove implicit bias.
posted by ambrosen at 9:20 AM on September 20, 2020 [9 favorites]
Worth mentioning that PoC have been pointing this out for over a decade.
posted by basalganglia at 9:22 AM on September 20, 2020 [4 favorites]
posted by basalganglia at 9:22 AM on September 20, 2020 [4 favorites]
The go-to answer is to blame it on the algorithm, but algorithms are built with assumptions in mind, and the assumptions with Zoom at least are that everyone who uses the tech will look like the people who developed and tested the tech who--because diversity is still a massive problem in the technology industry (and countless other fields)--are almost all white. The result is of course racist. It might be unintentional, but that doesn't make it also not racist.
The video of the racist soap dispenser was making its rounds a few years back; and racism is built into the core of photography itself. That doesn't make it not a problem and/or not solvable; but to make any progress on solving it companies would have to make EDI an actual priority rather than a line on their annual report.
posted by johnofjack at 9:31 AM on September 20, 2020 [11 favorites]
The video of the racist soap dispenser was making its rounds a few years back; and racism is built into the core of photography itself. That doesn't make it not a problem and/or not solvable; but to make any progress on solving it companies would have to make EDI an actual priority rather than a line on their annual report.
posted by johnofjack at 9:31 AM on September 20, 2020 [11 favorites]
A while back I saw Netflix claim they were using algorithms to pick frames for key art. And then I saw on Netflix, Do The Right Thing using as key art the guy who owns a brownstone and was born in Brooklyn.
posted by RobotHero at 9:32 AM on September 20, 2020 [5 favorites]
posted by RobotHero at 9:32 AM on September 20, 2020 [5 favorites]
We mostly use Skype at work, but we had a Zoom conference once and we had a lot of problem with some of the Engineers not showing up properly on camera, and this explains what the problem was.
posted by jacquilynne at 9:41 AM on September 20, 2020
posted by jacquilynne at 9:41 AM on September 20, 2020
We regret to inform you that oximeters are also racist
posted by Tom-B at 10:12 AM on September 20, 2020 [8 favorites]
posted by Tom-B at 10:12 AM on September 20, 2020 [8 favorites]
This reply testing it seems to have made the algorithm flip to the other person. On that thread, they’re speculating that it has to do with contrast detection. Doesn’t really matter why it happens if the end result is racist.
posted by montag2k at 10:19 AM on September 20, 2020
posted by montag2k at 10:19 AM on September 20, 2020
Perfectly handled by Better off Ted way back in 2009: Racial Sensitivity
"The company's position is that it's actually the opposite of racist because it's not targeting black people. It's just ignoring them."
posted by MengerSponge at 10:22 AM on September 20, 2020 [13 favorites]
"The company's position is that it's actually the opposite of racist because it's not targeting black people. It's just ignoring them."
posted by MengerSponge at 10:22 AM on September 20, 2020 [13 favorites]
The entire Twitter thread in the link is in Portuguese. Is that the case for others? Is it some misdirected discrimination because I am viewing this from Puerto Rico?
posted by dances_with_sneetches at 10:53 AM on September 20, 2020
posted by dances_with_sneetches at 10:53 AM on September 20, 2020
It seems to be a conversation among Portuguese speaking twitter users?
posted by jacquilynne at 11:05 AM on September 20, 2020 [3 favorites]
posted by jacquilynne at 11:05 AM on September 20, 2020 [3 favorites]
I'm the OP, I'm Brazilian, and this is making the rounds of our corner of Twitter these days, that's why the thread is in Portuguese. I thought that the images were self-explanatory enough to post it. I was looking for Colin's thread explaining it in English, couldn't find it, and figured that soon enough someone would post it in the comments, as it happened. You can also pop the link url into Google Translate, and if something doesn't make sense in auto-translate, ask me via MeMail, I'll be glad to translate.
posted by Tom-B at 11:06 AM on September 20, 2020 [21 favorites]
posted by Tom-B at 11:06 AM on September 20, 2020 [21 favorites]
When I first came across this discourse, I was extremely confused, because the thumbnails were blank, with no people visible. That's because I was using the app Tweetbot by Tapbots, which does not use this algorithm and just shows the center of a cropped image. Kudos to Tapbots for not trying to increase engagement by excluding a sizable chunk of the world population.
posted by ejs at 11:42 AM on September 20, 2020 [13 favorites]
posted by ejs at 11:42 AM on September 20, 2020 [13 favorites]
20 articles on algorithmic racism, computer vision and facial recognition (auto-translated Twitter thread)
posted by Tom-B at 11:58 AM on September 20, 2020 [1 favorite]
posted by Tom-B at 11:58 AM on September 20, 2020 [1 favorite]
Years ago, way ago in Internet time, maybe 2002? even 1999? I remember image analysts working on auto-detecting porn being wildly relieved that an algorithm trained on pictures of white people was equally successful with pictures of black people. (IIRC they assumed they were going to have to develop and run a bunch of separate tests on each image.) On inspection, their algorithm wasn’t looking for light or dark, but the veiled red of blood beneath the skin.
Where did that algorithm go, and why are the training sets worse?
posted by clew at 12:55 PM on September 20, 2020 [3 favorites]
Where did that algorithm go, and why are the training sets worse?
posted by clew at 12:55 PM on September 20, 2020 [3 favorites]
You could probably detect porn using Fourier analysis of the motion vectors alone
posted by sjswitzer at 1:20 PM on September 20, 2020 [2 favorites]
posted by sjswitzer at 1:20 PM on September 20, 2020 [2 favorites]
And then make porn using innocuous household objects. Huh.
(They were trying to make the img tag safe for comments, I think. An innocent world.)
posted by clew at 1:26 PM on September 20, 2020 [1 favorite]
(They were trying to make the img tag safe for comments, I think. An innocent world.)
posted by clew at 1:26 PM on September 20, 2020 [1 favorite]
The video of the racist soap dispenser was making its rounds a few years back; and racism is built into the core of photography itself.
had the thought a couple weeks ago: what if it turns out black & brown people are safe from the machine revolution because the murderbots, like the soap dispensers, are unable to see them?
posted by taquito sunrise at 2:26 PM on September 20, 2020 [10 favorites]
had the thought a couple weeks ago: what if it turns out black & brown people are safe from the machine revolution because the murderbots, like the soap dispensers, are unable to see them?
posted by taquito sunrise at 2:26 PM on September 20, 2020 [10 favorites]
had the thought a couple weeks ago: what if it turns out black & brown people are safe from the machine revolution because the murderbots, like the soap dispensers, are unable to see them?
I just had the opposite thought -- as a big believer in autonomous driving (/user of Tesla Autopilot) I certainly hope that they are training their algorithms better than Twitter or soap dispensers. Also, regulators need to be very aware of this in more and more areas.
posted by zeikka at 3:42 PM on September 20, 2020 [6 favorites]
I just had the opposite thought -- as a big believer in autonomous driving (/user of Tesla Autopilot) I certainly hope that they are training their algorithms better than Twitter or soap dispensers. Also, regulators need to be very aware of this in more and more areas.
posted by zeikka at 3:42 PM on September 20, 2020 [6 favorites]
had the thought a couple weeks ago: what if it turns out black & brown people are safe from the machine revolution because the murderbots, like the soap dispensers, are unable to see them?
Bring ignored or misidentified by sensors is as deadly as being targeted.
I just had the opposite thought -- as a big believer in autonomous driving (/user of Tesla Autopilot) I certainly hope that they are training their algorithms better than Twitter or soap dispensers.
For example: Last year, Georgia Tech researchers found that people with dark skin were 10% less likely to be detected by the sensors used by self driving cars than people with light skin. Summary from the Guardian.
But, worse, this and similar technology is likely to be used to target POC, because it's being implemented by a racist security apparatus. So inaccurate findings will be used to wrongly identify POC as criminals, and the police will then brutalise them. Recently, facial recognition systems wrongly identified a black man in Detroit as the perpetrator of a shoplifting theft, and he was predictably treated extremely poorly. He had an alibi, and they didn't even bother checking.
posted by His thoughts were red thoughts at 4:13 PM on September 20, 2020 [8 favorites]
Bring ignored or misidentified by sensors is as deadly as being targeted.
I just had the opposite thought -- as a big believer in autonomous driving (/user of Tesla Autopilot) I certainly hope that they are training their algorithms better than Twitter or soap dispensers.
For example: Last year, Georgia Tech researchers found that people with dark skin were 10% less likely to be detected by the sensors used by self driving cars than people with light skin. Summary from the Guardian.
But, worse, this and similar technology is likely to be used to target POC, because it's being implemented by a racist security apparatus. So inaccurate findings will be used to wrongly identify POC as criminals, and the police will then brutalise them. Recently, facial recognition systems wrongly identified a black man in Detroit as the perpetrator of a shoplifting theft, and he was predictably treated extremely poorly. He had an alibi, and they didn't even bother checking.
posted by His thoughts were red thoughts at 4:13 PM on September 20, 2020 [8 favorites]
I'm seeing a lot of speculation about what sort of algorithm Twitter might be using, but the engineering team already disclosed in 2018 that they're using saliency detection, which is a model trained on eyeball trackers that determined which parts of different images people were likely to focus on. "In general, people tend to pay more attention to faces, text, animals, but also other objects and regions of high contrast."
As others mentioned, the result can still be racist, and merely seeing this kind of automated discrimination in action (regardless of intent or actual statistical results) is traumatizing for some people, which is worth keeping in mind when making decisions about deploying technology like this.
posted by Ryon at 4:52 PM on September 20, 2020 [5 favorites]
As others mentioned, the result can still be racist, and merely seeing this kind of automated discrimination in action (regardless of intent or actual statistical results) is traumatizing for some people, which is worth keeping in mind when making decisions about deploying technology like this.
posted by Ryon at 4:52 PM on September 20, 2020 [5 favorites]
I’ve seen arguments that contrast issues make detecting darker faces more difficult. But the thing is that detecting faces is difficult already. The reason we can detect faces at all is that researchers chose to address that hard problem. If they don’t detect darker faces as well is only because it wasn’t as important to them (whether consciously or not) to tackle that problem.
As noted above the same problem happened with photographic film. Once film could capture the faces they cared about... good enough.
posted by sjswitzer at 5:21 PM on September 20, 2020 [2 favorites]
As noted above the same problem happened with photographic film. Once film could capture the faces they cared about... good enough.
posted by sjswitzer at 5:21 PM on September 20, 2020 [2 favorites]
I wonder if the same thing happens when photos are converted to L*a*b* space? I seem to recall that everyone's about the same, or at least within a standard deviation, in the b* channel....
posted by notsnot at 5:59 PM on September 20, 2020
posted by notsnot at 5:59 PM on September 20, 2020
I hope this means there will be further testing of edge cases and openness on the part of Twitter as to what went wrong and what they did to change it. While Silicon Valley likes to beat their chests on how proprietary their models are, the big secret is that most use a handful of models based on well known research. Facial recognition is already an incredibly touchy topic due the legality surrounding it. I doubt it is as simple as there being contrast issues but you never know. I'm sure in whatever test cases the developers used to develop the software "giant vertical image with two faces" was not one of them. That said even beyond race, how machine learning or advanced statistical analysis determines things is a hot button issue. A credit firm approached me due to their lack of understanding how they tag fraudulent activity, or rather I should say the frustration from upper management when well-heeled firms get denied and they can't tell them exactly why. So this is not really just limited to race.
posted by geoff. at 10:07 PM on September 20, 2020
posted by geoff. at 10:07 PM on September 20, 2020
I am far from an expert, but:
My understanding is that salience in images is not a universal homogeneous quality but differs from person to person including on racial lines, and this difference is particularly prominent when several objects of interest are present in an image. Further, there are studies showing differences between individuals looking at people of their own race, vs those of another race. such as Adults Scan Own- and Other-Race Faces Differently.
Most saliency eye tracking data sets are, afaik, from college students, and my admittedly cursory efforts couldn't find any studies on racial differences in saliency directly, but college students are majority white, and most eye tracking is done on college students.
Of course the bias can come the other way too, given how few images of black people there are in many image saliency detection datasets.
They don't specifically cite their datasets or training sets in their blog post, but the person in that blog post still at twitter suggested that they did do racial and gender bias testing initially for their salience detection feature.
posted by Chrysopoeia at 10:19 PM on September 20, 2020 [4 favorites]
My understanding is that salience in images is not a universal homogeneous quality but differs from person to person including on racial lines, and this difference is particularly prominent when several objects of interest are present in an image. Further, there are studies showing differences between individuals looking at people of their own race, vs those of another race. such as Adults Scan Own- and Other-Race Faces Differently.
Most saliency eye tracking data sets are, afaik, from college students, and my admittedly cursory efforts couldn't find any studies on racial differences in saliency directly, but college students are majority white, and most eye tracking is done on college students.
Of course the bias can come the other way too, given how few images of black people there are in many image saliency detection datasets.
They don't specifically cite their datasets or training sets in their blog post, but the person in that blog post still at twitter suggested that they did do racial and gender bias testing initially for their salience detection feature.
posted by Chrysopoeia at 10:19 PM on September 20, 2020 [4 favorites]
What I would like to know is why the cropping is automated in the first place. What problem does the it solve for Twitter as a business?
Why not just let users crop the thumbnails themselves or default to centering the image? Is it so awful to let users control this? Or is it that somebody learned how to do machine learning and it sounded like a cool idea, so who cares about the benefits or problems it brings?
posted by harriet vane at 4:42 AM on September 21, 2020 [2 favorites]
Why not just let users crop the thumbnails themselves or default to centering the image? Is it so awful to let users control this? Or is it that somebody learned how to do machine learning and it sounded like a cool idea, so who cares about the benefits or problems it brings?
posted by harriet vane at 4:42 AM on September 21, 2020 [2 favorites]
Presumably, at that scale, a significant proportion if not vast majority of users couldn't be bothered to think about cropping, and naïve autocropping would result in suboptimal user experience, lower engagement and thus advertising revenue being left on the table. So there's money in making everything look as sugary, sexy and primally addictive as possible. The fact that engagement-maximisation algorithms trained on data including racial bias would reproduce said bias was presumably not considered.
posted by acb at 6:53 AM on September 21, 2020
posted by acb at 6:53 AM on September 21, 2020
Why not just let users crop the thumbnails themselves or default to centering the image? Is it so awful to let users control this? Or is it that somebody learned how to do machine learning and it sounded like a cool idea, so who cares about the benefits or problems it brings?
Because the vast majority of people want to post quick and post from tiny mobile phones. Autocropping actually works really well most of the time in doing a better job then most people in creating far more engaging thumbnails.
posted by geoff. at 9:28 AM on September 21, 2020 [1 favorite]
Because the vast majority of people want to post quick and post from tiny mobile phones. Autocropping actually works really well most of the time in doing a better job then most people in creating far more engaging thumbnails.
posted by geoff. at 9:28 AM on September 21, 2020 [1 favorite]
I've got ponderings about what it's doing to discourse that these various systems are using likely faulty proxies to measure user intent. Clicks-to-zoom may mean the picture is interesting, it also may mean that the thumbnail isn't properly representative.
Similarly, interaction (replies, retweets) may mean the thumbnail is inappropriately inflammatory and not representative of the actual intent of the poster.
And then I expand that to how Twitter (even in alleged "Latest Tweets" mode) and Facebook seem to want to re-hash ancient posts that aren't actually indicative of the overall sense of the poster's account, and think about all that we've given up by letting the social media sites take over from blogs I chose to read or not read.
posted by straw at 11:49 AM on September 21, 2020 [2 favorites]
Similarly, interaction (replies, retweets) may mean the thumbnail is inappropriately inflammatory and not representative of the actual intent of the poster.
And then I expand that to how Twitter (even in alleged "Latest Tweets" mode) and Facebook seem to want to re-hash ancient posts that aren't actually indicative of the overall sense of the poster's account, and think about all that we've given up by letting the social media sites take over from blogs I chose to read or not read.
posted by straw at 11:49 AM on September 21, 2020 [2 favorites]
when this happened I immediately remembered Damien Williams, who posted his presentation slides and audio that he gave at the 21st Conference of the Society for Philosophy and Technology to his blog last year:
“Any Sufficiently Advanced Neglect is Indistinguishable from Malice”
posted by zenwerewolf at 8:50 AM on September 22, 2020 [1 favorite]
“Any Sufficiently Advanced Neglect is Indistinguishable from Malice”
posted by zenwerewolf at 8:50 AM on September 22, 2020 [1 favorite]
had the thought a couple weeks ago: what if it turns out black & brown people are safe from the machine revolution because the murderbots, like the soap dispensers, are unable to see them?
Bring ignored or misidentified by sensors is as deadly as being targeted.
(just to be really clear here I'm not arguing that any of this is good or that we should keep doing it, including the machine revolution)
posted by taquito sunrise at 12:18 PM on September 22, 2020
Bring ignored or misidentified by sensors is as deadly as being targeted.
(just to be really clear here I'm not arguing that any of this is good or that we should keep doing it, including the machine revolution)
posted by taquito sunrise at 12:18 PM on September 22, 2020
« Older Penguin Pandemic | The People Machine Newer »
This thread has been archived and is closed to new comments
Also Zoom
posted by rodlymight at 8:48 AM on September 20, 2020