An antidote for fear

August 2, 2024 3:05 AM

The question of student surveillance is made more difficult by a lack of clear data on whether it works and if so, whether the collateral damage to privacy is justified. School officials across the country defend the use of such surveillance by arguing that if it saves just one life, it’s worth it. But is it worth it if it turns schools into virtual prisons? “Through a careful review of the existing evidence, and through interviews with dozens of school staff, parents and others,” wrote a group of Rand researchers in February, “we found that AI based monitoring, far from being a solution to the persistent and growing problem of youth suicide, might well give rise to more problems than it seeks to solve.” from Spyware turned this Kansas high school into a ‘red zone’ of dystopian surveillance [Kansas Reflector]
posted by chavenet (25 comments total)

Words cannot express my response to this piece. But I’d start with, if this software has prevented 5,000 suicides over 5 years, and the rate is 10,000 a year, has the rate gone done 10%? Or way more in districts where stupid or naive people have actually bought this Orwellian snake oil?

I realize the US is an extremely unusual and violent country, with school shootings, active shooter drills, “comfort kits” for kindergarteners and metal detectors. I realize schools cannot mandate gun control and I know parents of teens fear for their kids - I have two. But at what point do you realize you have made the kids the prisoners?

I have teens. Like most parents, I worry about them, a lot. Sending my kids to school is occasionally terrifying - even here in Toronto my son was in lockdown for a situation last year (a student was reported to be carrying a gun…there was none found.) That’s a long hour and a half. And suicide is a real fear. But you can’t chain your kid down.

Also, how the hell can kids write essays about say Of Mice and Men, or residential schools, or Shakespeare on these school devices?

According to reported Fast Company magazine, Gaggle previously flagged the words “gay” and “lesbian” in assignments and chat messages.

Of course it did.
posted by warriorqueen at 3:45 AM on August 2 [17 favorites]

When I was young and first read "Dune", one of the parts that struck me as crappy worldbuilding was the Butlerian Jihad, where in their distant past the people had risen up and killed off the "thinking machines". Now? I totally get it. And yes, "AI" is not intelligent and does not think, but it should still all be burned to the ground.

Fun fact: at the university where I work, faculty from a bunch of departments in the humanities sat down in the spring semester and sorted out how to ChatGPT-proof our courses: this was actually very easy, and mostly involved kicking it old-school with blue books and shit. Now, however, the upper administration, who only cares about graduation rates, grade inflation be damned, is starting to put the brakes on this, with a profoundly poorly-worded statement about how teaching kids to use ChatGPT was a great skill. It's going over really well.
posted by outgrown_hobnail at 4:36 AM on August 2 [20 favorites]

Nuke it from orbit
posted by Reasonably Everything Happens at 4:54 AM on August 2 [3 favorites]

In my small way, I continue to fight this shit.
posted by humbug at 5:13 AM on August 2 [3 favorites]

But at what point do you realize you have made the kids the prisoners?

Right? This is made all the more ironic for a country that claims to value "freedom."
posted by Kitteh at 5:23 AM on August 2 [5 favorites]

From the article:

“People in authority can violate your rights while believing they are protecting you,” the Budget staff wrote in explaining their coverage. “It’s up to you to protect your work process and product. Adults didn’t tell us to fight the good fight. We did it ourselves.”

From my high school reading:

'You are ruling over us for our own good,' he said feebly. 'You believe that human beings are not fit to govern themselves, and therefore --'

He started and almost cried out. A pang of pain had shot through his body. O'Brien had pushed the lever of the dial up to thirty-five.

'That was stupid, Winston, stupid!' he said. 'You should know better than to say a thing like that.'

He pulled the lever back and continued:

'Now I will tell you the answer to my question. It is this. The Party seeks power entirely for its own sake. We are not interested in the good of others; we are interested solely in power. Not wealth or luxury or long life or happiness: only power, pure power.
posted by AlSweigart at 5:28 AM on August 2 [5 favorites]

Thinking of all the texts I assigned as a high school English teacher, I can think of very few of them that I could have used in a panopticon-ish world like Lawrence High School. I would have been responsible for getting them flagged for alarming literary analysis.
posted by kozad at 5:28 AM on August 2 [4 favorites]

I'll believe this nonsense is effective when they install it on principals' computers.
posted by AlSweigart at 5:29 AM on August 2 [19 favorites]

Words cannot express my response to this piece. But I’d start with, if this software has prevented 5,000 suicides over 5 years, and the rate is 10,000 a year, has the rate gone done 10%? Or way more in districts where stupid or naive people have actually bought this Orwellian snake oil?

From a public health perspective, I guffawed at the description of this company's estimate of their results. They're basically saying "If a student has these flags, then they definitely would have died" which is an absurd proposition. And, as you point out, the volume is ridiculously high.

From a public health perspective, the way to measure this software's outcomes is to identify comparable districts , some with and some without the spybot, and compare them to each other, before and after spybot implementation. Roughly:
- Calculate suicide rates for the population of interest prior to the implementation in all areas ("before")
- Calculate the "expected" rates after the implementation
- Calculate the observed rates
- Calculate the difference in expected vs observed

In the nonspybot places you'd expect expected and observed to be similar; in spybot places you'd look for observed to be lower than expected, with an inflection at spybot implementation. All this would prove challenging because you probably have few enough suicides that it's hard to calculate stable rates, you'd need very large time periods or very large populations of implementation in order to see anything that's not statistical noise.

This, of course, does not do this. It's literally some people in an orwellian AI goldrush doing some back of the napkin math while they lie to school administrators who have their hands on the money faucet.
posted by entropone at 5:40 AM on August 2 [16 favorites]

Sartre was wrong. Hell used to be other people. Now it’s high school.
school is other people?
posted by HearHere at 5:48 AM on August 2 [6 favorites]

Their claim for number of lives saved is exactly as strong as if I were to claim that I have an algorithm for predicting that someone will die based on their horoscope, but also can prevent it with an automated intercessory prayer, and that I have used this algorithm to save 5,790 lives.

In other words, it's bullshit.
posted by biogeo at 6:06 AM on August 2 [10 favorites]

The difference of course being that in my bullshit example, I'm not subjecting people to mass surveillance, targeting minority groups for administrative harassment, or actively interfering with the attempts of students in crisis to seek help. Unlike, it would seem, Gaggle.
posted by biogeo at 6:09 AM on August 2 [5 favorites]

This is made all the more ironic for a country that claims to value "freedom."

Not for kids, it doesn't. American law has historically treated children as chattel until they turn 18 and/or 21, at which point they are completely responsible for everything ever and a failure until proven otherwise. It's why we haven't cooperated with international laws on children's rights; the US traditionally considers the parents' rights to corporal punishment or arranging child marriages to be paramount.

That's why you see this framing of "if it saves just one life." Adults aren't forced into, say, speed limits of 55 mph nationally, even though it would save many lives. But kids are fungible and can be treated as such. It's also why conservatives have no interest in protecting children's First Amendment rights, unless it involves their shitty kids going to school in MAGA shirts or praying at people.
posted by Countess Elena at 7:08 AM on August 2 [20 favorites]

"I'll believe this nonsense is effective when they install it on principals' computers."

It might be effective. That's why they don't install it on the principals' computers.

It's like why low level employees get drug tested and executives don't.

Alternatively, I'll believe this isn't an offense against quality of life when it's installed on the principals' computers.
posted by Nancy Lebovitz at 7:10 AM on August 2 [4 favorites]

Alternatively, I'll believe this isn't an offense against quality of life when it's installed on the principals' computers.

Not just the principals', but each and every member of the board of education, and their family members' PERSONAL computers.
posted by mikelieman at 8:32 AM on August 2 [12 favorites]

Children aren't suicidal because their school isn't spying on them enough. If I was subjected to this as a child it'd drive me deeper into despair, no doubt.
posted by wafehling at 8:47 AM on August 2 [8 favorites]

Company writes checks to local school board. Checks clear. Who cares what the results might be.
Checks cleared. We're done here.
posted by pthomas745 at 9:29 AM on August 2 [3 favorites]

Now, however, the upper administration, who only cares about graduation rates, grade inflation be damned, is starting to put the brakes on this, with a profoundly poorly-worded statement about how teaching kids to use ChatGPT was a great skill. It's going over really well.

Fake it 'til you make everything fake.

I saw school as a prison from day one, and one of the big things I could not stand was a bunch of adults watching everything I did. When I was out of our house, I hardly had anything to do with adults until I went to school.

But after refusing to go to school for a few days sometime during second grade, I realized I had to go there anyway, because that’s where all the other kids were. That was a dark moment.
posted by jamjam at 9:29 AM on August 2 [9 favorites]

The surveilance and control is the goal, protecting kids is either a welcomed bonus or a flimsy coverstory. This ubiquitous prisonification of work and home is happening everywhere. Your owners want your data, need to control you, need to make you more predicatble and channel you to their ends. They have zero interest jn other interventions improve student wellbeing and mental health that don't involve authoritarianism. Take sleep and start times, or access to free dental care etc. No, more cameras, more sensors, more chains.
posted by No Climate - No Food, No Food - No Future. at 9:48 AM on August 2 [7 favorites]

I used to maintain a local school's IT systems for a living. If that school had asked me to deploy shit like this I would have refused after giving them a long, detailed and expletive-laden explanation as to why, and if they'd insisted I would have resigned.

As a parent, if I found out my kid's school had installed this package on my kid's school-provided IT gear, my first response would be to break the security on that gear and uninstall it, after which I would raise hell with the school, and if I or my kid got any pushback about that and my kid wanted to move schools I would absolutely support them in doing so.

Any school administrator who comes away from a sales pitch for this software thinking "yeah, that seems like a good idea" is a complete fucking rube.
posted by flabdablet at 2:00 PM on August 2 [9 favorites]

Where this shit has been decisively stopped, it's been parents and students who did it. See, for example, the InBloom implosion.

I've managed to get to dean level with my concerns. Once. For the most part, the people who implement surveillance shit on campus treat me as an annoying gadfly they can either ignore or try to swat down.

So yeah. I hear your frustration and I share it, flabdablet.
posted by humbug at 5:19 PM on August 2 [3 favorites]

I should note that in Daughter 1's high school this year (it just started yesterday), they've spent thousands of dollars on Yonder Pouches, which are magnetic-locked Faraday cages, and any phone seen out of a pouch will be confiscated for the rest of the year. The locks are on timers, so they won't open until after the end of the school day. Kids aren't allowed to have their phones even at lunch. I've got $100 in the over/under pool for 10 September as to when that gets rescinded.

D1 came to me today and showed me how a little internet research taught her how to hack the pouch with a fishing magnet, and honestly I've never been so proud of the lazy little anarchist.
posted by outgrown_hobnail at 6:28 PM on August 2 [11 favorites]

Apart from being fractally wrong from the start, this shitful package's name is Gaggle. Goose is who?
posted by flabdablet at 10:27 PM on August 2 [4 favorites]

Our schools became prisons a long time ago
posted by Ray Walston, Luck Dragon at 12:33 PM on August 4 [2 favorites]

That's mostly what makes the kids not want to go
posted by flabdablet at 9:47 PM on August 4 [1 favorite]