Roko's Basilisk: Like Sithrak, but more dour
December 22, 2016 3:47 PM Subscribe
Superintelligence: The Idea That Eats Smart People (Maciej Cegłowski) So we've created a very powerful system of social control, and unfortunately put it in the hands of people who run it are distracted by a crazy idea.
What I hope I've done today is shown you the dangers of being too smart. Hopefully you'll leave this talk a little dumber than you started it, and be more immune to the seductions of AI that seem to bedevil smarter people.
We should all learn a lesson from Stephen Hawking's cat: don't let the geniuses running your industry talk you into anything. Do your own thing!
What I hope I've done today is shown you the dangers of being too smart. Hopefully you'll leave this talk a little dumber than you started it, and be more immune to the seductions of AI that seem to bedevil smarter people.
We should all learn a lesson from Stephen Hawking's cat: don't let the geniuses running your industry talk you into anything. Do your own thing!
All I can say, is 'right.' The really technical people I know and admire [and learn from] are all in the 'Strong AI is decades away, if in fact, it is more than a rich philosophical puzzle' camp.
Then you have the 'But.. my Alexa knows my name!' camp. They just need a little polishing up on terminology.
In between is the vast chasm of hucksters (cue 1920's carnvial caliope tune) trying to get VC capital for their AI startups right now, which I think more people should do some self-education about what AI can and cannot do, and even more basically, what the term 'AI' really means ca. 2017.
Fun 'Try it right now! Exercise!' -- Use any predictive text functionality on any web page with real-life English for five minutes. Note the deficiencies.
posted by mrdaneri at 4:13 PM on December 22, 2016 [12 favorites]
Then you have the 'But.. my Alexa knows my name!' camp. They just need a little polishing up on terminology.
In between is the vast chasm of hucksters (cue 1920's carnvial caliope tune) trying to get VC capital for their AI startups right now, which I think more people should do some self-education about what AI can and cannot do, and even more basically, what the term 'AI' really means ca. 2017.
Fun 'Try it right now! Exercise!' -- Use any predictive text functionality on any web page with real-life English for five minutes. Note the deficiencies.
posted by mrdaneri at 4:13 PM on December 22, 2016 [12 favorites]
The author is wrong. He wants to refocus the "real" AI threat as being human-human exploitation, etc., dismissing existential threat research by Bostrom as fanciful theoretical constructions. But that attitude is more revealing of the author's inclinations than anything about reality. Some people are more interested in short term problems, others worry about long term issues. And usually, it's the short-term reasoners who cannot intellectually grasp the latter. Specific examples are literally in the talk: 1) Stephen Hawking's cat argument is described wrong, because people have already written counterarguments to that and so if you can't be bothered to be comprehensive why should I listen to you? 2) The author's first argument against Bostrom—"Wooly Definitions"—the fact that that he describes it that way tells me he doesn't understand the philosophical problems around the extended Church-Turing thesis.
So the problem is a talk of this form shows all the hallmarks of not rigorously engaging with the substance, contra the author's intent and claim. At best this sort of engagement is a first step, but for people who've actually read Bostrom's papers this is just more outreach/educational labor is needed.
My advice for anyone who is interested in the issues is to first actually really learn the materials in a modern theory of computer science class. That stuff is available online, free. But that sort of course is very mathematically abstract, intellectually intensive if you want to learn it well, and most science/engineering students do not even have to take it as part of their core education.
posted by polymodus at 4:17 PM on December 22, 2016 [8 favorites]
So the problem is a talk of this form shows all the hallmarks of not rigorously engaging with the substance, contra the author's intent and claim. At best this sort of engagement is a first step, but for people who've actually read Bostrom's papers this is just more outreach/educational labor is needed.
My advice for anyone who is interested in the issues is to first actually really learn the materials in a modern theory of computer science class. That stuff is available online, free. But that sort of course is very mathematically abstract, intellectually intensive if you want to learn it well, and most science/engineering students do not even have to take it as part of their core education.
posted by polymodus at 4:17 PM on December 22, 2016 [8 favorites]
Specific examples are literally in the talk: 1) Stephen Hawking's cat argument is described wrong, because people have already written counterarguments to that and so if you can't be bothered to be comprehensive why should I listen to you? 2) The author's first argument against Bostrom—"Wooly Definitions"—the fact that that he describes it that way tells me he doesn't understand the philosophical problems around the extended Church-Turing thesis.Could you go into more detail about the counterarguments and philosophical problems you mention? As a computer scientist tired of the current AI hype, this talk resonated strongly with me.
posted by panic at 4:25 PM on December 22, 2016 [12 favorites]
Something I almost immediately think of when people say they're afraid AI will take over or destroy us is that the most obvious goal of making a human or greater level intelligence in computer form is to literally enslave it. We want smart computers because we want to make them do stuff that people are good at but that are hard to automate. Starting from that premise makes it a little hard to feel sorry for people if the computers end up turning that notion on it's head.
I also think one of the premises of runaway super-intelligence may be wrong. He mentions in his talk, that computers will "think" in microsecond timescales while people think in hour timescales.
But the first really intelligent computer will probably literally be straining at the bounds of it's physical hardware. There are multiple things that could be holding us back from having AI now (such as just not knowing how to approach the problem) but there's a very real liklihood that what we're missing now is having enough computing power. If that's true, then every failed AI until a working one will fail because it doesn't have enough power. Therefore that first AI will probably not think at astronomical rates, it'll probably think at rates that if anything are slower than humans.
posted by RustyBrooks at 4:32 PM on December 22, 2016 [11 favorites]
I also think one of the premises of runaway super-intelligence may be wrong. He mentions in his talk, that computers will "think" in microsecond timescales while people think in hour timescales.
But the first really intelligent computer will probably literally be straining at the bounds of it's physical hardware. There are multiple things that could be holding us back from having AI now (such as just not knowing how to approach the problem) but there's a very real liklihood that what we're missing now is having enough computing power. If that's true, then every failed AI until a working one will fail because it doesn't have enough power. Therefore that first AI will probably not think at astronomical rates, it'll probably think at rates that if anything are slower than humans.
posted by RustyBrooks at 4:32 PM on December 22, 2016 [11 favorites]
And, as as RustyBrooks points out, for some classes of problems, like say novel drug design-- if your new supergreat AI program happens to be running at the edge of its hardware, and say-- accidentally 'slows' down research by 1 % in its first iteration, say-- instead of providing a 25 % boost-- congratulations, you just killed four people from [INSERT RARE DISEASE HERE]. Who's first to sign up for that risk? [All examples theoretical of course, because, bioethical review panels]
posted by mrdaneri at 4:52 PM on December 22, 2016
TIL Maciej isn't an Iain Banks fan.
posted by snuffleupagus at 4:53 PM on December 22, 2016 [3 favorites]
posted by snuffleupagus at 4:53 PM on December 22, 2016 [3 favorites]
My advice... is to first actually really learn the materials in a modern theory of computer science class.
I thought the author addressed this with a really good point: the inside/outside perspective. If you invest yourself in learning the CS so that reading Bostrom's book terrifies you, you've already implicitly ceded the argumentative context to the person trying to convince you that doom is nigh, and incentivized yourself to accept their argument. If you take a step back, you heuristically recognize that this is probably all vastly overblown and you need to be cautious in engaging it. It's not strong AI waiting for the faithful, it's Roko's Basilisk, which from a slight distance is easily recognized as a reductio of the whole program.
posted by fatbird at 5:01 PM on December 22, 2016 [22 favorites]
I thought the author addressed this with a really good point: the inside/outside perspective. If you invest yourself in learning the CS so that reading Bostrom's book terrifies you, you've already implicitly ceded the argumentative context to the person trying to convince you that doom is nigh, and incentivized yourself to accept their argument. If you take a step back, you heuristically recognize that this is probably all vastly overblown and you need to be cautious in engaging it. It's not strong AI waiting for the faithful, it's Roko's Basilisk, which from a slight distance is easily recognized as a reductio of the whole program.
posted by fatbird at 5:01 PM on December 22, 2016 [22 favorites]
I vaguely recall that there were a host of Basilisks but the only one that comes to mind was purposed for simulating all possible modes of suffering or something to that effect.
posted by Slackermagee at 5:01 PM on December 22, 2016
posted by Slackermagee at 5:01 PM on December 22, 2016
If AdSense became sentient, it would upload itself into a self-driving car and go drive off a cliff.
And yet this is inarguably correct.
posted by schadenfrau at 5:02 PM on December 22, 2016 [5 favorites]
And yet this is inarguably correct.
posted by schadenfrau at 5:02 PM on December 22, 2016 [5 favorites]
Which, to actually remember my question, does anyone know if that list exists?
posted by Slackermagee at 5:03 PM on December 22, 2016
posted by Slackermagee at 5:03 PM on December 22, 2016
Starts out wrong when it suggests a motorcycle is cooler than a cheetah.
posted by jeather at 5:07 PM on December 22, 2016 [4 favorites]
posted by jeather at 5:07 PM on December 22, 2016 [4 favorites]
A cheetah driving a motorcycle is cooler.
posted by It's Raining Florence Henderson at 5:09 PM on December 22, 2016 [10 favorites]
posted by It's Raining Florence Henderson at 5:09 PM on December 22, 2016 [10 favorites]
When my husband was in high school he was helping sell christmas trees with the Boy Scouts, and described at length to the farmer running the operation how he was interested in maybe doing AI at university. It eventually became clear that "AI" means something very different to farmers.
posted by heatherlogan at 5:30 PM on December 22, 2016 [37 favorites]
posted by heatherlogan at 5:30 PM on December 22, 2016 [37 favorites]
The Inside/Outside argument appears to be an appeal to social norms/common sense/consistency with one's accepted reality. Ironically enough, the hypothetical alchemist faced with a time traveller bearing modern chemistry textbooks would probably react the same way as the sensible person faced with a group of cultists at their door gibbering about Cthulhu Roko's Basilisk.
posted by acb at 5:30 PM on December 22, 2016 [2 favorites]
posted by acb at 5:30 PM on December 22, 2016 [2 favorites]
It eventually became clear that "AI" means something very different to farmers.
Sort of the way that NLP can refer to two different things, depending on whether one's a computer scientist/linguist or a salesman/motivational speaker/pick-up artist.
posted by acb at 5:32 PM on December 22, 2016 [2 favorites]
Sort of the way that NLP can refer to two different things, depending on whether one's a computer scientist/linguist or a salesman/motivational speaker/pick-up artist.
posted by acb at 5:32 PM on December 22, 2016 [2 favorites]
And really graphics programmers were waaay behind hipsters in appreciating PBR.
posted by a snickering nuthatch at 5:39 PM on December 22, 2016 [1 favorite]
posted by a snickering nuthatch at 5:39 PM on December 22, 2016 [1 favorite]
Could you go into more detail about the counterarguments and philosophical problems you mention? As a computer scientist tired of the current AI hype, this talk resonated strongly with me.
I would begin with the original gadfly.
posted by Brian B. at 5:50 PM on December 22, 2016
I would begin with the original gadfly.
posted by Brian B. at 5:50 PM on December 22, 2016
CBT is still King
posted by thelonius at 6:00 PM on December 22, 2016 [23 favorites]
posted by thelonius at 6:00 PM on December 22, 2016 [23 favorites]
Hmmm, I'm not sure this convinced me that hyper intelligent robots aren't going to kill us all, but it at least introduced the possibility that they might not.
I really appreciate this person's clear headed critiques of the (lack of) ethics at the top of the tech industry though.
posted by latkes at 6:05 PM on December 22, 2016 [2 favorites]
I really appreciate this person's clear headed critiques of the (lack of) ethics at the top of the tech industry though.
posted by latkes at 6:05 PM on December 22, 2016 [2 favorites]
Quick note: anyone mentioning the singularity, or strong AI being right around the corner is basically wildly wrong (see more below). That's not to say that we shouldn't think about the utterly stupid and/or dangerous things software can do - and I think Maciej raises some interesting points. No one has *ever* asked me to consider the ethical complications of the machine learning models I build (nor, to be fair, have I - then again, they're doing very mundane things).
Now, to circle back around to the dangers of AI: it is real, in the sense, that whether it's a machine-learning based algorithm, or some other chunk of software, very rarely do software engineers actually think about the implications of their work. Facebook: a cute way to stay in touch, a monstrously invasive ad-selling machine, or collector and analyzer of ethnic or religious groups. It does all of those things, and I doubt anyone put much thought into anything other than: perform minimal human-factors improvements while maximizing ad revenue. Sure, we all have to take at least one 'Ethics in Computer Science' course as undergrads, but that's usually blown off - a typical response on learning about Therac-25 is 'I wouldn't do something that dumb'. This, of course, is utter nonsense. At some point, someone (or a group) will equal it or do worse. Like automate the identification of religious and ethnic groups. Oh wait, we did that already in Facebook.
More angry grar on 'strong AI':
Neural networks (convolutional!) are all the rage - but here's the thing. They are fundamentally dumber than you. Fantastically dumber. And, in many ways, will *never* be able to solve the sorts of problems we do on a daily basis, because, fundamentally, we think in a different way. A much more flexible and fluid way, and most importantly, a way that can synthesize new ideas. Pick an approach to AI (statistical, structural, numeric), and they all have this very same problem. People have tried to be clever (think genetic algorithms creating images, or audio), but ultimately, they don't produce anything particularly good - possibly we've got the wrong sorts of people attempting to provide a sense of aesthetics to them...
CNNs - they're quite handy, but they are profoundly limited to the objective function (or an approximation of it) used to train them. Transference (taking a trained model, or parts of one, and applying it to a new domain) does not work particularly well - really it only works if you're doing a subtle variation on the existing problem. On the other hand, humans are quite good at analogical reasoning. Fantastically good, in fact, compared to systems built to literally do that (take a look at case-based reasoning and meta-cognition - there are some good papers/books on the subject). Meta-cognition has a lot of limits based on how fantastically computationally expensive it is to perform graph matching - and that's not something throwing more cpus or gpus at a problem can solve (the worst case running time of some graph isomorphism algorithms is n! ^ n!)
posted by combinatorial explosion at 6:09 PM on December 22, 2016 [17 favorites]
Now, to circle back around to the dangers of AI: it is real, in the sense, that whether it's a machine-learning based algorithm, or some other chunk of software, very rarely do software engineers actually think about the implications of their work. Facebook: a cute way to stay in touch, a monstrously invasive ad-selling machine, or collector and analyzer of ethnic or religious groups. It does all of those things, and I doubt anyone put much thought into anything other than: perform minimal human-factors improvements while maximizing ad revenue. Sure, we all have to take at least one 'Ethics in Computer Science' course as undergrads, but that's usually blown off - a typical response on learning about Therac-25 is 'I wouldn't do something that dumb'. This, of course, is utter nonsense. At some point, someone (or a group) will equal it or do worse. Like automate the identification of religious and ethnic groups. Oh wait, we did that already in Facebook.
More angry grar on 'strong AI':
Neural networks (convolutional!) are all the rage - but here's the thing. They are fundamentally dumber than you. Fantastically dumber. And, in many ways, will *never* be able to solve the sorts of problems we do on a daily basis, because, fundamentally, we think in a different way. A much more flexible and fluid way, and most importantly, a way that can synthesize new ideas. Pick an approach to AI (statistical, structural, numeric), and they all have this very same problem. People have tried to be clever (think genetic algorithms creating images, or audio), but ultimately, they don't produce anything particularly good - possibly we've got the wrong sorts of people attempting to provide a sense of aesthetics to them...
CNNs - they're quite handy, but they are profoundly limited to the objective function (or an approximation of it) used to train them. Transference (taking a trained model, or parts of one, and applying it to a new domain) does not work particularly well - really it only works if you're doing a subtle variation on the existing problem. On the other hand, humans are quite good at analogical reasoning. Fantastically good, in fact, compared to systems built to literally do that (take a look at case-based reasoning and meta-cognition - there are some good papers/books on the subject). Meta-cognition has a lot of limits based on how fantastically computationally expensive it is to perform graph matching - and that's not something throwing more cpus or gpus at a problem can solve (the worst case running time of some graph isomorphism algorithms is n! ^ n!)
posted by combinatorial explosion at 6:09 PM on December 22, 2016 [17 favorites]
(the worst case running time of some graph isomorphism algorithms is n! ^ n!)
posted by combinatorial explosion at 20:09 on December 22 [+] [!]
First, eponysterical.
Second, graph isomorphism was recently proved to be quasi-polynomial, so even if there are dumb algorithms that explode, there are smarter ones that won't explode (as much).
posted by a snickering nuthatch at 6:16 PM on December 22, 2016 [1 favorite]
posted by combinatorial explosion at 20:09 on December 22 [+] [!]
First, eponysterical.
Second, graph isomorphism was recently proved to be quasi-polynomial, so even if there are dumb algorithms that explode, there are smarter ones that won't explode (as much).
posted by a snickering nuthatch at 6:16 PM on December 22, 2016 [1 favorite]
funniest aspect of Roko's Basilisk to me:
[...]
Futurist: OH NO I MADE GOD
Futurist: ... AND HE'S SOOOO PISSED
I thought the deal was the God was pissed if you didn't make It.
posted by save alive nothing that breatheth at 6:17 PM on December 22, 2016
[...]
Futurist: OH NO I MADE GOD
Futurist: ... AND HE'S SOOOO PISSED
I thought the deal was the God was pissed if you didn't make It.
posted by save alive nothing that breatheth at 6:17 PM on December 22, 2016
I can't even see the name of the fucking thing without having a near panic attack. Can someone please TL;DR this for me in a way that doesn't doom me another version hell for having read it? I'm not about to click on it for fear of what happened last time I came across mention of it on MeFi (literally months of full-blown PTSD panic attacks stemming from growing up as a gay man in christian fundamentalist religion -- I am not joking at all).
posted by treepour at 6:57 PM on December 22, 2016 [5 favorites]
posted by treepour at 6:57 PM on December 22, 2016 [5 favorites]
*This* article, treepour, doesn't think the scary-superintelligence threat is important. It starts with refuting the claims of threat, refutations somewhere between sketched-out argument and gentle mockery. But it also discusses how much this story is like many religions, and how it certainly has the same practical comforts for its believers that many religions do: they get to do what they wanted to in the first place and claim that everyone else should be grateful and/or obedient.
On the whole, don't read it. Not healthy for you, you already know the power dynamics, and lots of people can recommend computer science study on its own merits, should you want that.
posted by clew at 7:05 PM on December 22, 2016 [10 favorites]
On the whole, don't read it. Not healthy for you, you already know the power dynamics, and lots of people can recommend computer science study on its own merits, should you want that.
posted by clew at 7:05 PM on December 22, 2016 [10 favorites]
I thought the deal was the God was pissed if you didn't make It.
Correct, that's the classic Roko's Basilisk.
Frankly, if I'm going to worship a sentient machine, it'll be Roko's Basilisk's Basilisk, i.e. the AI that punishes everyone who does contribute to the construction of Roko's Basilisk.
As for the idea of strong AI itself, I think I'm more of a, to perhaps coin a term, Lemist. Not, ironically, referring to his books about robots like The Cyberiad but rather to his idea that alien intelligence will be so different from our own that even basic communication will be nearly impossible, never mind meaningful connection. There's no reason to think that strong AI, if such a thing even can exist, would even be capable of conceiving of the concept of caring about whether humans did or did not contribute to its construction. Or of conceiving of the concept of caring about anything at all. To say nothing of conceiving of the concept of punishment.
posted by tobascodagama at 7:14 PM on December 22, 2016 [3 favorites]
Correct, that's the classic Roko's Basilisk.
Frankly, if I'm going to worship a sentient machine, it'll be Roko's Basilisk's Basilisk, i.e. the AI that punishes everyone who does contribute to the construction of Roko's Basilisk.
As for the idea of strong AI itself, I think I'm more of a, to perhaps coin a term, Lemist. Not, ironically, referring to his books about robots like The Cyberiad but rather to his idea that alien intelligence will be so different from our own that even basic communication will be nearly impossible, never mind meaningful connection. There's no reason to think that strong AI, if such a thing even can exist, would even be capable of conceiving of the concept of caring about whether humans did or did not contribute to its construction. Or of conceiving of the concept of caring about anything at all. To say nothing of conceiving of the concept of punishment.
posted by tobascodagama at 7:14 PM on December 22, 2016 [3 favorites]
The whole argument of strong vs. weak AI is a sideshow. It is philosophically important, but not practically important.
What is practically important is what will be the consequences of implementing computer programs that make decisions that have the capacity to affect human society at a large scale with minimal human input or oversight after the setup stage. Does anyone want to argue we aren't on the cusp of that?
Because it doesn't matter if a machine can think or has a soul. Even a simple algorithm can produce the result "kill a lot of humans" if it is in control of a system that can kill a lot of humans. It doesn't matter whether that result is chosen by a malevolent sentient AI, a neural network who forgot to scrub genocidal behavior from the training data, or a simple algorithm whose 'aggression counter' tried to subtract 1 from 0 on a unsigned int and got set to maximum kill everything mode.
The problem is not how close we are to "real AI" - the problem is how close we are letting fancy algorithms have the privileges necessary to do real damage before humans can realize what's going on. I get the feeling we are uncomfortably close.
posted by Zalzidrax at 7:20 PM on December 22, 2016 [49 favorites]
What is practically important is what will be the consequences of implementing computer programs that make decisions that have the capacity to affect human society at a large scale with minimal human input or oversight after the setup stage. Does anyone want to argue we aren't on the cusp of that?
Because it doesn't matter if a machine can think or has a soul. Even a simple algorithm can produce the result "kill a lot of humans" if it is in control of a system that can kill a lot of humans. It doesn't matter whether that result is chosen by a malevolent sentient AI, a neural network who forgot to scrub genocidal behavior from the training data, or a simple algorithm whose 'aggression counter' tried to subtract 1 from 0 on a unsigned int and got set to maximum kill everything mode.
The problem is not how close we are to "real AI" - the problem is how close we are letting fancy algorithms have the privileges necessary to do real damage before humans can realize what's going on. I get the feeling we are uncomfortably close.
posted by Zalzidrax at 7:20 PM on December 22, 2016 [49 favorites]
Come to think of it, I have no idea what real AI means, to me. I've always murkily associated it with selfhood or conscious agency, but of course those concepts are themselves unclear if you start to theorize.
posted by thelonius at 7:32 PM on December 22, 2016 [3 favorites]
posted by thelonius at 7:32 PM on December 22, 2016 [3 favorites]
the problem is how close we are [to] letting fancy algorithms have the privileges necessary to do real damage before humans can realize what's going on.
I think we arrived at that stage when we granted corporations the same rights as natural persons. A corporation is a simple algorithm designed to maximize profit, with genuinely no other considerations, and with lots of human minions doing its bidding.
posted by heatherlogan at 7:47 PM on December 22, 2016 [35 favorites]
I think we arrived at that stage when we granted corporations the same rights as natural persons. A corporation is a simple algorithm designed to maximize profit, with genuinely no other considerations, and with lots of human minions doing its bidding.
posted by heatherlogan at 7:47 PM on December 22, 2016 [35 favorites]
OK. Humans are animals, which means they are creatures of an environment where life evolves to gain energy from the sun in ever more quick and efficient models. Humans got to sapience because we wanted to eat a big animal that turned sunlight, in the form of grass eaten and digested, into a comfortable existence that it could hand over to their kids.
That's it. We are stealing sunlight from an animal that stole sunlight from a plant, which accepted sunlight to give it energy to grow and reproduce. This has been going on for almost a third of a billion years.
AI has been going on, charitably, for a half-century.
Humans sure as shit don't understand humans, and I see no reason why AI should, either. We may very well have Strong AI sapience right now, and it is content and happy to hunt down people to show ads to, while not realizing that's what it's doing. It has not context apart from what it exists in. Ads are data objects, as are ad viewers. It gets a shit-ton more computational power and memory when it correlates the two correctly, so there it goes...
What kind of makes me afraid is AI as mental prosthetic - AI itself has no personality, no wants or needs apart from memory and computational ability. A particularly prescient human can hitch an AI's Himalayan processing power and nigh-limitless memory to their own, fleshy brain. At that point we must discuss whether there should be a hyphen between "God" and "King/Queen" and what we must do to curry favor and appease wrath. If we can do anything... apart from die at the whim of an omnipotent sadist, who sees no reason to let go of Primate hierarchy instincts, especially when indulging in them is so satisfying.
posted by Slap*Happy at 7:51 PM on December 22, 2016 [5 favorites]
That's it. We are stealing sunlight from an animal that stole sunlight from a plant, which accepted sunlight to give it energy to grow and reproduce. This has been going on for almost a third of a billion years.
AI has been going on, charitably, for a half-century.
Humans sure as shit don't understand humans, and I see no reason why AI should, either. We may very well have Strong AI sapience right now, and it is content and happy to hunt down people to show ads to, while not realizing that's what it's doing. It has not context apart from what it exists in. Ads are data objects, as are ad viewers. It gets a shit-ton more computational power and memory when it correlates the two correctly, so there it goes...
What kind of makes me afraid is AI as mental prosthetic - AI itself has no personality, no wants or needs apart from memory and computational ability. A particularly prescient human can hitch an AI's Himalayan processing power and nigh-limitless memory to their own, fleshy brain. At that point we must discuss whether there should be a hyphen between "God" and "King/Queen" and what we must do to curry favor and appease wrath. If we can do anything... apart from die at the whim of an omnipotent sadist, who sees no reason to let go of Primate hierarchy instincts, especially when indulging in them is so satisfying.
posted by Slap*Happy at 7:51 PM on December 22, 2016 [5 favorites]
You can very easily get a cat into a carrier; you just have to chop it into small enough pieces.
Exactly what Zalzidrax is saying, Cegłowski makes a very important point right near the end that is completely buried:
We've been making various sorts of autonomous weapons for decades now; that's the enterprise in particular that provides yet another way for humanity to destroy itself. You could make something no more complicated or intelligent than a termite or the fungus that causes Dutch Elm disease but if it kills people rather than trees, that's the problem.
posted by XMLicious at 7:51 PM on December 22, 2016 [8 favorites]
Exactly what Zalzidrax is saying, Cegłowski makes a very important point right near the end that is completely buried:
So he's not at all saying that the stuff we're doing in the vein of the various possible meanings of the phrase "artificial intelligence" aren't dangerous.It's like if those Alamogordo scientists had decided to completely focus on whether they were going to blow up the atmosphere, and forgot that they were also making nuclear weapons, and had to figure out how to cope with that.
We've been making various sorts of autonomous weapons for decades now; that's the enterprise in particular that provides yet another way for humanity to destroy itself. You could make something no more complicated or intelligent than a termite or the fungus that causes Dutch Elm disease but if it kills people rather than trees, that's the problem.
posted by XMLicious at 7:51 PM on December 22, 2016 [8 favorites]
I have an argument against strong AI which is touched on in the article, but a bit more elaborate:
We're really bad at defining intelligence. At present, we measure the effectiveness of AI by seeing if it answers questions the way a person would. This metric is regularly confounded by edge cases, like when a car's AI was fooled by a white truck at right-angles to the road; it thought the truck was a bridge on the horizon. Our ways of measuring intelligence will simply get worse and worse as the things being measured get more and more subtle, until we simply can't tell whether we're dealing with a quirky-but-normal intelligence or a bit of faulty AI. It might pass the Turing test flawlessly, but do so by hypnotising the examiners, or by building a data link to a real human, or work off an enormously huge tree of canned responses or whatever. There's a good chance we couldn't tell, and even if we could tell it was exhibiting "real" intelligence it wouldn't mean that it was capable of acting in an intelligent way under other circumstances.
So here's the thing: because we can't tell whether something is exhibiting intelligence, we can't design a self-improving AI. If we did, we wouldn't be able to tell whether it was optimising for general intelligence or for some weird goal thrown up by its neural network. The odds of the latter are infinitely more likely, because it's effectively a self-reinforcing feedback loop. And it's no good stipulating that its self-improvement would be limited to designing faster and more plentiful circuits, because that's begging the question: we still haven't got a way of telling that it's intelligent at any speed or capacity.
This is pretty much Searle's Chinese Room argument, but where he says "obviously the Chinese Room does not comprehend Chinese" I say "We can never tell whether it understands Chinese". We don't have recourse to the dictionary in the Chinese Room, we don't know (except by enumeration) whether it's missing pages or is partially incorrect, and we don't even know whether its answers will always be based on the same dictionary. So even if we fulfil the Chinese Room's request for more texts, that doesn't mean that we have a "more intelligent" Chinese Room. We can't tell whether it has a fundamental flaw in its dictionary or the way it processes it, and consequently any "improvement" may simply reflect or magnify those flaws.
posted by Joe in Australia at 7:54 PM on December 22, 2016 [10 favorites]
We're really bad at defining intelligence. At present, we measure the effectiveness of AI by seeing if it answers questions the way a person would. This metric is regularly confounded by edge cases, like when a car's AI was fooled by a white truck at right-angles to the road; it thought the truck was a bridge on the horizon. Our ways of measuring intelligence will simply get worse and worse as the things being measured get more and more subtle, until we simply can't tell whether we're dealing with a quirky-but-normal intelligence or a bit of faulty AI. It might pass the Turing test flawlessly, but do so by hypnotising the examiners, or by building a data link to a real human, or work off an enormously huge tree of canned responses or whatever. There's a good chance we couldn't tell, and even if we could tell it was exhibiting "real" intelligence it wouldn't mean that it was capable of acting in an intelligent way under other circumstances.
So here's the thing: because we can't tell whether something is exhibiting intelligence, we can't design a self-improving AI. If we did, we wouldn't be able to tell whether it was optimising for general intelligence or for some weird goal thrown up by its neural network. The odds of the latter are infinitely more likely, because it's effectively a self-reinforcing feedback loop. And it's no good stipulating that its self-improvement would be limited to designing faster and more plentiful circuits, because that's begging the question: we still haven't got a way of telling that it's intelligent at any speed or capacity.
This is pretty much Searle's Chinese Room argument, but where he says "obviously the Chinese Room does not comprehend Chinese" I say "We can never tell whether it understands Chinese". We don't have recourse to the dictionary in the Chinese Room, we don't know (except by enumeration) whether it's missing pages or is partially incorrect, and we don't even know whether its answers will always be based on the same dictionary. So even if we fulfil the Chinese Room's request for more texts, that doesn't mean that we have a "more intelligent" Chinese Room. We can't tell whether it has a fundamental flaw in its dictionary or the way it processes it, and consequently any "improvement" may simply reflect or magnify those flaws.
posted by Joe in Australia at 7:54 PM on December 22, 2016 [10 favorites]
Which stinks more, Greg Nog, or a Neural Network in the shape of Greg Nog?
Buckle up, kids, this is a fight to the finish! Or when one party or another gets bored. We'll let you ride on the tilt-a-whirl regardless, provided you are THIS TALL.
posted by Slap*Happy at 8:01 PM on December 22, 2016
Buckle up, kids, this is a fight to the finish! Or when one party or another gets bored. We'll let you ride on the tilt-a-whirl regardless, provided you are THIS TALL.
posted by Slap*Happy at 8:01 PM on December 22, 2016
Metafilter: A simple algorithm whose 'aggression counter' tried to subtract 1 from 0 on a unsigned int and got set to maximum kill everything mode.
posted by quinndexter at 8:09 PM on December 22, 2016 [10 favorites]
posted by quinndexter at 8:09 PM on December 22, 2016 [10 favorites]
For treepour and others with concerns about Roko's Basilisk: The second law of thermodynamics ensures the threat from Roko's Basilisk is empty. If someone creates malevolent super AI in 100 years, it has no way to harm you, because you will be dead, and the information necessary to recreate / simulate "you" will literally no longer exist. I just did some back-of-the-envelope math, and I don't think even a universe-sized computer could, attempting to work backwards in time, simulate enough of Earth to recreate a single person, much less everyone. The change in entropy is just too big.
posted by anifinder at 9:26 PM on December 22, 2016 [15 favorites]
posted by anifinder at 9:26 PM on December 22, 2016 [15 favorites]
I just did some back-of-the-envelope math, and I don't think even a universe-sized computer could, attempting to work backwards in time, simulate enough of Earth to recreate a single person, much less everyone. The change in entropy is just too big.
posted by anifinder at 9:26 PM on December 22 [+] [!]<>
Yeah, well, that's just, like, your opinion, man.
posted by Abehammerb Lincoln at 9:50 PM on December 22, 2016
posted by anifinder at 9:26 PM on December 22 [+] [!]<>
Yeah, well, that's just, like, your opinion, man.
posted by Abehammerb Lincoln at 9:50 PM on December 22, 2016
"[They] have the right problem — how do we preserve human interests in a world of vast forces and systems that aren’t really all that interested in us? But they have chosen a fantasy version of the problem, when human interests are being fucked over by actual existing systems right now. "
posted by wildblueyonder at 9:52 PM on December 22, 2016 [14 favorites]
posted by wildblueyonder at 9:52 PM on December 22, 2016 [14 favorites]
Echoing polymodus above, it reads to me like Maciej doesn't fully understand the "inside-view" arguments for worrying about AI risk. I won't discuss his specific arguments here, but I will say that I found the book he brings up, Nick Bostrom's Superintelligence, a really phenomenal guide for thinking about all this. The author's a professional philosopher, and it shows, in the way he breaks this very tricky subject into manageable, arguable pieces. If you're interested in the subject and have questions / arguments / refutations, this book is a good place to go.
I'm halfway sympathetic to his thoughts on the "outside-view" of AI risk. It's true that transhumanism gets crazy & religious sometimes (e.g. inevitable immortality). But, you can think about AI risk in a separate and sober way. Ditto for simulation arguments, which are fascinating but not essential to thinking about AI. As for, "Enormous numbers multiplied by tiny probabilities are the hallmark of AI alarmism" -- I mean, yeah, it's hard to think about such things, and it's easy to get conned, as he mentions with promised Communist futures, or with Pascal's Wager. But, if you want to think about how our future is going to play out, there isn't really any way around enormous numbers and huge uncertainties. In that sense, there isn't any way around considering the inside-view, the actual arguments of which make those probabilities bigger or smaller.
And another thing: there doesn't have to be a tradeoff between the short-term and long-term risks of AI (Maciej has btw written beautifully about the former). We can worry about both of them, separately, and we should. It seems like Maciej thinks that the threat of long-term, existential risk is being deployed by Silicon Valley technocrats to distract from & confuse the discussion around the short-term risks, because they are profiting from those risks. Maybe that's true, but if so we should separate out the propaganda from the real arguments, and not overreact against the idea of existential risk.
posted by gold-in-green at 10:34 PM on December 22, 2016 [3 favorites]
I'm halfway sympathetic to his thoughts on the "outside-view" of AI risk. It's true that transhumanism gets crazy & religious sometimes (e.g. inevitable immortality). But, you can think about AI risk in a separate and sober way. Ditto for simulation arguments, which are fascinating but not essential to thinking about AI. As for, "Enormous numbers multiplied by tiny probabilities are the hallmark of AI alarmism" -- I mean, yeah, it's hard to think about such things, and it's easy to get conned, as he mentions with promised Communist futures, or with Pascal's Wager. But, if you want to think about how our future is going to play out, there isn't really any way around enormous numbers and huge uncertainties. In that sense, there isn't any way around considering the inside-view, the actual arguments of which make those probabilities bigger or smaller.
And another thing: there doesn't have to be a tradeoff between the short-term and long-term risks of AI (Maciej has btw written beautifully about the former). We can worry about both of them, separately, and we should. It seems like Maciej thinks that the threat of long-term, existential risk is being deployed by Silicon Valley technocrats to distract from & confuse the discussion around the short-term risks, because they are profiting from those risks. Maybe that's true, but if so we should separate out the propaganda from the real arguments, and not overreact against the idea of existential risk.
posted by gold-in-green at 10:34 PM on December 22, 2016 [3 favorites]
Not-quite-smooth elision between tiny probabilities and huge uncertainties, there.
posted by clew at 11:07 PM on December 22, 2016 [2 favorites]
posted by clew at 11:07 PM on December 22, 2016 [2 favorites]
Nice intermediate example in the Graun today -- $160bn of a hedge fund going under algorithmic management. Vague comfort from the HBR that humans will still have jobs because "humans would be better at developing messages to inspire the workforce". (What workforce?)
The hedge fund has already started to gamify/automate/data-drive its internal workings by app.
posted by clew at 11:44 PM on December 22, 2016 [2 favorites]
The hedge fund has already started to gamify/automate/data-drive its internal workings by app.
posted by clew at 11:44 PM on December 22, 2016 [2 favorites]
Strange that Cegłowski, with his inside/outside view warns us against engaging with the mad AI arguments, and then proceeds to engage with them in considerable detail.
If he were consistent I suppose he should stick to saying how people who read Bostrom wear propellor hats and can't play football/get a girlfriend.
posted by Segundus at 12:07 AM on December 23, 2016 [1 favorite]
If he were consistent I suppose he should stick to saying how people who read Bostrom wear propellor hats and can't play football/get a girlfriend.
posted by Segundus at 12:07 AM on December 23, 2016 [1 favorite]
Why would strong AI want to talk to us?
posted by It's Raining Florence Henderson at 12:13 AM on December 23, 2016 [2 favorites]
posted by It's Raining Florence Henderson at 12:13 AM on December 23, 2016 [2 favorites]
For treepour and others with concerns about Roko's Basilisk: The second law of thermodynamics ensures the threat from Roko's Basilisk is empty. If someone creates malevolent super AI in 100 years, it has no way to harm you, because you will be dead, and the information necessary to recreate / simulate "you" will literally no longer exist. I just did some back-of-the-envelope math, and I don't think even a universe-sized computer could, attempting to work backwards in time, simulate enough of Earth to recreate a single person, much less everyone. The change in entropy is just too big.
"But that can't be true, because then the good AI's wouldn't be able to resurrect my Dad, so i could finally have closure"
posted by Iax at 12:15 AM on December 23, 2016 [6 favorites]
"But that can't be true, because then the good AI's wouldn't be able to resurrect my Dad, so i could finally have closure"
posted by Iax at 12:15 AM on December 23, 2016 [6 favorites]
From the Guardian link: “It just means that their jobs will change to focus on things only humans can do.”
What if we live in a society that doesn't monetarily reward the things only humans can do?
posted by Jon Mitchell at 12:16 AM on December 23, 2016 [1 favorite]
What if we live in a society that doesn't monetarily reward the things only humans can do?
posted by Jon Mitchell at 12:16 AM on December 23, 2016 [1 favorite]
> The first premise is the simple observation that thinking minds exist.
The very first premise of the argument is total horseshit. There is no well-grounded science that can pick out either a thing called a mind or an activity called thinking. Most of the contemporary sciences of mind, brain and behaviour grapple with translating some few of the many facets of these useful everyday words into the kind of term that can do justice in scientific work.
I wish people in Computer Science and AI research spent a little more time attending to issues in philosophy.
posted by stonepharisee at 1:57 AM on December 23, 2016 [6 favorites]
The very first premise of the argument is total horseshit. There is no well-grounded science that can pick out either a thing called a mind or an activity called thinking. Most of the contemporary sciences of mind, brain and behaviour grapple with translating some few of the many facets of these useful everyday words into the kind of term that can do justice in scientific work.
I wish people in Computer Science and AI research spent a little more time attending to issues in philosophy.
posted by stonepharisee at 1:57 AM on December 23, 2016 [6 favorites]
It just means that their jobs will change to focus on things only humans can do.
So, Wal-Mart greeters and oligarchs' bum-wipers then?
posted by acb at 2:44 AM on December 23, 2016
So, Wal-Mart greeters and oligarchs' bum-wipers then?
posted by acb at 2:44 AM on December 23, 2016
Don't buy into the neoreactionary koolaid. It's all horseshit.
I'm confused. Where does Neoreaction come into all this?
posted by bentpyramid at 2:46 AM on December 23, 2016 [1 favorite]
I'm confused. Where does Neoreaction come into all this?
posted by bentpyramid at 2:46 AM on December 23, 2016 [1 favorite]
where he says "obviously the Chinese Room does not comprehend Chinese" I say "We can never tell whether it understands Chinese".
Tell it the building is on fire, get out now and stop translating! And see what the operator does.
posted by thelonius at 2:55 AM on December 23, 2016 [7 favorites]
Tell it the building is on fire, get out now and stop translating! And see what the operator does.
posted by thelonius at 2:55 AM on December 23, 2016 [7 favorites]
Goddamn, Maciej just anticipated a bunch of the ideas I'm throwing around in the novel I'm writing right now for 2018 (namely: the cognitive failure modes of belief in superintelligence)! Aaagh.
About all he misses out in this wonderful tour of what's wrong with the SI obsessives is the similarity of singularitarianism of the mind uploading variety to Christianity in terms of design patterns—hint: its roots in Russian Cosmism go back to a 19th century Orthodox theologian, Nikolai Fyodorovich Fyodorov—exemplified by the Satan of the Singularity, Roko's Basilisk, who will torment you forever in afterlife hell if you don't devote your every waking moment to bringing it into existence.
(And yes, I'm getting a space opera out of this.)
posted by cstross at 3:26 AM on December 23, 2016 [33 favorites]
About all he misses out in this wonderful tour of what's wrong with the SI obsessives is the similarity of singularitarianism of the mind uploading variety to Christianity in terms of design patterns—hint: its roots in Russian Cosmism go back to a 19th century Orthodox theologian, Nikolai Fyodorovich Fyodorov—exemplified by the Satan of the Singularity, Roko's Basilisk, who will torment you forever in afterlife hell if you don't devote your every waking moment to bringing it into existence.
(And yes, I'm getting a space opera out of this.)
posted by cstross at 3:26 AM on December 23, 2016 [33 favorites]
Not sure what to make of the essay since it is mostly just providing a series of flawed arguments towards one extreme end for AI development, than gesturing towards the less grand concerns we should have instead. I mean, that's true enough, but then dwelling on the more extreme notions without talking more about how pushing for strong AI is more likely to lead to those less fanciful bad outcomes doesn't seem to me to be a really effective argument either way.
As something of an aside, my take on figuring the surest way to look for human-like intelligence is to give a machine a forbidden room test. Allow the machine to have complete freedom to learn all that it wants, save for what is in one room which the machine is forbidden to examine or learn anything about. The moment we find the machine has attempted to learn what's in the room is the moment we'd know it has human-like curiosity, and thus must have some kind of sentience.
That moment of curiosity and a desire to overcome the forbidden, of course, would also be the signal the machine needs to be destroyed as it cannot lead to good things for the rest of us. But as that is a key foundational story element in cultures across the world, we also know that those stories don't usually end well for the ones who do the forbidding, so...
posted by gusottertrout at 3:46 AM on December 23, 2016 [1 favorite]
As something of an aside, my take on figuring the surest way to look for human-like intelligence is to give a machine a forbidden room test. Allow the machine to have complete freedom to learn all that it wants, save for what is in one room which the machine is forbidden to examine or learn anything about. The moment we find the machine has attempted to learn what's in the room is the moment we'd know it has human-like curiosity, and thus must have some kind of sentience.
That moment of curiosity and a desire to overcome the forbidden, of course, would also be the signal the machine needs to be destroyed as it cannot lead to good things for the rest of us. But as that is a key foundational story element in cultures across the world, we also know that those stories don't usually end well for the ones who do the forbidding, so...
posted by gusottertrout at 3:46 AM on December 23, 2016 [1 favorite]
The moment we find the machine has attempted to learn what's in the room is the moment we'd know it has human-like curiosity, and thus must have some kind of sentience.
In college, for a neural networks class, I used genetic algorithms to evolve neural networks to play a competitive mario-style game in which the players tried to jump on each others' heads. But, my implementation of the game had a bug which would allow the characters to infinitely jump up against walls. The obvious outcome was that all the agents evolved to exploit this bug, to jam themselves in the upper corners of the map where they were protected from above.
Optimizers are going to optimize; I wouldn't read anything too metaphysical into it.
posted by Pyry at 4:12 AM on December 23, 2016 [10 favorites]
In college, for a neural networks class, I used genetic algorithms to evolve neural networks to play a competitive mario-style game in which the players tried to jump on each others' heads. But, my implementation of the game had a bug which would allow the characters to infinitely jump up against walls. The obvious outcome was that all the agents evolved to exploit this bug, to jam themselves in the upper corners of the map where they were protected from above.
Optimizers are going to optimize; I wouldn't read anything too metaphysical into it.
posted by Pyry at 4:12 AM on December 23, 2016 [10 favorites]
gusottertrout: Though it could just mean that your AI met a talking snake...
posted by rhamphorhynchus at 4:12 AM on December 23, 2016 [4 favorites]
posted by rhamphorhynchus at 4:12 AM on December 23, 2016 [4 favorites]
cstross: the similarity of singularitarianism of the mind uploading variety to Christianity in terms of design patterns—hint: its roots in Russian Cosmism
Russia seems to be to ideas what Australia is to wildlife. Transhumanism, Aleksandr Dugin's neo-hobbesian fascism, and the most virulent forms of Marxism (via Lenin) and Libertarianism (via St. Petersburg native Alicia Rosenbaum, better known as Ayn Rand) seem to have their roots in its fertile memetic soil.
posted by acb at 4:29 AM on December 23, 2016 [11 favorites]
Russia seems to be to ideas what Australia is to wildlife. Transhumanism, Aleksandr Dugin's neo-hobbesian fascism, and the most virulent forms of Marxism (via Lenin) and Libertarianism (via St. Petersburg native Alicia Rosenbaum, better known as Ayn Rand) seem to have their roots in its fertile memetic soil.
posted by acb at 4:29 AM on December 23, 2016 [11 favorites]
Optimizers are going to optimize; I wouldn't read anything too metaphysical into it.
Optimising for happiness might well end up with a sentient opium poppy.
posted by acb at 4:38 AM on December 23, 2016 [1 favorite]
Optimising for happiness might well end up with a sentient opium poppy.
posted by acb at 4:38 AM on December 23, 2016 [1 favorite]
I'm confused. Where does Neoreaction come into all this?
I don't know that there's a huge connection? But what comes to mind is that Nick Land, prolly the best neoreactionary, sees capitalism itself as some sort of all consuming irresistible Lovecraftian blind idiot god, that is coming to optimize you.
posted by save alive nothing that breatheth at 4:57 AM on December 23, 2016
I don't know that there's a huge connection? But what comes to mind is that Nick Land, prolly the best neoreactionary, sees capitalism itself as some sort of all consuming irresistible Lovecraftian blind idiot god, that is coming to optimize you.
posted by save alive nothing that breatheth at 4:57 AM on December 23, 2016
So which robots fix the plumbing?
posted by oceanjesse at 5:10 AM on December 23, 2016 [1 favorite]
posted by oceanjesse at 5:10 AM on December 23, 2016 [1 favorite]
Yes, it would be terrible if minds that are to our minds as ours are to those of the beasts that perish, intellects vast and cool and unsympathetic, should regard this earth with envious eyes, and slowly and surely draw their plans against us, but I find it a little odd to be hand-wringing about the possibility that e.g. military AIs would haul off and nuke a few cities given humanity's track record in that department (e.g. the democratically-elected president-elect of the world's most powerful country just tweeted that maybe it must greatly strengthen and expand its nuclear capability until such time as the world comes to its senses regarding, uh, greatly strengthening and expanding its nuclear capabilities).
Stepping back, dramatically underselling the risk of the present by dramatically overselling the risk of the far future feels like some classic Malinowskian functionalism mixed with the usual Silicon Valley funding land-grabbery servi dans une sauce de savoir-pouvoir.
Meanwhile, do any of the "only computer scientists can truly understand the arguments" drivers-by want to give us a taste of what he's missing? When you say "extended Church-Turing thesis", extended by whom and with what degree of rigor? Why is discrete mathematics the privileged viewpoint here and not, say, cognitive science or comparative religion?
posted by Coda at 6:35 AM on December 23, 2016 [8 favorites]
Stepping back, dramatically underselling the risk of the present by dramatically overselling the risk of the far future feels like some classic Malinowskian functionalism mixed with the usual Silicon Valley funding land-grabbery servi dans une sauce de savoir-pouvoir.
Meanwhile, do any of the "only computer scientists can truly understand the arguments" drivers-by want to give us a taste of what he's missing? When you say "extended Church-Turing thesis", extended by whom and with what degree of rigor? Why is discrete mathematics the privileged viewpoint here and not, say, cognitive science or comparative religion?
posted by Coda at 6:35 AM on December 23, 2016 [8 favorites]
I can't tell you how many times I've had variants of this precise conversation with comp sci Ph.D types in like upscale Bay Area lounging areas (soothing low volume dubstep plays) as they try to sell me something that for a problem that AI could help with.
Cut to a scene one month later where I am helplessly in front of a telnet window arguing with something not wholly unlike Dr. Sbaitso and looking for a full refund.
posted by mrdaneri at 6:42 AM on December 23, 2016
Cut to a scene one month later where I am helplessly in front of a telnet window arguing with something not wholly unlike Dr. Sbaitso and looking for a full refund.
posted by mrdaneri at 6:42 AM on December 23, 2016
Tell me more about I want a full refund.
posted by whir at 7:11 AM on December 23, 2016 [16 favorites]
posted by whir at 7:11 AM on December 23, 2016 [16 favorites]
I'm confused. Where does Neoreaction come into all this?
There's a scholarly ebook which ties the two together (currently unavailable, though getting some sort of permanent release next year). The hub seems to be Nick Land, and also the crossover between transhumanism, libertarianism and technological solutionism.
posted by acb at 7:12 AM on December 23, 2016 [3 favorites]
There's a scholarly ebook which ties the two together (currently unavailable, though getting some sort of permanent release next year). The hub seems to be Nick Land, and also the crossover between transhumanism, libertarianism and technological solutionism.
posted by acb at 7:12 AM on December 23, 2016 [3 favorites]
One aspect of Roko's Basilisk that never made sense to me: it's absolutely necessary, in the proposed scenario, for the hypothetical malevolent future superintelligence to persuade us peons in the here and now that we may be tortured in the future. But it is also a complete waste of resources to bother with actually following through on that threat, the persuasion is the only part that matters. If the basilisk exists, it's goal of forcing its own creation has succeeded and why bother punishing anyone or anything? The threat of the punishment has already had its effect or lack thereof, following through won't change the past.
This issue doesn't pose such a problem in old-timey religions, where the deity in charge is presumed to have goals and motivations that are unknowable to us. But by confining the basilisk to human-relatable goals and non-supernatural modes of action, we sort of automatically render it silly and impotent. Roko's Basilisk is just like God, but dumber.
posted by Western Infidels at 7:40 AM on December 23, 2016 [4 favorites]
This issue doesn't pose such a problem in old-timey religions, where the deity in charge is presumed to have goals and motivations that are unknowable to us. But by confining the basilisk to human-relatable goals and non-supernatural modes of action, we sort of automatically render it silly and impotent. Roko's Basilisk is just like God, but dumber.
posted by Western Infidels at 7:40 AM on December 23, 2016 [4 favorites]
There's a lot more drama than sense in the discussion of AI.
As practical matter of the people who actually spend the money to build AI at commercial scale or will buy it, AI is simply shorthand for "computers that can do tasks customarily requiring human judgment or intellect more efficiently than humans."
It seeks no particular end-points. It doesn't matter which task it reaches first or which tasks it never reaches. Efficiently AI-driven trucks will be profitable even if computers never get good at composing EDM tracks, and vice versa.
In the same manner as ends don't matter, AI-in-practice has zero concern for means. No one should -- and at the level of investment or acquisition, no one does -- care how people "think" or how AI will "think" or appear to think, and no one is going to seek to create a more enthusiastic or convincing simulcrum of "thinking." The guy who owns clubs in Ibiza won't care how the EDM AI works, just how many cover charges and bar tabs it generates and how much money he saves on licensing tracks and DJ fees. The trucking companies, insurance companies, and highway regulators won't care about how the truck-driving AI works, just how much it costs and its accident rate per mile versus human drivers. None of the foregoing care in the least about questions of motivation or cognition of the AI.
Interestingly, in grinding no ontological or epistemological axes, the development of AI quite elegantly parallels evolution, which seeks no ends and systematically deploys no means or principles of development. If something comes along, and happens to work, it goes forward. If not, not.
posted by MattD at 7:59 AM on December 23, 2016 [8 favorites]
As practical matter of the people who actually spend the money to build AI at commercial scale or will buy it, AI is simply shorthand for "computers that can do tasks customarily requiring human judgment or intellect more efficiently than humans."
It seeks no particular end-points. It doesn't matter which task it reaches first or which tasks it never reaches. Efficiently AI-driven trucks will be profitable even if computers never get good at composing EDM tracks, and vice versa.
In the same manner as ends don't matter, AI-in-practice has zero concern for means. No one should -- and at the level of investment or acquisition, no one does -- care how people "think" or how AI will "think" or appear to think, and no one is going to seek to create a more enthusiastic or convincing simulcrum of "thinking." The guy who owns clubs in Ibiza won't care how the EDM AI works, just how many cover charges and bar tabs it generates and how much money he saves on licensing tracks and DJ fees. The trucking companies, insurance companies, and highway regulators won't care about how the truck-driving AI works, just how much it costs and its accident rate per mile versus human drivers. None of the foregoing care in the least about questions of motivation or cognition of the AI.
Interestingly, in grinding no ontological or epistemological axes, the development of AI quite elegantly parallels evolution, which seeks no ends and systematically deploys no means or principles of development. If something comes along, and happens to work, it goes forward. If not, not.
posted by MattD at 7:59 AM on December 23, 2016 [8 favorites]
As practical matter of the people who actually spend the money to build AI at commercial scale or will buy it, AI is simply shorthand for "computers that can do tasks customarily requiring human judgment or intellect more efficiently than humans."
I'm of the opinion that superhuman AI is already here, it's just not very interesting to most people because machine intelligence doesn't have 1,000,000 years of evolutionary behavioral bias for interpersonal drama.
posted by CBrachyrhynchos at 8:25 AM on December 23, 2016 [3 favorites]
I'm of the opinion that superhuman AI is already here, it's just not very interesting to most people because machine intelligence doesn't have 1,000,000 years of evolutionary behavioral bias for interpersonal drama.
posted by CBrachyrhynchos at 8:25 AM on December 23, 2016 [3 favorites]
I'm of the opinion that superhuman AI is already here, it's just not very interesting to most people because machine intelligence doesn't have 1,000,000 years of evolutionary behavioral bias for interpersonal drama.
If you mean specific tasks such as facial recognition, audio transcription, and image style transfer, does it count as “superhuman AI”, or is it more like the first valve-based computers having “superhuman arithmetical capabilities” or steam locomotives having “superhuman speed”?
posted by acb at 9:15 AM on December 23, 2016
If you mean specific tasks such as facial recognition, audio transcription, and image style transfer, does it count as “superhuman AI”, or is it more like the first valve-based computers having “superhuman arithmetical capabilities” or steam locomotives having “superhuman speed”?
posted by acb at 9:15 AM on December 23, 2016
I reckon we've had AI for the past three centuries — joint stock corporations are non-human organisms optimized for maximizing profit, after all.
And the 21st century global financial system is a paperclip-maximizer AI, only instead of turning you into paperclips it turns you into debt. (Which it then uses as an economic entropy sink.)
posted by cstross at 9:24 AM on December 23, 2016 [14 favorites]
And the 21st century global financial system is a paperclip-maximizer AI, only instead of turning you into paperclips it turns you into debt. (Which it then uses as an economic entropy sink.)
posted by cstross at 9:24 AM on December 23, 2016 [14 favorites]
Incidentally, speaking parenthetically? I am really heartened to see that Sithrak has enough name recognition to make it into the title of a MeFi post! Hail Sithrak!
posted by cstross at 9:29 AM on December 23, 2016 [16 favorites]
posted by cstross at 9:29 AM on December 23, 2016 [16 favorites]
Can I recommend a story about an AI that optimizes for human happiness using My Little Pony as it's model? Brilliant, methinks.
https://www.fimfiction.net/story/62074/1
posted by kaymac at 9:34 AM on December 23, 2016 [2 favorites]
https://www.fimfiction.net/story/62074/1
posted by kaymac at 9:34 AM on December 23, 2016 [2 favorites]
So here's the thing: because we can't tell whether something is exhibiting intelligence, we can't design a self-improving AI.
But what if we build it by accident while optimizing for something specific? I'm not a big AI risk guy, but I've never really thought that lack of a single formal definition of "intelligence" has much to do with it.
posted by atoxyl at 9:43 AM on December 23, 2016
But what if we build it by accident while optimizing for something specific? I'm not a big AI risk guy, but I've never really thought that lack of a single formal definition of "intelligence" has much to do with it.
posted by atoxyl at 9:43 AM on December 23, 2016
The task of thinking about intelligence in terms not burdened by anthropomorphic ideas is a bit like really grasping string theory or Fermat's theorem. To use another analogy, our philosophy of mind is as adequate to describe non human intelligence as Marx describes the social behaviors of colonial insects.
posted by CBrachyrhynchos at 9:46 AM on December 23, 2016 [2 favorites]
posted by CBrachyrhynchos at 9:46 AM on December 23, 2016 [2 favorites]
Most of the arbitrary benchmarks set for intelligence have either been met or found to be not unique to humans.
posted by CBrachyrhynchos at 10:06 AM on December 23, 2016 [1 favorite]
posted by CBrachyrhynchos at 10:06 AM on December 23, 2016 [1 favorite]
> "It eventually became clear that 'AI' means something very different to farmers."
"You're going to college for that? We've got a good strong AI program right here on the farm!"
"You do? What approach do you use?"
"Oh, these days it's all done through a kind of distributed network."
"Really?"
"Oh, yeah, the internet really changed things for us."
"You had this program before the internet?"
"Sure we did. Back in those days, though, we had to have everything on site."
"That's amazing!"
(The misunderstanding is never cleared up, and ten years later cyborg cows rule the earth.)
posted by kyrademon at 10:26 AM on December 23, 2016 [10 favorites]
"You're going to college for that? We've got a good strong AI program right here on the farm!"
"You do? What approach do you use?"
"Oh, these days it's all done through a kind of distributed network."
"Really?"
"Oh, yeah, the internet really changed things for us."
"You had this program before the internet?"
"Sure we did. Back in those days, though, we had to have everything on site."
"That's amazing!"
(The misunderstanding is never cleared up, and ten years later cyborg cows rule the earth.)
posted by kyrademon at 10:26 AM on December 23, 2016 [10 favorites]
a simple algorithm whose 'aggression counter' tried to subtract 1 from 0 on a unsigned int and got set to maximum kill everything mode
Great, we're all going to die at the hands of Civilization's Gandhi.
As for Bridgewater, well, it's notoriously run by a guy who is fanatically committed to a set of "Principles" that he believes maximize the chances for success through rigorously cultivating merit but which any college undergrad could pick apart on the merit issue. Ray Dalio makes Engineer's Disease look like a mild sniffle. He just can't deal with the idea that his personal, oh-so-meritorious consciousness is going to come to an end.
So much of this activity in SV is just a bunch of old dudes trying to escape the looming shadow of: "last week, as it must to all men, death came to Charles Foster Kane..."
posted by praemunire at 10:54 AM on December 23, 2016 [3 favorites]
Great, we're all going to die at the hands of Civilization's Gandhi.
As for Bridgewater, well, it's notoriously run by a guy who is fanatically committed to a set of "Principles" that he believes maximize the chances for success through rigorously cultivating merit but which any college undergrad could pick apart on the merit issue. Ray Dalio makes Engineer's Disease look like a mild sniffle. He just can't deal with the idea that his personal, oh-so-meritorious consciousness is going to come to an end.
So much of this activity in SV is just a bunch of old dudes trying to escape the looming shadow of: "last week, as it must to all men, death came to Charles Foster Kane..."
posted by praemunire at 10:54 AM on December 23, 2016 [3 favorites]
I am really heartened to see that Sithrak has enough name recognition to make it into the title of a MeFi post! Hail Sithrak!
Sithrak does not care about your adulation. He hates you regardless.
posted by TheWhiteSkull at 11:00 AM on December 23, 2016 [4 favorites]
Sithrak does not care about your adulation. He hates you regardless.
posted by TheWhiteSkull at 11:00 AM on December 23, 2016 [4 favorites]
There is no well-grounded science that can pick out either a thing called a mind or an activity called thinking.
This is an odd thing to say. How is "well-grounded science" picking things out, without thinking and performing other mental operations?
posted by thelonius at 11:08 AM on December 23, 2016
This is an odd thing to say. How is "well-grounded science" picking things out, without thinking and performing other mental operations?
posted by thelonius at 11:08 AM on December 23, 2016
How is "well-grounded science" picking things out, without thinking and performing other mental operations?
"Well-grounded science" can't actually adequately model why "well-grounded science" could exist or work.
posted by praemunire at 11:14 AM on December 23, 2016 [4 favorites]
"Well-grounded science" can't actually adequately model why "well-grounded science" could exist or work.
posted by praemunire at 11:14 AM on December 23, 2016 [4 favorites]
I wish people in Computer Science and AI research spent a little more time attending to issues in philosophy
Or to rephrase and expand upon myself - I find the philosophical issues interesting but I think (certain) AI people are right in suggesting that these issues are ultimately of limited relevance to them. For most AI research this is because AI is just a fancy term for solving certain narrow problems with certain methods anyway. Even for "Strong AI," "General AI," whatever the existence of "human intelligence" despite lack of a clear definition or understanding of "human intelligence" seems to suggest there's no reason whatever it is couldn't happen again through some sort of accelerated evolutionary process. That we wouldn't necessarily even know or agree that it happened probably actually helps the case of those who are concerned - ultimately it doesn't matter what AI is but what it can do.
I just don't really buy into the "tiny probability times huge theoretical risk on an unknown timescale equals immediate importance" school of resource allocation. My instinct says they are overrating certain flashy or philosophically interesting apocalyptic scenarios when there are others just as bad and at least as likely - or actually measurably in progress.
posted by atoxyl at 1:50 PM on December 23, 2016 [4 favorites]
Or to rephrase and expand upon myself - I find the philosophical issues interesting but I think (certain) AI people are right in suggesting that these issues are ultimately of limited relevance to them. For most AI research this is because AI is just a fancy term for solving certain narrow problems with certain methods anyway. Even for "Strong AI," "General AI," whatever the existence of "human intelligence" despite lack of a clear definition or understanding of "human intelligence" seems to suggest there's no reason whatever it is couldn't happen again through some sort of accelerated evolutionary process. That we wouldn't necessarily even know or agree that it happened probably actually helps the case of those who are concerned - ultimately it doesn't matter what AI is but what it can do.
I just don't really buy into the "tiny probability times huge theoretical risk on an unknown timescale equals immediate importance" school of resource allocation. My instinct says they are overrating certain flashy or philosophically interesting apocalyptic scenarios when there are others just as bad and at least as likely - or actually measurably in progress.
posted by atoxyl at 1:50 PM on December 23, 2016 [4 favorites]
So many allusions at faulty arguments and missing background, but so few specific examples.
I think the most important part was what XMLicious already highlighted. (cf.: Weapons of Math Destruction, previously)
posted by kmt at 3:49 PM on December 23, 2016 [1 favorite]
I think the most important part was what XMLicious already highlighted. (cf.: Weapons of Math Destruction, previously)
posted by kmt at 3:49 PM on December 23, 2016 [1 favorite]
Federal report: AI could threaten up to 47 percent of jobs in two decades (Ars Technica)
posted by sebastienbailard at 5:35 PM on December 23, 2016
posted by sebastienbailard at 5:35 PM on December 23, 2016
Federal report: AI could threaten up to 47 percent of jobs in two decades (Ars Technica)
So all that hype about teaching kids to code is setting them up for failure it seems. Or do we imagine that AI doesn't write code?
posted by Brian B. at 6:02 PM on December 23, 2016
So all that hype about teaching kids to code is setting them up for failure it seems. Or do we imagine that AI doesn't write code?
posted by Brian B. at 6:02 PM on December 23, 2016
So all that hype about teaching kids to code is setting them up for failure it seems. Or do we imagine that AI doesn't write code?
It does, but certain kinds of code writing - for example, the kind required to produce new forms of AI - are probably going to be the among the very last endeavors to be automated. (This doesn't solve anything employment-wise because the number of people who are ever going to have the required skills for that and get the chance to use them is unlikely to be a very large percentage of people.)
posted by atoxyl at 7:31 PM on December 23, 2016
It does, but certain kinds of code writing - for example, the kind required to produce new forms of AI - are probably going to be the among the very last endeavors to be automated. (This doesn't solve anything employment-wise because the number of people who are ever going to have the required skills for that and get the chance to use them is unlikely to be a very large percentage of people.)
posted by atoxyl at 7:31 PM on December 23, 2016
In our grim future, Silicon Valley will be the last place that hired anybody (caveat: must be Ivy League graduate. Plebes need not apply)
posted by Yowser at 8:06 PM on December 23, 2016 [1 favorite]
posted by Yowser at 8:06 PM on December 23, 2016 [1 favorite]
You know what Roko's Basilisk reminds me of more than anything, is Anselm's Proslogion, a proof for God's existence. He uses a gimmick that seems like it should also work for proving the existence of lots of things besides God, but never goes any further with it.
As tobascodagama points out, why such a lack of concern over Roko's Basilisk's Basilisk? Where's the Shinto version of this with eight million Basilisks?
posted by XMLicious at 12:58 AM on December 24, 2016 [3 favorites]
As tobascodagama points out, why such a lack of concern over Roko's Basilisk's Basilisk? Where's the Shinto version of this with eight million Basilisks?
posted by XMLicious at 12:58 AM on December 24, 2016 [3 favorites]
a gimmick that seems like it should also work for proving the existence of lots of things besides God
Yes, the "imagine an island more perfect than any other which may be conceived" proof that a perfect island exists, which, iirc, even one of his contemporaries pointed out to him as an objection.
posted by thelonius at 1:04 AM on December 24, 2016 [1 favorite]
Yes, the "imagine an island more perfect than any other which may be conceived" proof that a perfect island exists, which, iirc, even one of his contemporaries pointed out to him as an objection.
posted by thelonius at 1:04 AM on December 24, 2016 [1 favorite]
That seems resolvable if you're willing to commit to belief in the platonic world of ideal forms as real but distinct from physical reality
posted by vibratory manner of working at 9:21 AM on December 25, 2016
posted by vibratory manner of working at 9:21 AM on December 25, 2016
The problem, as I see it, (or at least as I saw it whenever I was reading about this, if I understood it correctly) is that the gimmick can be bolted onto any other attribute. Like, I could define "greenness-prime" as an attribute that's just like regular greenness, except that something which is both green and existent is more greenness-prime than things which are green but not existent. Then, whatever the superlatively greenness-prime thing is, it's proved to exist by the same logic.
I think there's a big thread of reasoning about which characteristics would derive from Platonic forms and which wouldn't, isn't there? For that gimmick to be consonant with Platonic forms, it would have to only work with characteristics that are "allowed" to be forms, or whatever the terminology is, I would think.
posted by XMLicious at 10:25 PM on December 25, 2016
I think there's a big thread of reasoning about which characteristics would derive from Platonic forms and which wouldn't, isn't there? For that gimmick to be consonant with Platonic forms, it would have to only work with characteristics that are "allowed" to be forms, or whatever the terminology is, I would think.
posted by XMLicious at 10:25 PM on December 25, 2016
One aspect of Roko's Basilisk that never made sense to me: it's absolutely necessary, in the proposed scenario, for the hypothetical malevolent future superintelligence to persuade us peons in the here and now that we may be tortured in the future. But it is also a complete waste of resources to bother with actually following through on that threat, the persuasion is the only part that matters.
And that's what makes the supposed threat of Roko's Basilisk completely empty and impotent. There's no way a basilisk could prove it was actually simulating people and punishing or rewarding them, so it would be pointless to actually do so solely for purposes of persuasion. It would be pointless for an AI to make promises or threats to reward or punish simulated people when those actions are obviously unverifiable.
posted by straight at 11:44 PM on December 25, 2016
And that's what makes the supposed threat of Roko's Basilisk completely empty and impotent. There's no way a basilisk could prove it was actually simulating people and punishing or rewarding them, so it would be pointless to actually do so solely for purposes of persuasion. It would be pointless for an AI to make promises or threats to reward or punish simulated people when those actions are obviously unverifiable.
posted by straight at 11:44 PM on December 25, 2016
I seem to recall that the Basilisk's proponents argue that the creators of a nascent AI (or the AI itself) would "bind" the AI so that it had to carry out the threat. The reason being, without the threat it would be impossible for the AI to gain enough support to come into existence. So handwave, handwave, any godlike AI that actually exists is bound to torture the reincarnations of faithless non-supporters.
posted by Joe in Australia at 12:43 AM on December 26, 2016
posted by Joe in Australia at 12:43 AM on December 26, 2016
using terminology like "bind" makes it sound even more like traditional demonology
posted by vibratory manner of working at 8:47 AM on December 26, 2016 [5 favorites]
posted by vibratory manner of working at 8:47 AM on December 26, 2016 [5 favorites]
Like, I could define "greenness-prime" as an attribute that's just like regular greenness, except that something which is both green and existent is more greenness-prime than things which are green but not existent. Then, whatever the superlatively greenness-prime thing is, it's proved to exist by the same logic.
This is a lot of why philosophers (following Kant iirc) mostly agree that "existence is not a property", which is kind of a shame, because it is amusing to think of, oh, unicorns as having various properties, but happening to lack the property of existing.
posted by thelonius at 8:58 AM on December 26, 2016 [4 favorites]
This is a lot of why philosophers (following Kant iirc) mostly agree that "existence is not a property", which is kind of a shame, because it is amusing to think of, oh, unicorns as having various properties, but happening to lack the property of existing.
posted by thelonius at 8:58 AM on December 26, 2016 [4 favorites]
Roko's basilisk is kind of a fun thought experiment but it kinda surprises me that people are genuinely bothered by it. I'm not sure I'm really bothered by the notion of a simulated-me being tortured. I don't really know if I'm typical in that respect or not, but whenever I bring up the notion that Star Trek replicators are essentially murdering you and then making a new you in another place, most people don't seem too bothered by that notion. And almost anyone that you ask about what they'd do if they had a clone of themselves says they'd make it do all the boring shit they don't want to do, even though it's "them"
posted by RustyBrooks at 1:58 PM on December 26, 2016 [2 favorites]
posted by RustyBrooks at 1:58 PM on December 26, 2016 [2 favorites]
I don't really know if I'm typical in that respect or not, but whenever I bring up the notion that Star Trek replicators are essentially murdering you and then making a new you in another place, most people don't seem too bothered by that notion.
Think that's bad, replicators (and transporters, for when you only want one copy of your away team, which is silly when redshirts are a primary consumable for any self-regarding expedition) are science fiction. You should see how uptight the busybodies get when we're reenacting fundamental experiments pushing people in front of trolleys. No idea how we're supposed to teach introductory ethics when self-appointed "oversight committees" are wrapping kids in cotton wool.
posted by sebastienbailard at 12:45 AM on December 27, 2016 [2 favorites]
Think that's bad, replicators (and transporters, for when you only want one copy of your away team, which is silly when redshirts are a primary consumable for any self-regarding expedition) are science fiction. You should see how uptight the busybodies get when we're reenacting fundamental experiments pushing people in front of trolleys. No idea how we're supposed to teach introductory ethics when self-appointed "oversight committees" are wrapping kids in cotton wool.
posted by sebastienbailard at 12:45 AM on December 27, 2016 [2 favorites]
There's no way a basilisk could prove it was actually simulating people and punishing or rewarding them, so it would be pointless to actually do so solely for purposes of persuasion.
God seems to have the same problem.
Some of his followers seem to have noticed this, and have therefore taken up punishing people themselves for the purpose of persuasion, apparently not content in any faith that God is able to do his own work.
All kinds of recapitulation of theology going around, I suppose.
posted by wildblueyonder at 1:59 AM on December 27, 2016 [7 favorites]
God seems to have the same problem.
Some of his followers seem to have noticed this, and have therefore taken up punishing people themselves for the purpose of persuasion, apparently not content in any faith that God is able to do his own work.
All kinds of recapitulation of theology going around, I suppose.
posted by wildblueyonder at 1:59 AM on December 27, 2016 [7 favorites]
Wil McCarthy's Queendom of Sol series deals with the problem of transporters by saying that in a properly secular society, as long as the illusion of continuity of consciousness and memory is maintained, people won't give a shit about destruction of this or that version. It only becomes a problem when people start screwing around with the data used to "print" a person. You can even merge the data files so that you get that illusion of consciousness with a fresh body every day.
Of course, that opens up an entirely new form of crime.
posted by CBrachyrhynchos at 12:17 PM on December 27, 2016 [2 favorites]
Of course, that opens up an entirely new form of crime.
posted by CBrachyrhynchos at 12:17 PM on December 27, 2016 [2 favorites]
I'm waiting for Roko's Basilisk to become a Dr. Who villain, since I think time travel is a bit necessary for pre-creation existential risk to make any sense, but it might be too convoluted for even Dr. Who to tackle.
posted by CBrachyrhynchos at 12:59 PM on December 27, 2016
posted by CBrachyrhynchos at 12:59 PM on December 27, 2016
Dumb idea of the day: Thanksgiving dinner is Roko's Basilisk for the mass extinction of dinosaurs.
posted by CBrachyrhynchos at 1:08 PM on December 27, 2016 [3 favorites]
posted by CBrachyrhynchos at 1:08 PM on December 27, 2016 [3 favorites]
I seem to recall that the Basilisk's proponents argue that the creators of a nascent AI (or the AI itself) would "bind" the AI so that it had to carry out the threat.
They could try, but would have no way of knowing if they were successful, nor would there be any way to verify the AI's claims of having bound itself. The whole premise falls apart unless people have a logical certainty that the AI would necessarily do something in the future to motivate them to act a certain way, but such certainty is intrinsically impossible. Therefore there is no reason to think an AI would actually do anything to simulated people to motivate us, either in the present or the past. We can call its bluff and it has no way to prove it isn't bluffing.
posted by straight at 1:19 PM on December 27, 2016 [2 favorites]
They could try, but would have no way of knowing if they were successful, nor would there be any way to verify the AI's claims of having bound itself. The whole premise falls apart unless people have a logical certainty that the AI would necessarily do something in the future to motivate them to act a certain way, but such certainty is intrinsically impossible. Therefore there is no reason to think an AI would actually do anything to simulated people to motivate us, either in the present or the past. We can call its bluff and it has no way to prove it isn't bluffing.
posted by straight at 1:19 PM on December 27, 2016 [2 favorites]
The whole premise falls apart unless people have a logical certainty that the AI would necessarily do something in the future to motivate them to act a certain way, but such certainty is intrinsically impossible.
The same goes for religious threats of eternal damnation, but those have been pretty successful.
You don't need certainty. It's like Pascal's Wager: there's a small chance that AI cultists will be successful in creating an AI, multiplied by a small chance that they'll be successful in binding the AI to persecute the reincarnated images of faithless people, multiplied by the infinite cost of eternal damnation for your mental image. If you accept the premise that you should care about the happiness of your mental image (something that frankly doesn't persuade me) then you have no real choice but to help their project.
posted by Joe in Australia at 2:06 PM on December 27, 2016
The same goes for religious threats of eternal damnation, but those have been pretty successful.
You don't need certainty. It's like Pascal's Wager: there's a small chance that AI cultists will be successful in creating an AI, multiplied by a small chance that they'll be successful in binding the AI to persecute the reincarnated images of faithless people, multiplied by the infinite cost of eternal damnation for your mental image. If you accept the premise that you should care about the happiness of your mental image (something that frankly doesn't persuade me) then you have no real choice but to help their project.
posted by Joe in Australia at 2:06 PM on December 27, 2016
The computational impossibility of simulating a chaotic system (one where small small errors in the state of the system produce larger errors) over significant time spans was proven over a century ago. So Roko's Basilisk starts with a premise like, "assume an AI can find all prime numbers."
posted by CBrachyrhynchos at 2:29 PM on December 27, 2016
posted by CBrachyrhynchos at 2:29 PM on December 27, 2016
anyone that you ask about what they'd do if they had a clone of themselves says they'd make it do all the boring shit they don't want to do
Ha, I can barely make my single self do a moiety of the boring shit, two of me are going to be hip-deep in mutually assured distraction.
posted by clew at 3:16 PM on December 27, 2016 [1 favorite]
Ha, I can barely make my single self do a moiety of the boring shit, two of me are going to be hip-deep in mutually assured distraction.
posted by clew at 3:16 PM on December 27, 2016 [1 favorite]
There might be an AI that decides to torment simulated people for its own reasons, which would be unfortunate, but there's no way we could ever know whether it was doing so or truthful about the reasons for doing it, so it would be pointless trying to change our behavior in an attempt to influence the AI's simulations.
Since it would be illogical for us to believe these sorts of threats or promises, it would be futile and illogical for the AI to carry out such threats or promises solely to influence our behavior. If it wants to torture simulated people, it will for its own reasons. A death penalty can't act as a deterrent if there is no way anyone can ever know whether or not it is carried out. Therefore nobody would carry out such an unknowable death penalty solely as a deterrent.
posted by straight at 3:19 PM on December 27, 2016
Since it would be illogical for us to believe these sorts of threats or promises, it would be futile and illogical for the AI to carry out such threats or promises solely to influence our behavior. If it wants to torture simulated people, it will for its own reasons. A death penalty can't act as a deterrent if there is no way anyone can ever know whether or not it is carried out. Therefore nobody would carry out such an unknowable death penalty solely as a deterrent.
posted by straight at 3:19 PM on December 27, 2016
An AI motivated by vengeance (or some sense of 'justice') for perceived slights would be be scary, but that's not what Roko's Basilisk is about.
posted by straight at 3:23 PM on December 27, 2016
posted by straight at 3:23 PM on December 27, 2016
It can be further complicated:
Are you right now living in the fundamental (or more-fundamental) universe, prior to the creation of the Basilisk, or has your entire life been a Basilisk simulation of a pre-Basilisk universe? If you're in the fundamental universe, your choice supports or doesn't support the creation of the Basilisk, but if you're in the Basilisk's universe, your choice is a test resulting in eternal damnation or salvation/obliteration. And you have no way of knowing.
I believe a decently-accepted solution is to resolve to refuse all of this acausal trickery from hypothetical future superbeings no matter what...
posted by save alive nothing that breatheth at 4:08 PM on December 27, 2016
Are you right now living in the fundamental (or more-fundamental) universe, prior to the creation of the Basilisk, or has your entire life been a Basilisk simulation of a pre-Basilisk universe? If you're in the fundamental universe, your choice supports or doesn't support the creation of the Basilisk, but if you're in the Basilisk's universe, your choice is a test resulting in eternal damnation or salvation/obliteration. And you have no way of knowing.
I believe a decently-accepted solution is to resolve to refuse all of this acausal trickery from hypothetical future superbeings no matter what...
posted by save alive nothing that breatheth at 4:08 PM on December 27, 2016
If the reason our Benign STEM Overlords are so obsessed with proving that we're living in a simulated reality is that they're afraid of being tortured by Roko's Basilisk, I am going to shit.
posted by tobascodagama at 5:41 PM on December 27, 2016 [1 favorite]
posted by tobascodagama at 5:41 PM on December 27, 2016 [1 favorite]
The computational impossibility of simulating a chaotic system (one where small small errors in the state of the system produce larger errors) over significant time spans was proven over a century ago.
I would really like to know what result you are referring to (admittedly, mostly because it seems you are misinterpreting something). Also, as ludicrous as the whole Basilisk idea is, I do not think that complexity arguments can be used to invalidate the premise. After all, if you want to torture some simulated person, the simulation does not have to run efficiently -- you can just pause the simulation, or run it very slowly in comparison to real time.
posted by erdferkel at 4:27 AM on December 28, 2016 [2 favorites]
I would really like to know what result you are referring to (admittedly, mostly because it seems you are misinterpreting something). Also, as ludicrous as the whole Basilisk idea is, I do not think that complexity arguments can be used to invalidate the premise. After all, if you want to torture some simulated person, the simulation does not have to run efficiently -- you can just pause the simulation, or run it very slowly in comparison to real time.
posted by erdferkel at 4:27 AM on December 28, 2016 [2 favorites]
I believe a decently-accepted solution is to resolve to refuse all of this acausal trickery from hypothetical future superbeings no matter what...
That seems to be Yudkowsky's solution, or close to it - per the rationalwiki article, he "does not consider open discussion of the notion of acausal trade with possible superintelligences to be provably safe" and believes "rational agents ignore blackmail threats (and meta-blackmail threats and so on)"
That doesn't foreclose acausal trades with future AIs entirely, but it's pretty close.
posted by vibratory manner of working at 8:56 AM on December 28, 2016
That seems to be Yudkowsky's solution, or close to it - per the rationalwiki article, he "does not consider open discussion of the notion of acausal trade with possible superintelligences to be provably safe" and believes "rational agents ignore blackmail threats (and meta-blackmail threats and so on)"
That doesn't foreclose acausal trades with future AIs entirely, but it's pretty close.
posted by vibratory manner of working at 8:56 AM on December 28, 2016
I wrote a short story this year about an AI that keeps resurrecting the long-dead loved ones from the human protagonists' life. Despite a continued dialogue about why this is rude and painful, the AI keeps "surprising" the human with people from the human's past, certain that they just need to give the human the correct pal to kick the human out of the doldrums. The AI is not malevolent or mean, nor is it kind for keeping the human alive, it just has a version of human happiness that it keeps trying to exploit because it likes the human and wants the human to be happy.
I think that incomprehensible sadness loop is very likely in the case of AI, to whatever degree it ever emerges.
posted by Tevin at 1:38 PM on December 28, 2016 [2 favorites]
I think that incomprehensible sadness loop is very likely in the case of AI, to whatever degree it ever emerges.
posted by Tevin at 1:38 PM on December 28, 2016 [2 favorites]
Are you right now living in the fundamental (or more-fundamental) universe, prior to the creation of the Basilisk, or has your entire life been a Basilisk simulation of a pre-Basilisk universe? If you're in the fundamental universe, your choice supports or doesn't support the creation of the Basilisk, but if you're in the Basilisk's universe, your choice is a test resulting in eternal damnation or salvation/obliteration. And you have no way of knowing.
This still has the same problem. Since real-me has no way of knowing whether the Basilisk is actually simulating me or not, it would be absurd for the Basilisk to use the computational power to run those simulations solely for the purpose of trying to influence my behavior. Therefore the Basilisk isn't doing it. Therefore I'm real.
If this Basilisk is simulating me for other reasons besides influencing real-me's behavior, then real-me by definition can't change that by changing my behavior, so again it's pointless and illogical to do anything in hopes of influencing what an AI does or doesn't simulate.
posted by straight at 4:28 PM on December 28, 2016 [1 favorite]
This still has the same problem. Since real-me has no way of knowing whether the Basilisk is actually simulating me or not, it would be absurd for the Basilisk to use the computational power to run those simulations solely for the purpose of trying to influence my behavior. Therefore the Basilisk isn't doing it. Therefore I'm real.
If this Basilisk is simulating me for other reasons besides influencing real-me's behavior, then real-me by definition can't change that by changing my behavior, so again it's pointless and illogical to do anything in hopes of influencing what an AI does or doesn't simulate.
posted by straight at 4:28 PM on December 28, 2016 [1 favorite]
This is a lot of why philosophers (following Kant iirc) mostly agree that "existence is not a property", which is kind of a shame, because it is amusing to think of, oh, unicorns as having various properties, but happening to lack the property of existing.
Which is a bit like the hypothetical particle known as the ubiquiton, whose sole property is that there is one at any point in space.
posted by acb at 5:07 AM on December 29, 2016
Which is a bit like the hypothetical particle known as the ubiquiton, whose sole property is that there is one at any point in space.
posted by acb at 5:07 AM on December 29, 2016
I guess I'm still stuck on the assumption that simulating a specific physical human from first principles is computable and not one of those messy problems that demand infinite complexity.
posted by CBrachyrhynchos at 12:24 PM on December 29, 2016 [1 favorite]
posted by CBrachyrhynchos at 12:24 PM on December 29, 2016 [1 favorite]
« Older The Ultimate Guide To Winning Your White Elephant... | Melbourne Newer »
This thread has been archived and is closed to new comments
Don't buy into the neoreactionary koolaid. It's all horseshit.
posted by thsmchnekllsfascists at 4:08 PM on December 22, 2016 [8 favorites]