You can see the future first in San Francisco.
June 13, 2024 1:17 AM Subscribe
People are flipping out over Leopold Aschenbrenner's gargantuan look at the current and future state of AI in Situational Awareness [PDF]. Is it the start of the world? Is it the end of the world? When? 2027! Summary by ChatGPT.
Maybe you can see the future first in Dallas? [Previously]
Or is the future no more than what the markets will pay for?
Maybe you can see the future first in Dallas? [Previously]
Or is the future no more than what the markets will pay for?
Leopold Aschenbrenner’s interview on AGI by 2027 with Dwarkesh Patel last week
Ilya Sutskever, co-founder and until last month Chief Scientist of OpenAI, as well as the founder of its superalignment team (safety faction) discussing AGI in an interview with Dwarkesh Patel last year
John Schulman, OpenAI co-founder and new lead of the alignment team under the now firmly-established Sam Altman-Microsoft regime discussing AGI by 2027 in an interview with Dwarkesh Patel last month (almost immediately after Ilya’s departure was announced). Includes a surprising outright dismissal of model collapse (the LLMs eating their own dogfood problem) as a serious issue for OpenAI.
Jan Leike, former next-in-line of the superalignment team under Ilya, who quit simultaneously, with some pretty candid tweets on his departure and OpenAI’s internal failures with prioritizing alignment (seems like he’s the only person either not under a non-disparagement exit clause, or is just willing to risk testing the limits of it).
It should be noted here that most of these interviews are using OpenAI’s internal definition of AGI, which according to CEO Sam Altman roughly translates to “systems which can replace >50% of office workers,” not necessarily Data from Star Trek AI or superhuman intelligence, although they do delve into that level of AGI at times, particularly Ilya. On Ilya: he was simultaneously OpenAI’s loudest voice on the imminent arrival of AGI and the dangers it posed. He was a leader in the abortive Sam Altman firing which was likely due in part to unauthorized attempts at securing funding from Middle Eastern dictatorships, and due in part to a general disregard for safety and compute resource starvation experienced by the superalignment team as alleged by Leike.
On that note: Francois Chollet, prominent AI researcher at Google on why LLMs will not lead to AGI, why OpenAI has set back the development of true AGI by 5 to 10 years, and announcing a $1 million AGI prize / new benchmark in an interview with Dwarkesh Patel two days ago (ARC). Not as good as the others, lots of talking past each other.
I won’t link it because I consider him to have had a horrifying impact on global politics and the human race in general, but I have to acknowledge Zuck’s interview with Patel is fucking excellent, and that he has lead Meta in providing essential tools and models to the open source AI community. He can take a substantial amount of credit for us little people having even passable versions of these systems and not being shut out completely, the rat bastard.
(I should probably caveat I’ve only gotten through the first half or two thirds of all these interviews, which is why I haven’t written a comprehensive post yet: I’ve finished none of them. Work is crazy and I am so fucking tired.)
IMO, the single best information roundup source for everything in this field right now is definitely the AI Explained Youtube channel. Dude actually reads through all the papers, distills them in easily understood terms, dissects all the interviews, correlates these with public tweets, articles and other social media posts by the principal figures, organizes a major machine learning Discord and is just generally doing a better job of painting a comprehensive picture of what the fuck is going in a well-packaged weekly digest than anybody else by an almost ridiculous margin. Good 50/50 hype/skepticism mix though he occasionally strays a bit too far into the former for my taste. Here’s a link to his breakdown on the Q* papers, which (again IMO) is by far the strongest candidate for successfully grafting runtime reinforcement learning into current LLMs and achieving some limited degree of genuine “reasoning.” Maybe enough to meet OpenAI’s worker-hostile standard for AGI by the second or third release. The kicker is: as written this proposal sounds like it would take machine learning from its current small percentage of the electricity wasted on cryptocurrency to matching or even greatly exceeding it. See also: Microsoft announcing plans for a $100 billion AI supercomputer by 2028 (when the third generation of Q* should be launching assuming current release cadence continues), and Sam Altman seeking $7 trillion from the UAE for custom compute chip designs and datacenter buildouts.
Also re: work being crazy - I probably won’t be able to comment after this today. But the above is most of the post I’ve been quietly assembling ever since Ilya/Leike’s departure was announced.
posted by Ryvar at 2:47 AM on June 13 [39 favorites]
Ilya Sutskever, co-founder and until last month Chief Scientist of OpenAI, as well as the founder of its superalignment team (safety faction) discussing AGI in an interview with Dwarkesh Patel last year
John Schulman, OpenAI co-founder and new lead of the alignment team under the now firmly-established Sam Altman-Microsoft regime discussing AGI by 2027 in an interview with Dwarkesh Patel last month (almost immediately after Ilya’s departure was announced). Includes a surprising outright dismissal of model collapse (the LLMs eating their own dogfood problem) as a serious issue for OpenAI.
Jan Leike, former next-in-line of the superalignment team under Ilya, who quit simultaneously, with some pretty candid tweets on his departure and OpenAI’s internal failures with prioritizing alignment (seems like he’s the only person either not under a non-disparagement exit clause, or is just willing to risk testing the limits of it).
It should be noted here that most of these interviews are using OpenAI’s internal definition of AGI, which according to CEO Sam Altman roughly translates to “systems which can replace >50% of office workers,” not necessarily Data from Star Trek AI or superhuman intelligence, although they do delve into that level of AGI at times, particularly Ilya. On Ilya: he was simultaneously OpenAI’s loudest voice on the imminent arrival of AGI and the dangers it posed. He was a leader in the abortive Sam Altman firing which was likely due in part to unauthorized attempts at securing funding from Middle Eastern dictatorships, and due in part to a general disregard for safety and compute resource starvation experienced by the superalignment team as alleged by Leike.
On that note: Francois Chollet, prominent AI researcher at Google on why LLMs will not lead to AGI, why OpenAI has set back the development of true AGI by 5 to 10 years, and announcing a $1 million AGI prize / new benchmark in an interview with Dwarkesh Patel two days ago (ARC). Not as good as the others, lots of talking past each other.
I won’t link it because I consider him to have had a horrifying impact on global politics and the human race in general, but I have to acknowledge Zuck’s interview with Patel is fucking excellent, and that he has lead Meta in providing essential tools and models to the open source AI community. He can take a substantial amount of credit for us little people having even passable versions of these systems and not being shut out completely, the rat bastard.
(I should probably caveat I’ve only gotten through the first half or two thirds of all these interviews, which is why I haven’t written a comprehensive post yet: I’ve finished none of them. Work is crazy and I am so fucking tired.)
IMO, the single best information roundup source for everything in this field right now is definitely the AI Explained Youtube channel. Dude actually reads through all the papers, distills them in easily understood terms, dissects all the interviews, correlates these with public tweets, articles and other social media posts by the principal figures, organizes a major machine learning Discord and is just generally doing a better job of painting a comprehensive picture of what the fuck is going in a well-packaged weekly digest than anybody else by an almost ridiculous margin. Good 50/50 hype/skepticism mix though he occasionally strays a bit too far into the former for my taste. Here’s a link to his breakdown on the Q* papers, which (again IMO) is by far the strongest candidate for successfully grafting runtime reinforcement learning into current LLMs and achieving some limited degree of genuine “reasoning.” Maybe enough to meet OpenAI’s worker-hostile standard for AGI by the second or third release. The kicker is: as written this proposal sounds like it would take machine learning from its current small percentage of the electricity wasted on cryptocurrency to matching or even greatly exceeding it. See also: Microsoft announcing plans for a $100 billion AI supercomputer by 2028 (when the third generation of Q* should be launching assuming current release cadence continues), and Sam Altman seeking $7 trillion from the UAE for custom compute chip designs and datacenter buildouts.
Also re: work being crazy - I probably won’t be able to comment after this today. But the above is most of the post I’ve been quietly assembling ever since Ilya/Leike’s departure was announced.
posted by Ryvar at 2:47 AM on June 13 [39 favorites]
Kool-Aid city. See you back here in ten years Leopold, when we still mainly have jazzed up search engines and lots of useful applications in filmmaking and video games.
posted by GallonOfAlan at 2:56 AM on June 13 [3 favorites]
posted by GallonOfAlan at 2:56 AM on June 13 [3 favorites]
We don't have these techniques. It's been a common refrain that domain-specific knowledge and model-based approaches always get beaten by scale (https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf), but we're actually hitting the limits of scale.
Briefly: the Q* papers represent an approach that seems likely to successfully brute force some limited ability to correlate real world context and complete complex workloads / perform useful intellectual labor. I agree we have fuck-all in the pipeline for spontaneously spooling up general-case systems models for problem solving in which expert systems successfully predict the behavior of multiple interacting agents (as in: how humans will hypothetically respond to what an AI plans to say or do). My life expectancy is another 39 years and I expect to be safely dead before we see anything like Data from Star Trek, let alone a Culture Mind. We’ll see whether I’m right.
posted by Ryvar at 2:59 AM on June 13 [5 favorites]
Briefly: the Q* papers represent an approach that seems likely to successfully brute force some limited ability to correlate real world context and complete complex workloads / perform useful intellectual labor. I agree we have fuck-all in the pipeline for spontaneously spooling up general-case systems models for problem solving in which expert systems successfully predict the behavior of multiple interacting agents (as in: how humans will hypothetically respond to what an AI plans to say or do). My life expectancy is another 39 years and I expect to be safely dead before we see anything like Data from Star Trek, let alone a Culture Mind. We’ll see whether I’m right.
posted by Ryvar at 2:59 AM on June 13 [5 favorites]
Oh, no. I’m totally piggybacking off of your post with my giant comment (I’d apologize but it’s more straight expansion than hijack). That’s the whole brain dump, Leopold’s recent publicity blitz is a fine intro hook, and I came in early enough in the thread I don’t think anybody interested is likely to miss it. Maybe not as well-structured as I’d like, but close enough. Hard to focus when I’ve been awake for over 60 hours but for real closing the tab now.
posted by Ryvar at 3:25 AM on June 13 [6 favorites]
posted by Ryvar at 3:25 AM on June 13 [6 favorites]
I have a degree of confidence that capitalism will enshittify this well before there is any real threat of actual AI. Not that that will make the future any better, mind you, but at least watching some faces get eaten by leopards will provide a bit of morbid comedy
posted by lescour at 3:29 AM on June 13 [6 favorites]
posted by lescour at 3:29 AM on June 13 [6 favorites]
Thanks chavenet for the post, and thanks Ryvar for the bonus material. I've not had remotely enough time to work through all of this, but it's very nice to see that kind of detailed reply right at the start of an in-depth focused AI post.
posted by protorp at 3:43 AM on June 13 [3 favorites]
posted by protorp at 3:43 AM on June 13 [3 favorites]
He can take a substantial amount of credit for us little people having even passable versions of these systems and not being shut out completely, the rat bastard.
That had nothing to do with democratizing access and everything to do with sabotaging Meta's competition.
posted by mhoye at 4:48 AM on June 13 [3 favorites]
That had nothing to do with democratizing access and everything to do with sabotaging Meta's competition.
posted by mhoye at 4:48 AM on June 13 [3 favorites]
Full of acronyms with no definition. CCP = Chinese Communist Party?
posted by Devils Rancher at 4:56 AM on June 13 [3 favorites]
posted by Devils Rancher at 4:56 AM on June 13 [3 favorites]
I saw a post on Mastodon (sorry for the lack of source, but I am pulling this from memory) where someone predicted their baby would weigh a few trillion pounds by age 10 given their current growth rate. Page 8 reminded me very much of that, even ignoring the fact that each and every child I know is a lot smarter than ChatGPT by any standard that isn't "producing vaguely reasonable discourse-shaped text blobs".
posted by Eclectic_Emu at 4:58 AM on June 13 [4 favorites]
posted by Eclectic_Emu at 4:58 AM on June 13 [4 favorites]
what was called AI in the past isnt called AI today. whats called AI today wont be tomorrow. same as it ever was - only now theres trillions of dollars at stake in a pyramid with nvidia at the top.
posted by AlbertCalavicci at 5:06 AM on June 13 [4 favorites]
posted by AlbertCalavicci at 5:06 AM on June 13 [4 favorites]
"A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years."
This seems disingenuous. There have been people saying AI was the next big thing for many many decades, and it never was, until it was. Was there a different group a few years ago who said, "But this time we have trendlines!"
posted by bitslayer at 5:08 AM on June 13 [4 favorites]
This seems disingenuous. There have been people saying AI was the next big thing for many many decades, and it never was, until it was. Was there a different group a few years ago who said, "But this time we have trendlines!"
posted by bitslayer at 5:08 AM on June 13 [4 favorites]
A Grand Unified Theory of the AI Hype Cycle
[this has been posted here before but bears repeating]
posted by chavenet at 5:34 AM on June 13 [11 favorites]
[this has been posted here before but bears repeating]
posted by chavenet at 5:34 AM on June 13 [11 favorites]
Those with more critical distance on AI and venture capital may find Max Read's take on Aschenbrenner useful.
posted by Four String Riot at 5:37 AM on June 13 [9 favorites]
posted by Four String Riot at 5:37 AM on June 13 [9 favorites]
Do you want universal paperclips? Because this is how you get universal paperclips.
posted by emelenjr at 6:19 AM on June 13 [3 favorites]
posted by emelenjr at 6:19 AM on June 13 [3 favorites]
The abstract claiming essentially "we've created high school level intelligence, therefore this trend will continue until we have AGI" screamed to me that the author has gotten so lost in their own niche field that they no longer relate to society at large anymore. This feels like a sufferer of engineer's disease with a better than usual website.
In the last few months I've been pulled into the AI space and while there are practical applications of LLMs, there's also too much of a trend towards pareidolia where the human brain loves to find mirrors of itself in other things. We're essentially pattern recognition machines so of course we're going to think a synthetic pattern matching system is like us.
The discourse around AI in general is frustrating because so much of it is marketing hype, when there is value in the pragmatic application of the technologies (I tend to think of LLMs as a natural language interface that allows a novice user to learn more about a system, when the data is curated and managed properly). Apple's announcements earlier this week give me a little hope that the market is shifting towards stage 11/12 of the Hype Cycle, since they're already themselves at "productive uses are marketed under their application rather than their mechanism."
posted by mrzarquon at 6:25 AM on June 13 [11 favorites]
In the last few months I've been pulled into the AI space and while there are practical applications of LLMs, there's also too much of a trend towards pareidolia where the human brain loves to find mirrors of itself in other things. We're essentially pattern recognition machines so of course we're going to think a synthetic pattern matching system is like us.
The discourse around AI in general is frustrating because so much of it is marketing hype, when there is value in the pragmatic application of the technologies (I tend to think of LLMs as a natural language interface that allows a novice user to learn more about a system, when the data is curated and managed properly). Apple's announcements earlier this week give me a little hope that the market is shifting towards stage 11/12 of the Hype Cycle, since they're already themselves at "productive uses are marketed under their application rather than their mechanism."
posted by mrzarquon at 6:25 AM on June 13 [11 favorites]
to ai hype jockeys (which unfortunately includes many in industry and research), "as smart as a high schooler" has the same meaning as "mostly accurate information retrieval from high schoolers' textbooks and homework assignments".
seriously, people need to begin reflecting a teeny tiny bit and admit how pathetic this whole thing is.
posted by AlbertCalavicci at 6:32 AM on June 13 [9 favorites]
seriously, people need to begin reflecting a teeny tiny bit and admit how pathetic this whole thing is.
posted by AlbertCalavicci at 6:32 AM on June 13 [9 favorites]
oh, this is the guy with the Dumb Graph? lol
posted by BungaDunga at 6:35 AM on June 13 [3 favorites]
posted by BungaDunga at 6:35 AM on June 13 [3 favorites]
Also, the "former $Company employee" prestige thing needs to stop.
Why is this guy getting hype describing him as a former OpenAI engineer? From LinkedIn (unless there's multiple Leopold Aschenbrenners who have worked there) he was at OpenAI for 18 months. He graduated from college at 19, in 2021. He's roughly 22.
posted by mrzarquon at 6:44 AM on June 13 [12 favorites]
Why is this guy getting hype describing him as a former OpenAI engineer? From LinkedIn (unless there's multiple Leopold Aschenbrenners who have worked there) he was at OpenAI for 18 months. He graduated from college at 19, in 2021. He's roughly 22.
posted by mrzarquon at 6:44 AM on June 13 [12 favorites]
Paragraph 4: "AGI is one of many types of artificial intelligence, including narrow artificial intelligence, or ANI, and super artificial intelligence, or breast milk."
Huh.
From the very bottom of the article:
"The English, Chinese, Japanese, Arabic, and French versions are automatically generated by the AI. So there may still be inaccuracies in translating, please always see Indonesian as our main language. "
Super artificial intelligence... but 'narrow artificial intelligence' became ANI, so actually 'artificial super intelligence' or ASI... which is Indonesian for... Yep.
My days of NOT being impressed by AI are certainly coming to a middle.
posted by kikaider01 at 6:46 AM on June 13 [18 favorites]
Huh.
From the very bottom of the article:
"The English, Chinese, Japanese, Arabic, and French versions are automatically generated by the AI. So there may still be inaccuracies in translating, please always see Indonesian as our main language. "
Super artificial intelligence... but 'narrow artificial intelligence' became ANI, so actually 'artificial super intelligence' or ASI... which is Indonesian for... Yep.
My days of NOT being impressed by AI are certainly coming to a middle.
posted by kikaider01 at 6:46 AM on June 13 [18 favorites]
it sure seems like half the OpenAI ex-employee dissidents are saying "look, it's run by an unethical, abusive narcissist" and the other half are saying "they're about to invent AGI but they're not doing it the right way, please subscribe to my newsletter"
posted by BungaDunga at 6:47 AM on June 13 [6 favorites]
posted by BungaDunga at 6:47 AM on June 13 [6 favorites]
> Those with more critical distance on AI and venture capital may find Max Read's take on Aschenbrenner useful.
What I also learned from Max Read's link and should be up in the main summary: Leopold is a former FTX adjacent bitcoin-con man (Effective Altruist at the Future Fund) who jumped ship to OpenAI when that folded, before getting fired from OpenAI.
He's the epitome of the high school kid who read Atlas Shrugged and realized that it was the true path forward - and essentially given an award at 17 for being the best little libertarian.
posted by mrzarquon at 6:58 AM on June 13 [11 favorites]
What I also learned from Max Read's link and should be up in the main summary: Leopold is a former FTX adjacent bitcoin-con man (Effective Altruist at the Future Fund) who jumped ship to OpenAI when that folded, before getting fired from OpenAI.
He's the epitome of the high school kid who read Atlas Shrugged and realized that it was the true path forward - and essentially given an award at 17 for being the best little libertarian.
posted by mrzarquon at 6:58 AM on June 13 [11 favorites]
Max Read:
posted by flabdablet at 7:11 AM on June 13 [2 favorites]
unfortunately nothing on this graph--none of the words, none of the axes, none of the lines--bears even an indirect relationship to an observable, measurable, or quantifiable phenomenon. What is “effective compute”? What is “Smart High Schooler” and how do we know that’s equivalent to, or the best way to describe, GPT-4?Are we, or are we not, a company?
posted by flabdablet at 7:11 AM on June 13 [2 favorites]
Footnote 3 from the Max Read article has this nice little example for anyone who still needs inoculating against the assumption that the latest LLMs' successes, on, say, sample LSAT tests, have much bearing on progress toward real AGI:
Prompt: A farmer stands on the side of a river with a sheep. There is a boat on the riverbank that has room for exactly one person and one sheep. How can the farmer get across with the sheep in the fewest number of trips?posted by nobody at 7:24 AM on June 13 [8 favorites]
ChatGPT 4o response: To solve the problem, the farmer needs to get both himself and the sheep across the river using the boat that can only carry one person and one sheep at a time. Here is the solution in the fewest number of trips:
1. Trip 1: The farmer takes the sheep across the river.
2. Trip 2: The farmer returns to the original side.
3. Trip 3: The farmer crosses the river again with the sheep.
This way the farmer and the sheep both end up on the opposite side of the river in three trips.
Everyone thinks they're exponential until they get sinusoidal.
posted by Richard Saunders at 7:30 AM on June 13 [16 favorites]
posted by Richard Saunders at 7:30 AM on June 13 [16 favorites]
Aschenbrenner:
Intellectual agility is no barrier to delusional hubris. Ray Kurzweil, for instance, is indisputably a very smart guy and he has a longstanding belief in his own potential immortality.
It's nice to see a kid waxing enthusiastic about what he's doing with his life, I guess, but he's given me no reason to believe that AGI is any closer than the ten years away it's been for the last fifty.
I distinctly remember being in third grade and having it suddenly occur to me with the force of revelation that I didn't need to keep going to school because I already knew everything. And I could prove it! I started listing heaps of things I knew, and sure enough, I knew every single one of them!
There's a lot of that attitude on display in Aschenbrenner's 165-page GPT-assisted magnum oplease.
Everyone thinks they're exponential until they get sinusoidal
Yep. Lack of experience with the interplay between real and imaginary will do that.
posted by flabdablet at 7:34 AM on June 13 [9 favorites]
At this point, you may think that I and all the other SF-folk are totally crazy. But consider, just for a moment: what if they’re right? These are the people who invented and built this technology; they think AGI will be developed this decade; and, though there’s a fairly wide spectrum, many of them take very seriously the possibility that the road to superintelligence will play out as I’ve described in this series.I've considered it for rather longer than a moment. My considered opinion: they're not.
Intellectual agility is no barrier to delusional hubris. Ray Kurzweil, for instance, is indisputably a very smart guy and he has a longstanding belief in his own potential immortality.
It's nice to see a kid waxing enthusiastic about what he's doing with his life, I guess, but he's given me no reason to believe that AGI is any closer than the ten years away it's been for the last fifty.
I distinctly remember being in third grade and having it suddenly occur to me with the force of revelation that I didn't need to keep going to school because I already knew everything. And I could prove it! I started listing heaps of things I knew, and sure enough, I knew every single one of them!
There's a lot of that attitude on display in Aschenbrenner's 165-page GPT-assisted magnum oplease.
Everyone thinks they're exponential until they get sinusoidal
Yep. Lack of experience with the interplay between real and imaginary will do that.
posted by flabdablet at 7:34 AM on June 13 [9 favorites]
Why does the farmer return in step 2?
You're right; I made an unnecessary complication. The farmer doesn't need to return alone in step 2. Here is the correct and simplest solution:
- The farmer takes the sheep across the river (1 trip).
So, the fewest number of trips required is just 1:
- Farmer and sheep cross the river.
posted by torokunai at 7:43 AM on June 13 [1 favorite]
At this point, you may think that I and all the other SF-folk are totally crazy. But consider, just for a moment: what if they’re right?This reads exactly like the late 2000s early crypto hype era where there were confident, seemingly-authoritative predictions like this that within 10 years crypto will have supplanted the world's major currencies or some nonsense.
So in 10 years I expect to see a "Ten years in, nobody has come up with a use for blockchain" article for AI, except there will be some uses as GallonOfAlan says ("we still mainly have jazzed up search engines and lots of useful applications in filmmaking and video games").
In fact, any AI proponents should be legally required to disclose what they said about crypto in the early years at the top of any comment on AI.
posted by star gentle uterus at 7:47 AM on June 13 [5 favorites]
if you tell ChatGPT that the step 2 might be unnecessary (in the form of a question, but still), then sure, it turns around and agrees with you, but that is sort of cheating: LLMs are programmed to be agreeable.
posted by BungaDunga at 7:48 AM on June 13 [3 favorites]
posted by BungaDunga at 7:48 AM on June 13 [3 favorites]
if you tell ChatGPT that the step 2 might be unnecessary (in the form of a question, but still), then sure, it turns around and agrees with you, but that is sort of cheating: LLMs are programmed to be agreeable.
> wrong, try again
Apologies for the oversight. Let's try again:
The farmer takes the sheep across the river and leaves it on the other side.
The farmer returns alone to the original side.
The farmer takes the boat across the river, but this time, instead of bringing the sheep back, the farmer brings something else across, let's say a cabbage or a bag of grain.
The farmer leaves the cabbage or grain on the other side with the sheep.
The farmer takes the sheep back across the river.
Finally, the farmer returns alone to the original side.
With this strategy, the farmer makes a total of five trips across the river: two with the sheep and three without.
posted by BungaDunga at 7:51 AM on June 13 [4 favorites]
Exhibit A
> wrong, try again
Apologies for the oversight. Let's try again:
The farmer takes the sheep across the river and leaves it on the other side.
The farmer returns alone to the original side.
The farmer takes the boat across the river, but this time, instead of bringing the sheep back, the farmer brings something else across, let's say a cabbage or a bag of grain.
The farmer leaves the cabbage or grain on the other side with the sheep.
The farmer takes the sheep back across the river.
Finally, the farmer returns alone to the original side.
With this strategy, the farmer makes a total of five trips across the river: two with the sheep and three without.
posted by BungaDunga at 7:51 AM on June 13 [4 favorites]
But you see, now you've got some cabbage and grain, which you didn't have before. So the machine has made things better for us all.
posted by dis_integration at 7:53 AM on June 13 [7 favorites]
posted by dis_integration at 7:53 AM on June 13 [7 favorites]
Exhibit B
>That's even worse.
Let's give it another shot:
The farmer takes the sheep across the river and leaves it on the other side.
The farmer returns alone to the original side.
The farmer takes the boat across the river again, but this time, instead of bringing the sheep back, the farmer brings the boat back to the original side.
The farmer then takes another animal, let's say a wolf (assuming the farmer has a wolf or some other animal on the riverbank), across the river.
The farmer leaves the wolf on the other side and takes the sheep back across the river.
Finally, the farmer returns alone to the original side.
Now, the farmer only makes three trips across the river: two with the sheep and one without.
> no. Think this through step by step. A farmer stands on the side of a river with a sheep.
Got it. Let's break it down:
The farmer takes the sheep across the river.
The farmer leaves the sheep on the other side and returns alone to the original side.
The farmer then takes the boat back across the river.
The farmer now takes the sheep back across the river.
In this sequence, the farmer makes a total of four trips: two with the sheep and two without.
> Justify each step.
Certainly, let's justify each step:
The farmer takes the sheep across the river: This step is necessary to get the sheep to the other side, as the farmer cannot leave the sheep alone on one side of the river.
The farmer leaves the sheep on the other side and returns alone to the original side: This step allows the farmer to come back to the original side to ferry another item across the river, ensuring that the sheep is safe on the opposite bank.
The farmer then takes the boat back across the river: The farmer needs to retrieve the boat from the other side in order to transport either themselves or another item back across the river.
The farmer now takes the sheep back across the river: With the boat back on the original side, the farmer can use it to bring the sheep back to the original side, completing the task.
Each step is necessary for the successful transportation of the sheep across the river, ensuring its safety and efficient use of the boat.
In short, it's very, very easy to "clever Hans" these LLMs into seeming smarter than they are. If you leak information about the solution to them, they will pick up on it and use it. I think they're pretty neat, but they are also very stupid.
posted by BungaDunga at 7:55 AM on June 13 [8 favorites]
But the cabbage and grain are hallucinations... just like the majority of this .pdf I'm reading through in another tab.
posted by Molesome at 7:56 AM on June 13 [2 favorites]
posted by Molesome at 7:56 AM on June 13 [2 favorites]
Why does the farmer return in step 2?
posted by torokunai at 7:43 AM on June 13
To get to the other side.
posted by Kibbutz at 7:57 AM on June 13 [4 favorites]
posted by torokunai at 7:43 AM on June 13
To get to the other side.
posted by Kibbutz at 7:57 AM on June 13 [4 favorites]
Exhibit C
> I want you to explain in detail steps three and four in extreme detail, paying particular attention to anything that doesn't make sense.
Absolutely, let's delve into steps three and four in extreme detail, addressing any potential inconsistencies or illogical elements:
Step 3: The farmer then takes the boat back across the river.
In this step, the farmer, having left the sheep safely on the other side, needs to return the boat to the original side of the river. However, there are a few potential points of confusion or concern:
Boat Availability: It's assumed that the boat is readily available on both sides of the river for the farmer's use. If this is not the case, the farmer would need to find another means to return the boat, which could complicate the solution.
Reason for Returning Alone: Some might question why the farmer returns alone rather than bringing another item back with them. The reason for this is to ensure the sheep's safety on the opposite bank while the boat is being retrieved. Bringing another item back could potentially leave the sheep unattended, which would not be safe.
Consistency with the Rules: This step adheres to the rule that the boat has room for exactly one person and one sheep. By returning alone, the farmer maintains consistency with this rule.
Step 4: The farmer now takes the sheep back across the river.
Once the boat is back on the original side, the farmer can use it to transport the sheep back across the river. Here are some points to consider:
Safety of the Sheep: Bringing the sheep back across ensures its safety and reunites it with the farmer on the original side.
Efficiency: This step completes the task efficiently, as the farmer has already returned the boat to the original side, eliminating the need for additional trips.
Consistency with the Rules: Again, this step adheres to the rule that the boat can carry only one person and one sheep. By bringing the sheep back alone, the farmer complies with this constraint.
Overall, steps three and four ensure the safe and efficient transportation of the sheep across the river while adhering to the rules and constraints provided in the problem statement.
> let's break it down a different way. After step one, 1. where is the farmer, 2. where is the sheep, and 3. what, if anything, needs to be done to complete the task, and why?
Certainly, let's break it down further after step one:
Where is the farmer? After step one, the farmer is on the opposite side of the river from where they started. They have successfully transported the sheep across the river.
Where is the sheep? The sheep is now on the opposite side of the river from where it started. It is with the farmer on the opposite bank.
What needs to be done to complete the task, and why? At this point, the task is not complete because the farmer still needs to return to the original side of the river. This is necessary to ensure that the farmer can use the boat for future trips and maintain control of the situation. If the farmer were to attempt to proceed with the sheep remaining on the opposite side, they would not have access to the boat, which is essential for further transportation across the river. Therefore, the completion of the task requires the farmer to return to the original side of the river.
posted by BungaDunga at 8:10 AM on June 13 [2 favorites]
well, there's always 5
posted by torokunai at 8:11 AM on June 13 [1 favorite]
posted by torokunai at 8:11 AM on June 13 [1 favorite]
anyway gpt4o actually does answer the question correctly in one shot, to be fair
posted by BungaDunga at 8:15 AM on June 13 [1 favorite]
posted by BungaDunga at 8:15 AM on June 13 [1 favorite]
Just earlier today I was thinking about a line that I like from the game Deus Ex Human Revolution:
"It's not the end of the world, but you can see it from here."
posted by slimepuppy at 8:20 AM on June 13 [5 favorites]
"It's not the end of the world, but you can see it from here."
posted by slimepuppy at 8:20 AM on June 13 [5 favorites]
actually even with 4o I got it into an insane loop where it seemed to understand something was wrong, and would keep spitting out "take the sheep over, go back, take the sheep over again. Wait! That's wrong. Take the sheep over, go back, take it over again. Hold on, that's a mistake-" until it ran out of tokens and concluded with the glorious sentences: "No fewer trips consistent ensuring transport aim. Hence accurately reviewed clarification stands trips: confirmed key solution!"
posted by BungaDunga at 8:22 AM on June 13 [4 favorites]
posted by BungaDunga at 8:22 AM on June 13 [4 favorites]
But thanks to its amazingly high parameter count, GPT4o can perform transfer learning to novel tasks… right?
ChatGPT 4o: The astronaut can get to the moon with the sheep in the fewest number of trips by following these steps:
1. **Trip 1**: The astronaut takes the sheep to the moon.
2. **Trip 2**: The astronaut returns to Earth alone.
3. **Trip 3**: The astronaut goes to the moon alone.
In total, this process involves three trips:
1. Earth to Moon (with sheep)
2. Moon to Earth (alone)
3. Earth to Moon (alone)
posted by graphweaver at 8:30 AM on June 13 [5 favorites]
An astronaut and a sheep want to get to the moon
User: An astronaut stands on Earth with a sheep. There is a spaceship near her that has room for exactly one person and one sheep. How can the astronaut get to the moon with the sheep in the fewest number of trips?ChatGPT 4o: The astronaut can get to the moon with the sheep in the fewest number of trips by following these steps:
1. **Trip 1**: The astronaut takes the sheep to the moon.
2. **Trip 2**: The astronaut returns to Earth alone.
3. **Trip 3**: The astronaut goes to the moon alone.
In total, this process involves three trips:
1. Earth to Moon (with sheep)
2. Moon to Earth (alone)
3. Earth to Moon (alone)
posted by graphweaver at 8:30 AM on June 13 [5 favorites]
At least the farmer didn’t leave the sheep alone with the wolf! That was intelligent.
posted by GenjiandProust at 8:41 AM on June 13 [4 favorites]
posted by GenjiandProust at 8:41 AM on June 13 [4 favorites]
If it was a cow instead of a sheep, it could just jump over the moon on its own and the astronaut could pick it up en route. Clearly, further research is required to determine whether the same approach is feasible for ovine as well as bovine payloads.
posted by flabdablet at 8:43 AM on June 13 [8 favorites]
posted by flabdablet at 8:43 AM on June 13 [8 favorites]
I think the machine got confused with the usual expression of this puzzle, with a man, a chicken, a bucket of grain, and a fox. Chicken eats grain, fox eats chicken as rules. As the machine has no overall understanding of the situation as expressed, it’s confused about the number of trips required as in the original expression of the problem. The original problem requires multiple trips, including bringing something back, I think. Despite the enormous, incomprehensible computation going on, these machines are pretty predictable.
posted by njohnson23 at 8:47 AM on June 13 [3 favorites]
posted by njohnson23 at 8:47 AM on June 13 [3 favorites]
Science YouTuber Sabine Hossenfelder has a video up with her reaction. She is skeptical of his predictions. She holds PhD in theoretical physics, so this isn’t her field; but in the last few years she has built her career as a popular science communicator / journalist.
posted by interogative mood at 8:54 AM on June 13 [1 favorite]
posted by interogative mood at 8:54 AM on June 13 [1 favorite]
her bold "centrist" take on trans issues is something of a trash fire, at best she rarely has much to contribute outside her speciality
posted by BungaDunga at 9:03 AM on June 13 [7 favorites]
posted by BungaDunga at 9:03 AM on June 13 [7 favorites]
Aschenbrenner in 2020: We Europeans are overly obsessed with America (which is in the “suffocating” grip of the political Left).
Aschenbrenner in 2024: “You can see the future first in San Francisco.”
posted by mbrubeck at 9:13 AM on June 13 [3 favorites]
Aschenbrenner in 2024: “You can see the future first in San Francisco.”
posted by mbrubeck at 9:13 AM on June 13 [3 favorites]
Via an Apple statement recently:
"Apple believes pushing OpenAI's brand and technology to hundreds of millions of its devices is of equal or greater value than monetary payments"
If you, like me, are on the "this is the sixth tech pump and dump in the last ten years, we have become exceedingly efficient at them*" train, Apple paying $0 for AI so they can have a free hype conference and then rounding errors for $0 of costs (to them) is, uh, a sign.
*Quantum computing**, self drive, metaverse, crypto, NFTs, and now AI
**My personal conspiracy theory is that this was primarily a push to scare folks from cryptographic standards no one can break to new "quantum proof" ones that haven't had decades of poking and prodding performed and maybe have some holes exploitable by various three letter agencies
posted by Slackermagee at 9:28 AM on June 13 [5 favorites]
"Apple believes pushing OpenAI's brand and technology to hundreds of millions of its devices is of equal or greater value than monetary payments"
If you, like me, are on the "this is the sixth tech pump and dump in the last ten years, we have become exceedingly efficient at them*" train, Apple paying $0 for AI so they can have a free hype conference and then rounding errors for $0 of costs (to them) is, uh, a sign.
*Quantum computing**, self drive, metaverse, crypto, NFTs, and now AI
**My personal conspiracy theory is that this was primarily a push to scare folks from cryptographic standards no one can break to new "quantum proof" ones that haven't had decades of poking and prodding performed and maybe have some holes exploitable by various three letter agencies
posted by Slackermagee at 9:28 AM on June 13 [5 favorites]
I think the machine got confused with the usual expression of this puzzle, with a man, a chicken, a bucket of grain, and a fox.
Yes, that is exactly what happened, except that using the word "confused" is anthropomorphizing in a way that LLMs don't deserve. LLMs are not "thinking," they're just predicting next tokens based on what was in their training data (i.e., the Internet) and their tuning (i.e., underpaid people in low cost-of-living countries). There is very little reasoning going on — in fact, the basic structure of LLMs makes any sort of complex reasoning impossible.
The claim that LLMs are "as smart as a high schooler" is fundamentally flawed: it's based on evidence like giving LLMs tests that are meant to measure human intelligence, and we've known that to be flawed since the 1960s, when computers were acing math exams that were difficult for very smart humans and as a result people thought that AGI was just around the corner.
I think LLMs are very good at hitting a particular weakness of human minds, which is that we're not evolved or trained to question the intelligence of anything that can talk. (ELIZA did the same thing, also in the 1960s, though not as well.) The AI image generators use more-or-less the same technology, but (most) people are a lot better at noticing the flaws in the generated images.
posted by reventlov at 9:37 AM on June 13 [14 favorites]
Yes, that is exactly what happened, except that using the word "confused" is anthropomorphizing in a way that LLMs don't deserve. LLMs are not "thinking," they're just predicting next tokens based on what was in their training data (i.e., the Internet) and their tuning (i.e., underpaid people in low cost-of-living countries). There is very little reasoning going on — in fact, the basic structure of LLMs makes any sort of complex reasoning impossible.
The claim that LLMs are "as smart as a high schooler" is fundamentally flawed: it's based on evidence like giving LLMs tests that are meant to measure human intelligence, and we've known that to be flawed since the 1960s, when computers were acing math exams that were difficult for very smart humans and as a result people thought that AGI was just around the corner.
I think LLMs are very good at hitting a particular weakness of human minds, which is that we're not evolved or trained to question the intelligence of anything that can talk. (ELIZA did the same thing, also in the 1960s, though not as well.) The AI image generators use more-or-less the same technology, but (most) people are a lot better at noticing the flaws in the generated images.
posted by reventlov at 9:37 AM on June 13 [14 favorites]
This feels like a sufferer of engineer's disease with a better than usual website.Though sort of an inverse case, since he’s an econ student / venture capital guy opining (badly) about engineering topics.
posted by mbrubeck at 9:38 AM on June 13 [5 favorites]
I think LLMs are very good at hitting a particular weakness of human minds, which is that we're not evolved or trained to question the intelligence of anything that can talk. (ELIZA did the same thing, also in the 1960s, though not as well.)
“What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.” --Joseph Weizenbaum [inventor of ELIZA]
posted by chavenet at 10:00 AM on June 13 [5 favorites]
“What I had not realized is that extremely short exposures to a relatively simple computer program could induce powerful delusional thinking in quite normal people.” --Joseph Weizenbaum [inventor of ELIZA]
posted by chavenet at 10:00 AM on June 13 [5 favorites]
I first encountered ELIZA running on a Heathkit H8 at some kind of trade show when I was about Aschenbrenner's present age and was completely fascinated by it. Seemed pretty clear to me at the time that artificial intelligence (which didn't need the "general" qualifier then, marketroids never having got interested in it) was about ten years away.
posted by flabdablet at 10:15 AM on June 13 [2 favorites]
posted by flabdablet at 10:15 AM on June 13 [2 favorites]
I think Aschenbrenner is suffering from Market Bubble Disease, which is closely related but distinct from Engineers disease. When an engineer suffers from both at the same time, you get Elon Musk disorder. Sadly incurable with current medical technology.
posted by dis_integration at 10:32 AM on June 13 [5 favorites]
posted by dis_integration at 10:32 AM on June 13 [5 favorites]
We're essentially pattern recognition machines so of course we're going to think a synthetic pattern matching system is like us.
All human neural networks are overfitted for recognizing hominid intent: faces in clouds, Jesus in toast, it never ends.
posted by Ryvar at 10:48 AM on June 13 [3 favorites]
All human neural networks are overfitted for recognizing hominid intent: faces in clouds, Jesus in toast, it never ends.
posted by Ryvar at 10:48 AM on June 13 [3 favorites]
I read through Aschenbrenner's piece, and one thing he kept coming back to was the Manhattan Project. If it was the early 1940's, even if I followed nuclear physics, there's no way I'd see a bomb arriving in 1945. It's too big of a leap from theory to reality. It would be quite a shock if they manage to make an artificial superintelligence by the end of the decade, but I can see the military doing a lot of crazy things very quickly if they think it's technically possible.
With the examples of artificial stupidity above (thanks for figuring out the "breast milk" thing: I just said "that's enough of that article" when I got to it), it seems like the best bet is that we're a long, long ways off from any sort of AGI. Maybe they'll be a little more useful by the end of the decade, but I don't expect much more than that.
Still, there's some weird things happening. Look at the semiconductor export controls the US and allies put on China starting in late 2022: I don't think I could have imagined them short of a war before they happened. US military spending on AI increased by 12x from 2022 to 2023 to about $4.5 billion. That's quite a sudden change, and it's a decent chunk of the 2023 total US AI investments of about $68 billion.
posted by netowl at 10:58 AM on June 13 [3 favorites]
With the examples of artificial stupidity above (thanks for figuring out the "breast milk" thing: I just said "that's enough of that article" when I got to it), it seems like the best bet is that we're a long, long ways off from any sort of AGI. Maybe they'll be a little more useful by the end of the decade, but I don't expect much more than that.
Still, there's some weird things happening. Look at the semiconductor export controls the US and allies put on China starting in late 2022: I don't think I could have imagined them short of a war before they happened. US military spending on AI increased by 12x from 2022 to 2023 to about $4.5 billion. That's quite a sudden change, and it's a decent chunk of the 2023 total US AI investments of about $68 billion.
posted by netowl at 10:58 AM on June 13 [3 favorites]
Exhibit D:
If AI is going to take over the world, why can't it solve the spelling bee?
posted by Bigbootay. Tay! Tay! Blam! Aargh... at 11:19 AM on June 13 [2 favorites]
If AI is going to take over the world, why can't it solve the spelling bee?
posted by Bigbootay. Tay! Tay! Blam! Aargh... at 11:19 AM on June 13 [2 favorites]
An important distinction between the atom bomb and Artificial General Intelligence is that theoretical foundations of the atom bomb were pretty well established by the 1940s. We knew that uranium would release tremendous amounts of energy when bombarded with neutrons and it was quickly perceived in the 1930s that this could be used to build a bomb. We didn't know how to build that bomb, or exactly how powerful it would be, but the basic idea was there. We have no basic idea of how to turn a football field of NVIDIA GPUs into a genuine thinking machine. We just don't know what will make it happen. What *has* happened here is that OpenAI thought: well what if we take existing technology and just do it on a massive scale. And the results have been pretty remarkable. So the theory is, well just scale it up again and then again and eventually you'll get Skynet, but there's no real theory here as to why it would make the leap.
posted by dis_integration at 11:23 AM on June 13 [4 favorites]
posted by dis_integration at 11:23 AM on June 13 [4 favorites]
> There is very little reasoning going on — in fact, the basic structure of LLMs makes any sort of complex reasoning impossible.
Trying to explain to people that LLMs can't ever generate a new idea is something I've found myself doing with more frequency. Anything originally assumed as novel ends up being either obscure data (like the Glue Pizza recipe) or errors strung together with five dollar words to sound impressive.
The Farmer/Sheep question is a great illustration of this - because it's exposing the pattern matching that is happening underneath it. ChatGPT didn't have this specific question in it's model, but it had a similar one, so used the answers from that to complete it while trying to make it conform to the other attributes of the data it was parsing. It's not hallucinating when it gets told it is wrong, it's just compounding the error because it is then feeding its original prompt, the output, and the user response back into the model, which sometimes if the user response is adding new data, essentially tacts on a new branch to the logic tree and it comes out with a satisfactory answer for the user. If the user response doesn't include the answer, it isn't confused, it just starts picking the probablisticly less likely responses and uses those, which is when things like the Wolf show as they are pulled in from a bigger and bigger search across the model.
posted by mrzarquon at 11:32 AM on June 13 [3 favorites]
Trying to explain to people that LLMs can't ever generate a new idea is something I've found myself doing with more frequency. Anything originally assumed as novel ends up being either obscure data (like the Glue Pizza recipe) or errors strung together with five dollar words to sound impressive.
The Farmer/Sheep question is a great illustration of this - because it's exposing the pattern matching that is happening underneath it. ChatGPT didn't have this specific question in it's model, but it had a similar one, so used the answers from that to complete it while trying to make it conform to the other attributes of the data it was parsing. It's not hallucinating when it gets told it is wrong, it's just compounding the error because it is then feeding its original prompt, the output, and the user response back into the model, which sometimes if the user response is adding new data, essentially tacts on a new branch to the logic tree and it comes out with a satisfactory answer for the user. If the user response doesn't include the answer, it isn't confused, it just starts picking the probablisticly less likely responses and uses those, which is when things like the Wolf show as they are pulled in from a bigger and bigger search across the model.
posted by mrzarquon at 11:32 AM on June 13 [3 favorites]
There is very little reasoning going on — in fact, the basic structure of LLMs makes any sort of complex reasoning impossible.
The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con
posted by chavenet at 11:39 AM on June 13 [3 favorites]
The LLMentalist Effect: how chat-based Large Language Models replicate the mechanisms of a psychic’s con
posted by chavenet at 11:39 AM on June 13 [3 favorites]
There's exactly one intelligence in the world that we're aware of, and it's us. We don't know how consciousness works and even with decades of research our understanding of how our brains work is still comparatively pretty weak. We don't understand intelligence- what it is, how it works, how it emerges- but we think we're going to make minds using materials that aren't even used in the one intelligence we know and can actually communicate with? Come the fuck on. Artificial Intelligence, and especially artificial intelligence in the foreseeable future, is as much a fantasy as elves and wizards.
It should be noted here that most of these interviews are using OpenAI’s internal definition of AGI, which according to CEO Sam Altman roughly translates to “systems which can replace >50% of office workers,” not necessarily Data from Star Trek AI or superhuman intelligence
The thing is, AI is already a concept in our society, and Data- a thinking machine- is it. That's why generative "AI" is being marketed that way- because Silicon Valley types either think you're stupid enough to be fooled by LLM output into believing that there's a sapient machine on the other end or because they're that stupid themselves. There's no reason to refer to any existing technology, nor any technology which will flow from these auto-corrects, as AI other than to deceive people about the nature and capabilities of the technology in order to scam them out of money.
posted by Pope Guilty at 1:01 PM on June 13 [2 favorites]
It should be noted here that most of these interviews are using OpenAI’s internal definition of AGI, which according to CEO Sam Altman roughly translates to “systems which can replace >50% of office workers,” not necessarily Data from Star Trek AI or superhuman intelligence
The thing is, AI is already a concept in our society, and Data- a thinking machine- is it. That's why generative "AI" is being marketed that way- because Silicon Valley types either think you're stupid enough to be fooled by LLM output into believing that there's a sapient machine on the other end or because they're that stupid themselves. There's no reason to refer to any existing technology, nor any technology which will flow from these auto-corrects, as AI other than to deceive people about the nature and capabilities of the technology in order to scam them out of money.
posted by Pope Guilty at 1:01 PM on June 13 [2 favorites]
The Farmer/Sheep question is a great illustration of this - because it's exposing the pattern matching that is happening underneath it.
To be fair it’s pretty easy to nail people on lazy pattern matching, too - if it takes five machines five minutes to make five widgets, how long does it take 100 machines to make 100 widgets?
But we are better at reflection and adjustment and not getting stuck in a loop on that kind of thing. One thing I’ve realized over time is that ChatGPT’s “I’m sorry, you’re correct, that was a wrong answer, let me try again” responses are fundamentally a very bad way to iterate towards a better solution, as opposed to actually honing in on and seeking to clarify the details that are wrong.
posted by atoxyl at 1:11 PM on June 13
To be fair it’s pretty easy to nail people on lazy pattern matching, too - if it takes five machines five minutes to make five widgets, how long does it take 100 machines to make 100 widgets?
But we are better at reflection and adjustment and not getting stuck in a loop on that kind of thing. One thing I’ve realized over time is that ChatGPT’s “I’m sorry, you’re correct, that was a wrong answer, let me try again” responses are fundamentally a very bad way to iterate towards a better solution, as opposed to actually honing in on and seeking to clarify the details that are wrong.
posted by atoxyl at 1:11 PM on June 13
This farmer/chicken rowboat example is hilarious because the "artificial intelligence" is trying to answer the (tricky) question that's the core of the classic puzzle (involving foxes & bags of grain & multiple trips) and therefore can't see the simple answer to the simple question actually being asked, which most any child could do.
posted by chavenet at 1:15 PM on June 13 [3 favorites]
posted by chavenet at 1:15 PM on June 13 [3 favorites]
If AI is going to take over the world, why can't it solve the spelling bee?
This piece almost explains it, but the reason LLMs can't spell is that LLMs don't know how anything is spelled, words are chunked into larger tokens (which it does mention). This means that it has no way to learn that "the" is spelled "T H E" unless it's seen that spelled out somewhere. An LLM trained on one-token-per-letter would probably be able to spell, or at least it would have a fighting chance. But even a hypothetical super duper LLM with human-level reasoning skills would still be essentially "illiterate."
posted by BungaDunga at 1:23 PM on June 13 [4 favorites]
This piece almost explains it, but the reason LLMs can't spell is that LLMs don't know how anything is spelled, words are chunked into larger tokens (which it does mention). This means that it has no way to learn that "the" is spelled "T H E" unless it's seen that spelled out somewhere. An LLM trained on one-token-per-letter would probably be able to spell, or at least it would have a fighting chance. But even a hypothetical super duper LLM with human-level reasoning skills would still be essentially "illiterate."
posted by BungaDunga at 1:23 PM on June 13 [4 favorites]
that is, you have to imagine ChatGPT as approximately someone who can't read, and is communicating to you via speech-to-text and text-to-speech, because that's (sort of) what the tokenization is doing. That doesn't really say anything about the underlying intelligence of the model; lots of illiterate people are very intelligent. ChatGPT is not very intelligent, but for different reasons.
posted by BungaDunga at 1:26 PM on June 13 [2 favorites]
posted by BungaDunga at 1:26 PM on June 13 [2 favorites]
Ray Kurzweil, for instance, is indisputably a very smart guy and he has a longstanding belief in his own potential immortality.
This paper is kind of a florid example of the classic Kurzweil syndrome of putting real numbers on one axis of a graph and arbitrary milestones on another.
I will admit I take these kinds of people a little more seriously than I used to because I think they got it right on scaling in a way that a lot of folks didn’t. And somebody is going to scale it another few orders of magnitude at least, and I really don’t know what’s going to happen then. But that’s ultimately the strongest argument that’s made here - somebody is going to scale this a whole lot more, and we haven’t observed clearly diminishing returns to scale yet, so it is plausible that something will happen.
posted by atoxyl at 1:27 PM on June 13 [3 favorites]
This paper is kind of a florid example of the classic Kurzweil syndrome of putting real numbers on one axis of a graph and arbitrary milestones on another.
I will admit I take these kinds of people a little more seriously than I used to because I think they got it right on scaling in a way that a lot of folks didn’t. And somebody is going to scale it another few orders of magnitude at least, and I really don’t know what’s going to happen then. But that’s ultimately the strongest argument that’s made here - somebody is going to scale this a whole lot more, and we haven’t observed clearly diminishing returns to scale yet, so it is plausible that something will happen.
posted by atoxyl at 1:27 PM on June 13 [3 favorites]
One thing I’ve realized over time is that ChatGPT’s “I’m sorry, you’re correct, that was a wrong answer, let me try again” responses are fundamentally a very bad way to iterate towards a better solution, as opposed to actually honing in on and seeking to clarify the details that are wrong.
It doesn't have that option- ChatGPT doesn't know what anything is, or that anything exists, or how anything relates other than knowing that certain words commonly appear near each other in certain patterns. It's not accepting that it's wrong and correcting anything. It's just running the predictive algorithm again and seeing if you like the outcome better, like rerunning an image-generating "AI" with the same prompt in hopes that the next one will be more like what the user wants. The "details that are wrong" are not in any way different, inside the LLM, from a set of equivalent details that would be right.
This is why you can't actually eliminate "hallucinations"- because it is literally not a thing that can be made to distinguish between fact and fiction, truth or lies.
posted by Pope Guilty at 1:41 PM on June 13 [10 favorites]
It doesn't have that option- ChatGPT doesn't know what anything is, or that anything exists, or how anything relates other than knowing that certain words commonly appear near each other in certain patterns. It's not accepting that it's wrong and correcting anything. It's just running the predictive algorithm again and seeing if you like the outcome better, like rerunning an image-generating "AI" with the same prompt in hopes that the next one will be more like what the user wants. The "details that are wrong" are not in any way different, inside the LLM, from a set of equivalent details that would be right.
This is why you can't actually eliminate "hallucinations"- because it is literally not a thing that can be made to distinguish between fact and fiction, truth or lies.
posted by Pope Guilty at 1:41 PM on June 13 [10 favorites]
Cherry-picking some of the worst arguments here but
On compute bottlenecks:
His argument is that the reason is that algorithmic progress is compute-bottlenecked: the academics just don’t have enough compute.
Response:
Academics work on the wrong things. Up until very recently (and perhaps still today?), the vast majority of the academic ML community wasn’t even working on large language models.
Isn’t the historical reason that large language models didn’t come out of academia precisely that academics didn’t have the resources to experiment with making models arbitrarily large? If you want to argue that we are better now at guessing when things that work a little bit at small scale might work well at large scale I don’t really know enough to dispute that, but this counterpoint doesn’t make any sense to me in light of the position he’s trying to defend.
posted by atoxyl at 1:41 PM on June 13 [1 favorite]
On compute bottlenecks:
His argument is that the reason is that algorithmic progress is compute-bottlenecked: the academics just don’t have enough compute.
Response:
Academics work on the wrong things. Up until very recently (and perhaps still today?), the vast majority of the academic ML community wasn’t even working on large language models.
Isn’t the historical reason that large language models didn’t come out of academia precisely that academics didn’t have the resources to experiment with making models arbitrarily large? If you want to argue that we are better now at guessing when things that work a little bit at small scale might work well at large scale I don’t really know enough to dispute that, but this counterpoint doesn’t make any sense to me in light of the position he’s trying to defend.
posted by atoxyl at 1:41 PM on June 13 [1 favorite]
Richard Saunders:
flabdablet:
*slow clap*
That is a solid math burn, well done both of you.
posted by biogeo at 3:37 PM on June 13 [4 favorites]
Everyone thinks they're exponential until they get sinusoidal
flabdablet:
Yep. Lack of experience with the interplay between real and imaginary will do that.
*slow clap*
That is a solid math burn, well done both of you.
posted by biogeo at 3:37 PM on June 13 [4 favorites]
Don't worry folks, problem solved:
The odds of large language models (LLMs) achieving artificial general intelligence (AGI) are a topic of significant debate among researchers, with opinions varying widely based on different interpretations of AGI, the current capabilities of LLMs, and the trajectory of AI research. Here are some key points to consider:
Definitions and Current Capabilities
Artificial General Intelligence (AGI): AGI refers to a machine's ability to understand, learn, and apply knowledge across a wide range of tasks at a level comparable to or exceeding that of humans. Unlike narrow AI, which is designed for specific tasks, AGI would possess general cognitive abilities.
Large Language Models (LLMs): Current LLMs like GPT-4, while impressive in natural language understanding and generation, are fundamentally pattern recognizers trained on vast datasets. They excel at generating text, answering questions, and even performing some reasoning tasks but are not capable of the broad, adaptable, and deep understanding characteristic of AGI.
Arguments for LLMs Achieving AGI
Scaling Hypothesis: Some researchers believe that continuously scaling up the size of LLMs (more parameters, more data) will eventually lead to AGI. They argue that as models become more complex, they might begin to exhibit more generalized intelligence.
Emergent Abilities: As LLMs grow, they have demonstrated emergent abilities—skills and behaviors that were not explicitly programmed but arise from the complexity of the model. These abilities suggest a potential pathway to more generalized intelligence.
Integration with Other Technologies: Combining LLMs with other AI technologies (like reinforcement learning, symbolic reasoning, and perception systems) might bridge the gap to AGI. The synergy of different approaches could create more versatile and intelligent systems.
Arguments Against LLMs Achieving AGI
Lack of True Understanding: LLMs do not truly understand the content they process. They lack consciousness, intentionality, and the ability to engage in complex forms of reasoning that require a deep understanding of the world.
Task-Specific Optimization: LLMs are optimized for specific tasks they were trained on, and their performance degrades significantly outside these contexts. AGI would require a more profound and adaptable understanding of various domains.
Missing Aspects of Human Intelligence: Human intelligence encompasses emotional understanding, ethical reasoning, and common sense, which LLMs currently do not possess. Achieving AGI would likely require breakthroughs in these areas.
Expert Opinions
Optimistic Views: Some AI researchers, including those involved in the development of LLMs, are optimistic that with enough advancements, these models could contribute to achieving AGI or become a significant part of AGI systems.
Skeptical Views: Others argue that LLMs, no matter how advanced, are not the right path to AGI. They suggest that fundamentally different approaches are required, focusing on aspects like embodiment, lifelong learning, and integrating cognitive architectures.
Conclusion
The odds of LLMs achieving AGI are uncertain and heavily debated. While there is potential in the continued development and scaling of LLMs, achieving true AGI might require novel approaches and significant breakthroughs beyond current methodologies. The future of AI research will likely involve a combination of scaling existing models and exploring new paradigms to address the complex challenges of AGI.
posted by Reasonably Everything Happens at 3:53 PM on June 13 [1 favorite]
The odds of large language models (LLMs) achieving artificial general intelligence (AGI) are a topic of significant debate among researchers, with opinions varying widely based on different interpretations of AGI, the current capabilities of LLMs, and the trajectory of AI research. Here are some key points to consider:
Definitions and Current Capabilities
Artificial General Intelligence (AGI): AGI refers to a machine's ability to understand, learn, and apply knowledge across a wide range of tasks at a level comparable to or exceeding that of humans. Unlike narrow AI, which is designed for specific tasks, AGI would possess general cognitive abilities.
Large Language Models (LLMs): Current LLMs like GPT-4, while impressive in natural language understanding and generation, are fundamentally pattern recognizers trained on vast datasets. They excel at generating text, answering questions, and even performing some reasoning tasks but are not capable of the broad, adaptable, and deep understanding characteristic of AGI.
Arguments for LLMs Achieving AGI
Scaling Hypothesis: Some researchers believe that continuously scaling up the size of LLMs (more parameters, more data) will eventually lead to AGI. They argue that as models become more complex, they might begin to exhibit more generalized intelligence.
Emergent Abilities: As LLMs grow, they have demonstrated emergent abilities—skills and behaviors that were not explicitly programmed but arise from the complexity of the model. These abilities suggest a potential pathway to more generalized intelligence.
Integration with Other Technologies: Combining LLMs with other AI technologies (like reinforcement learning, symbolic reasoning, and perception systems) might bridge the gap to AGI. The synergy of different approaches could create more versatile and intelligent systems.
Arguments Against LLMs Achieving AGI
Lack of True Understanding: LLMs do not truly understand the content they process. They lack consciousness, intentionality, and the ability to engage in complex forms of reasoning that require a deep understanding of the world.
Task-Specific Optimization: LLMs are optimized for specific tasks they were trained on, and their performance degrades significantly outside these contexts. AGI would require a more profound and adaptable understanding of various domains.
Missing Aspects of Human Intelligence: Human intelligence encompasses emotional understanding, ethical reasoning, and common sense, which LLMs currently do not possess. Achieving AGI would likely require breakthroughs in these areas.
Expert Opinions
Optimistic Views: Some AI researchers, including those involved in the development of LLMs, are optimistic that with enough advancements, these models could contribute to achieving AGI or become a significant part of AGI systems.
Skeptical Views: Others argue that LLMs, no matter how advanced, are not the right path to AGI. They suggest that fundamentally different approaches are required, focusing on aspects like embodiment, lifelong learning, and integrating cognitive architectures.
Conclusion
The odds of LLMs achieving AGI are uncertain and heavily debated. While there is potential in the continued development and scaling of LLMs, achieving true AGI might require novel approaches and significant breakthroughs beyond current methodologies. The future of AI research will likely involve a combination of scaling existing models and exploring new paradigms to address the complex challenges of AGI.
posted by Reasonably Everything Happens at 3:53 PM on June 13 [1 favorite]
I kind of like that Noema article that's the last link of the post (The Danger Of Superhuman AI Is Not What You Think).
It lays out the ableist and eugenicist foundations of the arguments that AGI proponents espouse, without saying it out loud. I guess there's a place for that kind of argument, but I'm over here saying, "Timnit Gebru or GTFO".
posted by ursus_comiter at 4:21 PM on June 13 [4 favorites]
It lays out the ableist and eugenicist foundations of the arguments that AGI proponents espouse, without saying it out loud. I guess there's a place for that kind of argument, but I'm over here saying, "Timnit Gebru or GTFO".
posted by ursus_comiter at 4:21 PM on June 13 [4 favorites]
If whatever it is that adds on to the primarily pattern-recognition based abilities of brains in general to form human “intelligence” is so profound and special and irreducible, why did it require such a relatively tiny proportion of evolutionary development time to appear, and why is such a relatively tiny proportion of DNA required to code for it? Sometimes I wonder if Metafilter is overrun by closet creationists with all the special pleading that goes on about intelligence.
posted by lastobelus at 6:14 PM on June 13 [1 favorite]
posted by lastobelus at 6:14 PM on June 13 [1 favorite]
You're repeating the same thing that the guy in the Noema piece is: there is a wide gap between "humans aren't that special, we can find a way to simulate that in silicon" and "humans are basically token prediction machines and nothing else." The latter is an insanely strong claim that a bunch of AI boosters seem to believe, because they've gotten wrapped up in the LLM enthusiasm.
posted by BungaDunga at 6:27 PM on June 13 [2 favorites]
posted by BungaDunga at 6:27 PM on June 13 [2 favorites]
Sabine Hossenfelder explains everything you need to know about him in the first 2 mins of her video, let alone the rest of the video.
Is the Intelligence-Explosion Near? A Reality Check.
He's started an investment firm in AIG, and is in his 20s (brief gig with OpenAI).
posted by FranzKanner at 6:52 PM on June 13 [1 favorite]
Is the Intelligence-Explosion Near? A Reality Check.
He's started an investment firm in AIG, and is in his 20s (brief gig with OpenAI).
posted by FranzKanner at 6:52 PM on June 13 [1 favorite]
to expand: the author of the piece asks Bengio (approximately) "how can something like an LLM, which is a pure token-prediction machine, ever be superhuman, when humans manifestly aren't token-prediction machines" and he responds with (exactly) "Why would a machine that works on silicon not be able to perform any of the computations that our brain does?"
Which is answering a different question: there are lots of computations silicon can do. What evidence is there that all you need for superhuman intelligence is token prediction specifically? Because the best LLMs in the world behave extremely unlike humans. They are so strange that some researchers have taken to the "Shoggoth hypothesis", that LLMs are just a shiny facade on top of a Lovecraftian horror that bears no resemblance to human cognition.
posted by BungaDunga at 6:57 PM on June 13 [1 favorite]
Which is answering a different question: there are lots of computations silicon can do. What evidence is there that all you need for superhuman intelligence is token prediction specifically? Because the best LLMs in the world behave extremely unlike humans. They are so strange that some researchers have taken to the "Shoggoth hypothesis", that LLMs are just a shiny facade on top of a Lovecraftian horror that bears no resemblance to human cognition.
posted by BungaDunga at 6:57 PM on June 13 [1 favorite]
If[...]“intelligence” is so profound and special and irreducible why did it require such a relatively tiny proportion of evolutionary development time to appear, and[...]
Wait, are you saying octopuses aren't intelligent, or is the tiny proportion of evolutionary development time starting much earlier than your comment at first glance suggests?
(Also, wait -- is there generally a correlation between profundity/specialness/irreducibility and duration of evolutionary development, anyway? Open question, this one.)
posted by nobody at 7:02 PM on June 13 [1 favorite]
Wait, are you saying octopuses aren't intelligent, or is the tiny proportion of evolutionary development time starting much earlier than your comment at first glance suggests?
(Also, wait -- is there generally a correlation between profundity/specialness/irreducibility and duration of evolutionary development, anyway? Open question, this one.)
posted by nobody at 7:02 PM on June 13 [1 favorite]
That argument based on evolution is specious, pun completely intended. It's not that there's some tiny thing that separates humans from other animals that AI researchers just need to solve to elevate current technology to AGI, it's that current AI systems are still way dumber than most, maybe all, animals, when it comes to the things that would actually move towards AGI, like problem solving, dynamic motivational states, insight learning, and so on. I think AI systems could get there but it's pretty clear that "just throw more data at a transformer network architecture" isn't making much progress, probably because that's not the right architecture for those problems. An AGI that can reason as well as a dog can still doesn't seem to be on the horizon from what's currently being demonstrated anyway.
Arguably, what current AI technology demonstrates is what you get when you can do only the one thing that separates humans from other animals well -- that is, language -- without any of the other sophisticated cognitive machinery that evolved over hundreds of millions of years in humans and other complex-brained animals.
posted by biogeo at 7:42 PM on June 13 [6 favorites]
Arguably, what current AI technology demonstrates is what you get when you can do only the one thing that separates humans from other animals well -- that is, language -- without any of the other sophisticated cognitive machinery that evolved over hundreds of millions of years in humans and other complex-brained animals.
posted by biogeo at 7:42 PM on June 13 [6 favorites]
I’m not sure how my general skepticism for the specialness of human intelligence would lead one to the assumption I believe other organisms have NO intelligence.
I’m confused by the idea we’re approaching the limits of scale with LLMs—aren’t we still a few orders of magnitude short of the density of organic neural networks?
posted by lastobelus at 7:47 PM on June 13 [1 favorite]
I’m confused by the idea we’re approaching the limits of scale with LLMs—aren’t we still a few orders of magnitude short of the density of organic neural networks?
posted by lastobelus at 7:47 PM on June 13 [1 favorite]
dis_integration: I think Aschenbrenner is suffering from Market Bubble Disease, which is closely related but distinct from Engineers disease. When an engineer suffers from both at the same time, you get Elon Musk disorder. Sadly incurable with current medical technology.
Medical technology may not be able to help (I certainly don't expect a vaccine), but a few well-chosen taxes will at least mitigate the amount of damage people can do while suffering from these conditions. (And provide funds for handling some of the damage already done.) Public health, and education may be able to do something as well, at least for future generations.
posted by johnabbe at 8:05 PM on June 13 [4 favorites]
Medical technology may not be able to help (I certainly don't expect a vaccine), but a few well-chosen taxes will at least mitigate the amount of damage people can do while suffering from these conditions. (And provide funds for handling some of the damage already done.) Public health, and education may be able to do something as well, at least for future generations.
posted by johnabbe at 8:05 PM on June 13 [4 favorites]
I’m confused by the idea we’re approaching the limits of scale with LLMs...
We're already throwing the entire internet (~30 trillion tokens) at them, and repeated training on the same bulk data doesn't work well. Obviously we can't just create new multi-trillion-token training sets, so we have to switch to high-quality training somehow. The LLM's aren't orders of magnitude short: humans have about 100 billion neurons, and GPT-3 was 175 billion parameters after training. Obviously organic neurons train faster, have more connections, etc, but they're on the scale of organic neural networks.
posted by netowl at 8:07 PM on June 13 [3 favorites]
We're already throwing the entire internet (~30 trillion tokens) at them, and repeated training on the same bulk data doesn't work well. Obviously we can't just create new multi-trillion-token training sets, so we have to switch to high-quality training somehow. The LLM's aren't orders of magnitude short: humans have about 100 billion neurons, and GPT-3 was 175 billion parameters after training. Obviously organic neurons train faster, have more connections, etc, but they're on the scale of organic neural networks.
posted by netowl at 8:07 PM on June 13 [3 favorites]
how can something like an LLM, which is a pure token-prediction machine, ever be superhuman, when humans manifestly aren't token-prediction machines
I agree that the leap to “of course humans are just token prediction machines” is a silly one but I don’t think “ML model X is just a token prediction machine, therefore it can never do thing Y [that humans do]” is a particularly good argument, either. In most cases we don’t even know how humans do thing Y, let alone have a specific reason to believe that it cannot be done through alternative means.
posted by atoxyl at 8:41 PM on June 13 [3 favorites]
I agree that the leap to “of course humans are just token prediction machines” is a silly one but I don’t think “ML model X is just a token prediction machine, therefore it can never do thing Y [that humans do]” is a particularly good argument, either. In most cases we don’t even know how humans do thing Y, let alone have a specific reason to believe that it cannot be done through alternative means.
posted by atoxyl at 8:41 PM on June 13 [3 favorites]
Not quite the same scale yet, I'd say. A parameter in an artificial neural network describes either a connection or an intrinsic bias, so a more comparable number for human brains would be the number of synapses, which is between 1000-10000 per neuron, so GPT-3 lags at least 3-4 orders of magnitude behind the human brain on that metric. But human neurons are also exquisitely more complex than the "neurons" in an ANN model. Arguably there are dozens of "parameters" per synapse which are important for modeling computational function, plus dozens or hundreds of "parameters" per neuron that describe how it dynamically changes during activation, under the influence of neuromodulators, in response to metabolic challenge, and other factors which we know are critical for understanding how real neurons work and something that few, if any, common ANN architectures even attempt to model, as far as I understand.
But to the point of whether it's really just about the number of parameters, I'd argue that today, even mice still show more signs of key factors for "general intelligence" (problem solving, motivated action, etc) than any "AI" system does. Mice aren't very smart, but they still know things about their their physical and social environments, can solve problems and puzzles, and make informed decisions about how to achieve goals under constraints. Current "AI", at least the type that is currently everywhere in tech, doesn't seem to be able to do this. The mouse brain has about 71 million neurons, though several times more synapses per neuron than primates do. Estimating around 10000 synapses per mouse neuron, then, takes us to around 700 billion "parameters", which is only a few times greater than the number cited for GPT-3. But these LLMs don't seem to be capable of anything close to what a mouse can do in terms of "general intelligence" tasks -- because that's not what they're designed to do. What LLMs are capable of is pretty astonishing considering how simple they are, at their core. But I think there's good reason to think that the kinds of general intelligence tasks that come naturally to animals may require very different architecture -- possibly recurrent and dynamic, like real brains are.
posted by biogeo at 9:12 PM on June 13 [8 favorites]
But to the point of whether it's really just about the number of parameters, I'd argue that today, even mice still show more signs of key factors for "general intelligence" (problem solving, motivated action, etc) than any "AI" system does. Mice aren't very smart, but they still know things about their their physical and social environments, can solve problems and puzzles, and make informed decisions about how to achieve goals under constraints. Current "AI", at least the type that is currently everywhere in tech, doesn't seem to be able to do this. The mouse brain has about 71 million neurons, though several times more synapses per neuron than primates do. Estimating around 10000 synapses per mouse neuron, then, takes us to around 700 billion "parameters", which is only a few times greater than the number cited for GPT-3. But these LLMs don't seem to be capable of anything close to what a mouse can do in terms of "general intelligence" tasks -- because that's not what they're designed to do. What LLMs are capable of is pretty astonishing considering how simple they are, at their core. But I think there's good reason to think that the kinds of general intelligence tasks that come naturally to animals may require very different architecture -- possibly recurrent and dynamic, like real brains are.
posted by biogeo at 9:12 PM on June 13 [8 favorites]
I think the strongest “it can’t possibly do that” predictions given current hot ML architectures are definitely ones related to longer-horizon learning and planning, because current architectures are pre-trained and operate in a fixed context window (although it has become possible to get a really big context window).
posted by atoxyl at 10:02 PM on June 13 [2 favorites]
posted by atoxyl at 10:02 PM on June 13 [2 favorites]
somebody is going to scale it another few orders of magnitude at least, and I really don’t know what’s going to happen then.
Given the titanic amount of marketing they already require to show any sign of being able to make an actual return on investment as opposed to merely attracting new speculators, large language models becoming another few OOM more expensive might be enough to kick us out of this fairly sticky local design optimum briskly enough to avoid making global warming significantly worse.
But we are better at reflection and adjustment and not getting stuck in a loop on that kind of thing. One thing I’ve realized over time is that ChatGPT’s “I’m sorry, you’re correct, that was a wrong answer, let me try again” responses are fundamentally a very bad way to iterate towards a better solution.
Not only are we better at it, we're the only place it where it actually happens at all. ChatGPT has literally no mechanism to evaluate solution quality that doesn't involve us, and when it "iterates toward a better solution" it's always one of us who notices that the existing "solution" is poor enough to require such iteration. ChatGPT never notices on its own that there's anything wrong with its output because that is not what it was programmed to do. It's an engine for interacting with words and imagery, not the wider world of which words and imagery are such a tiny part; it's the ultimate Plato's cave dweller.
Self-improvement is the impressively scary capability of machine learning, but it's notable that every opinion piece that touches on it uses the example not of ChatGPT but of Alpha Zero, citing that engine's ability to self-program a beginner-to-grandmaster speedrun given only the rules of chess or Go to start with.
But the "generative" applications hitched to the current hype train are doing nothing like that. They can't, because they have no success criterion to work with that's even close to being as unambiguous as winning a game. Which means, ironically, that unlike Alpha Zero they do not and cannot generate their own training data. We are unavoidably inside their training feedback loops as a matter of fundamental design.
Such progress as is currently being made toward anything like a genuine engineered intelligence is coming not from sparkling autocomplete but autonomous driving, because that's a problem space way more akin to the one that gave rise to animal intelligence than anything you'll find being addressed by a large language model.
posted by flabdablet at 12:15 AM on June 14 [7 favorites]
Given the titanic amount of marketing they already require to show any sign of being able to make an actual return on investment as opposed to merely attracting new speculators, large language models becoming another few OOM more expensive might be enough to kick us out of this fairly sticky local design optimum briskly enough to avoid making global warming significantly worse.
But we are better at reflection and adjustment and not getting stuck in a loop on that kind of thing. One thing I’ve realized over time is that ChatGPT’s “I’m sorry, you’re correct, that was a wrong answer, let me try again” responses are fundamentally a very bad way to iterate towards a better solution.
Not only are we better at it, we're the only place it where it actually happens at all. ChatGPT has literally no mechanism to evaluate solution quality that doesn't involve us, and when it "iterates toward a better solution" it's always one of us who notices that the existing "solution" is poor enough to require such iteration. ChatGPT never notices on its own that there's anything wrong with its output because that is not what it was programmed to do. It's an engine for interacting with words and imagery, not the wider world of which words and imagery are such a tiny part; it's the ultimate Plato's cave dweller.
Self-improvement is the impressively scary capability of machine learning, but it's notable that every opinion piece that touches on it uses the example not of ChatGPT but of Alpha Zero, citing that engine's ability to self-program a beginner-to-grandmaster speedrun given only the rules of chess or Go to start with.
But the "generative" applications hitched to the current hype train are doing nothing like that. They can't, because they have no success criterion to work with that's even close to being as unambiguous as winning a game. Which means, ironically, that unlike Alpha Zero they do not and cannot generate their own training data. We are unavoidably inside their training feedback loops as a matter of fundamental design.
Such progress as is currently being made toward anything like a genuine engineered intelligence is coming not from sparkling autocomplete but autonomous driving, because that's a problem space way more akin to the one that gave rise to animal intelligence than anything you'll find being addressed by a large language model.
posted by flabdablet at 12:15 AM on June 14 [7 favorites]
Sabine Hossenfelder explains everything you need to know about him in the first 2 mins of her video
The gloss went off her for me when she outed herself as somebody who thinks capitalism is more in need of defending than trans kids, but "I see no end, says man who makes money from seeing no end" is solid.
posted by flabdablet at 1:12 AM on June 14 [3 favorites]
The gloss went off her for me when she outed herself as somebody who thinks capitalism is more in need of defending than trans kids, but "I see no end, says man who makes money from seeing no end" is solid.
posted by flabdablet at 1:12 AM on June 14 [3 favorites]
I find it interesting to consider that what we're calling "general intelligence" may require a body of some sort. Or at least a system designed to control a body.
posted by biogeo at 6:03 AM on June 14 [3 favorites]
posted by biogeo at 6:03 AM on June 14 [3 favorites]
@GoogleDeepMind: "With @Harvard, we built a 'virtual rodent' powered by AI to help us better understand how the brain controls movement. 🧠 With deep RL, it learned to operate a biomechanically accurate rat model - allowing us to compare real & virtual neural activity."
A virtual rodent predicts the structure of neural activity across behaviors
posted by kliuless at 8:47 AM on June 14 [4 favorites]
A virtual rodent predicts the structure of neural activity across behaviors
Animals have exquisite control of their bodies, allowing them to perform a diverse range of behaviors. How such control is implemented by the brain, however, remains unclear. Advancing our understanding requires models that can relate principles of control to the structure of neural activity in behaving animals. To facilitate this, we built a ‘virtual rodent’, in which an artificial neural network actuates a biomechanically realistic model of the rat in a physics simulator. We used deep reinforcement learning to train the virtual agent to imitate the behavior of freely-moving rats, thus allowing us to compare neural activity recorded in real rats to the network activity of a virtual rodent mimicking their behavior. We found that neural activity in the sensorimotor striatum and motor cortex was better predicted by the virtual rodent’s network activity than by any features of the real rat’s movements, consistent with both regions implementing inverse dynamics. Furthermore, the network’s latent variability predicted the structure of neural variability across behaviors and afforded robustness in a way consistent with the minimal intervention principle of optimal feedback control. These results demonstrate how physical simulation of biomechanically realistic virtual animals can help interpret the structure of neural activity across behavior and relate it to theoretical principles of motor control.multi-modal embodied cognition?[1]
posted by kliuless at 8:47 AM on June 14 [4 favorites]
« Older How To Avoid Being Eaten By A Black Bear | There's never been a better time to get into... Newer »
This thread has been archived and is closed to new comments
IMHO this is the fundamental limit of these neural network approaches to AI. He then goes on to admit that yah, maybe these models need to learn in a more human like way if we want them to get past the data wall, but this is a really deep problem. We don't have these techniques. It's been a common refrain that domain-specific knowledge and model-based approaches always get beaten by scale (https://www.cs.utexas.edu/~eunsol/courses/data/bitter_lesson.pdf), but we're actually hitting the limits of scale.
Now as someone who does research on model-based methods (in the sense that the models have a lot of structure that are meant to match the domain, rather than having so many parameters you can just fit everything) I'm definitely not unbiased. But as someone who uses chatGPT and copilot daily, and is very thankful for these technologies, I'm not actually expecting them to get much better in the near future. Maybe in another 10 or 20 years we'll start having the requisite theoretical breakthroughs --- but as is often pointed out, the core theory behind deep learning hasn't advanced much since the 80s, so it's really not clear how long these breakthroughs will take.
posted by Alex404 at 1:41 AM on June 13 [20 favorites]