As someone who thinks a rapid (software-only) intelligence explosion is likely, I thought I would respond to this post and try to make the case in favor. I tend to think that AI 2027 is a quite aggressive, but plausible scenario.
---
I interpret the core argument in AI 2027 as:
- We're on track to build AIs which can fully automate research engineering in a few years. (Or at least, this is plausible, like >20%.) AI 2027 calls this level of AI "superhuman coder". (Argument: https://ai-2027.com/research/timelines-forecast)
- Once you have superhuman coders, unassisted humans would only take a moderate number of years to make AIs which can automate all of AI research ("superhuman AI researcher"), like maybe 5 years. (Argument: https://ai-2027.com/research/takeoff-forecast#human-only-timeline-from-sc-to-sar). And, because of AI R&D acceleration along the way, this will actually happen much faster.
- Superhuman AI researchers can obsolete and outcompete humans at all AI research. This includes messy data research, discovering new paradigms, and whatever humans might be doing which is important. These AIs will also be substantially faster and cheaper than humans, like fast and cheap enough that with 1/5 of our compute we can run 200k parallel copies each at 40x speed (for ~8 million parallel worker equivalents). Because labor is a key input into AI R&D, these AIs will be able to speed things up by ~25x. (Argument: https://ai-2027.com/research/takeoff-forecast#sar-would-25x-ai-randd)
- These speed ups don't stop just above the human range, they continue substantially beyond the human range, allowing for quickly yielding AIs which are much more capable than humans. (AI 2027 doesn't really argue for this, but you can see here: https://ai-2027.com/research/takeoff-forecast#siar-would-250x-ai-randd)
(There are a few other minor components like interpolating the AI R&D multipliers between these milestones and the argument for being able to get these systems to run at effective speeds which are much higher than humans. It's worth noting that these numbers are more aggressive than my actual median view, especially on the timeline to superhuman coder.)
---
Ok, now what is your counterargument?
Sections (1) and (2) don't seem in conflict with the AI 2027 view or the possibility of a software-only intelligence explosion. (I'm not claiming you intended these to be in conflict, just clarifying why I'm not responding. I assume you have these sections as relevant context for later arguments.)
As far as I can tell, section (3) basically says "You won't get large acceleration just via automating implementation and making things more compute efficient. Also, ML research is messy and thus it will be hard to get AI systems which can actually fully automate this."
AI 2027 basically agrees with these literal words:
- The acceleration from superhuman coders which fully automate research engineering (and run at high speeds) is "only" 5x, and to get the higher 25x (or beyond) acceleration, the AIs needed to be as good as the best ML researchers and fully automate ML research.
- AI 2027 thinks it would take ~5 years of unaccelerated human AI R&D progress to get from superhuman coder to superhuman AI researcher, so the forecast incorporates this being pretty hard.
There is presumably a quantitative disagreement: you probably think that the acceleration from superhuman coders is lower and the unassisted research time to go from superhuman coders to superhuman AI researchers is longer (much more than 5 years?).
(FWIW, I think AI 2027 is probably too bullish on the superhuman coder AI R&D multiplier, I expect more like 3x.)
I'll break this down a bit further.
Importantly, I think the claim:
> For machine learning research to accelerate at these rates, it needs to be entirely bottlenecked by compute efficiency and implementation difficulty.
Is probably missing the point: AI 2027 is claiming that we'll get these large multipliers by being able to make AIs which beat humans at the overall task of AI research! So, it just needs to be the case that machine learning research could be greatly accelerated by much faster (and eventually very superhuman) labor. I think the extent of this acceleration and the returns are an open question, but I don't think whether it is entirely bottlenecked by compute efficiency and implementation difficulty is the crux. (The extent to which compute efficiency and implementation difficulty bottleneck AI R&D is the most important factor for the superhuman coder acceleration, but not the acceleration beyond this.)
As far as the difficulty of making a superhuman AI researcher, I think your implicit vibes are something like "we can make AIs better and better at coding (or other easily checkable tasks), but this doesn't transfer to good research taste and intuitions". I think the exact quantitative difficulty of humans making AIs which can automate ML research is certainly an open question (and it would be great to get an overall better understanding), but I think there are good reasons to expect the time frame for unassisted humans (given a fixed compute stock) to be more like 3-10 years than like 30 years:
- ML went from AlexNet to GPT-4 in 10 years! Fast progress has happened historically. To be clear, this was substantially driven by compute scaling (for both training runs and experiments), but nonetheless, research progress can be fast. The gap from superhuman coder to superhuman AI research intuitively feels like a much smaller gap than the gap from AlexNet to GPT-4.
- For success on harder to check tasks, there are hopes for both extending our ability to check and generalization. See [this blog post for more discussion](https://helentoner.substack.com/p/2-big-questions-for-ai-progress-in). Concretely, I think we can measure whether AIs produce some research insight that ended up improving performance and so we can RL on this, at least at small scale (which might transfer). We can also train on things like forecasting the results of training runs etc. Beyond this, we might just be able to get AIs to generalize well: many of current AI capabilities are due to reasonably far generalization and I expect this to continue.
---
As far as section (4), I think the core argument is "solving real-world domains (e.g., things which aren't software only) will be hard". You might also be arguing that RL will struggle to teach AIs to be good ML researchers (it's a bit hard for me to tell how applicable your argument is to this).
I think the biggest response to the concern with real world domains is "the superhuman AI researchers can figure this out even if it would have taken humans a while". As far as RL struggling to teach AIs to be good scientists (really ML researchers in particular), I think this is addressed above, though I agree this is a big unknown. Note that we can (and will) explore approaches that differ some from the current RL paradigm and acceleration from superhuman coders might make this happen a bunch faster.
---
> In reality, with the large sums of capital at play, it is unlikely that labs give free rein to billions of dollars of compute to so called "AI researchers in the datacenter" because of how constrained compute is at all of the top labs.
I find this argument somewhat wild if you assume that the AI researchers in the datacenter are making extremely fast progress. Insofar as the software only singularity works, the AI researchers in the datacenter can rapidly improve efficiency using algorithmic progress saving far more compute than you spent to run them. Currently, inference costs for a given level of performance drop by around 9-40x per year depending on the exact task. (See the graph you included.) So, if AIs can make the progress happen much faster, it will easily pay for itself on inference costs alone. That said, I don't expect this is the largest source of returns...
There will be diminishing returns, but if the software only singularity ends up being viable, then these diminishing returns will be outpaced by ever more effective and smarter AI researchers in the datacenter meaning progress wouldn't slow until quite high limits are reached.
I assume your actual crux is about the returns to AI R&D at the point of full (or substantial) automation by AIs. You think the returns are low enough that companies will (rationally) invest elsewhere.
I agree that the fact that AI companies aren't investing all their compute into AI R&D is some update about the returns to algorithms progress, but the regime might differ substantially after full automation by AI researchers!
(And, I think the world would probably be better if AI companies all avoid doing a rapid intelligence explosion internally!)
Minimally, insofar as AI companies are racing as fast as possible, we'd expect that whatever route the companies take is the one they think is fastest! So, if there is another route than AI accelerating AI R&D which is faster, fair enough companies would do this instead, but if the AI automating AI R&D story is insanely fast, then this route must be even faster.
Nice! Thanks for the nice summary and engagement. You can see my post was a mix of AI 2027 feedback and general commentary on things I view as related. I've skimmed your answer and shared some things that caught my mind.
It is pretty clear to me that where I differ is on very low level assumptions.
Once you assume:
> ~5x accelerate AI R&D because ultra fast and cheap superhuman research engineers would be very helpful.
Rest is downstream. I don't think this is true, at least not additively as research is already much faster.
I also think this:
> AI 2027 thinks it would take ~5 years of unaccelerated human AI R&D progress to get from superhuman coder to superhuman AI researcher, so the forecast incorporates this being pretty hard.
is not well grounded or easy to continue building an arguement on. I think it'll be messier and more outwards than narrow.
Related.
> There is presumably a quantitative disagreement: you probably think that the acceleration from superhuman coders is lower and the unassisted research time to go from superhuman coders to superhuman AI researchers is longer (much more than 5 years?).
yes, but largely delves into "What is AGI" land, which is messy too. I think the areas we are making gains don't translate to necessarily more "intelligent" models with logic, scientific reasoning, and intuition.
> Is probably missing the point: AI 2027 is claiming that we'll get these large multipliers by being able to make AIs which beat humans at the overall task of AI research!
I think the multiples in research speed are from compute not ability, so am saying something different.
> I think there are good reasons to expect the time frame for unassisted humans (given a fixed compute stock) to be more like 3-10 years than like 30 years:
I agree. I dont think its for tasks like AI research, though.
> Rest is downstream. I don't think this is true, at least not additively as research is already much faster.
Maybe this is an interesting place to have future arguments as well as a good warning sign: how much acceleration do you get from AIs which can fully automate engineering? My view and AI 2027's view is something like "software/algorithmic progress goes 3-5x faster (for an overall progress multiplier of perhaps 2x-4x given that we'll be scaling up training compute in addition to software/algorithmic progress).
> is not well grounded or easy to continue building an arguement on.
I certainly agree this isn't well grounded. It seems very hard to predict, but to understand how the future will go, we have to make a guess at questions like this and I'm not currently aware of a better approach.
> I think the areas we are making gains don't translate to necessarily more "intelligent" models with logic, scientific reasoning, and intuition.
Yeah, this might be a big crux. If you don't think we're making basically any progress on large parts of the AGI problem (and that we couldn't make progress on these parts of the problem given a large focused effort), then we'd presumably get stuck somewhere before AGI, regardless of some acceleration along the way.
> I think the multiples in research speed are from compute not ability, so am saying something different.
I think I probably don't understand what you're saying here, maybe that no level of labor ability / speed (e.g., not even arbitrarily good researchers running 100x faster) would make much of a difference? Feel totally free to ignore (even if you decided to respond to my comment), I'm probably just confused and doesn't see that important.
> I agree. I dont think its for tasks like AI research, though.
I'm not sure I understand again. I was claiming that unassisted humans could advance from superhuman coder AIs to superhuman AI researcher AIs (with a fixed compute stock) in more like 3-10 years. I think probably you mean that what the humans would be doing in this time period is more like acquiring data or doing some other task which isn't well described as AI research. Given that we're assuming a fixed compute stock, I'm not sure what this could be other than acquiring data. As far as acquiring data, fair enough, but I think superhuman coder AIs may be able to accelerate this substantially (though less than normal experiments?), especially if the data isn't coming from humans and can instead be synthetic.
Yeah, thanks for the feedback again. I wouldn't say writing these forecasts is my strength, just trying to get what my reactions are to them in as clear of a way as possible. Will try and keep doing stuff like it every few months!
As a former ML researcher I broadly agree, but I also vibe with the 'unhobbling' argument of someone like leopold aschbrenner.
It's not impossible that some ML researcher wakes up, has an idea, and the next day ML systems are 1000x more data efficient during training and suddenly a whole avenue of new abilities has opened up for ML systems.
Yeah, I mean agents are really working this year now with o3! We'll see which types of problems they solve and we'll see when we get the next step change like inference-time compute (and RL for it)
Reading that Wikipedia text, isn't it obvious that the "intelligence explosion" has been slowly happening since the invention of writing, or machines in general, or a transistor. No super-intelligent agents, but a system where cognition grows faster in the infrastructure than in the human minds, and humans gradually become a margin. And the system still continues to grow, humans being important or not. It's exponential, so an explosion, with all the feedback loops. But the whole system instead of an agent, and in slow motion.
(Kind of a sidetrack, I think there was a lot of points on a technical level that are more interesting.)
I’ve had thoughts along these lines as well, particularly that moores law is the explosion. The singularity is usually framed in terms of the point when humans are no longer participating in research — a threshold I am not sure is that important, given that humans tend to continue participating at higher and higher levels of planning. Even if the humans are still involved, the rate of progress could still increase, and it still feels like an explosion.
The world is way too complicated for any one paradigm to simply take it by storm. We've seen that in self-driving cars, warehouse automation, cancer research, you name it.
It will take a lot of time to make the solutions robust, general, cheap, and easy to use.
Somewhat unrelated question: you characterized your position as 'overwhelmingly optimistic' (w.r.t. AI). It seems to me that the possible bad outcomes of the intelligence explosion were the main focus of the AI 2027 article. I'm curious about your assessment of these possibilities. Specifically, what predictions do you have concerning AI development with respect to safety measures, human flourishing, and related considerations?
The first one suggests that ideas were verified and they were good although they were in drug discovery.
The second points as you suggested points towards novel idea generation. But there is little proof towards testing those.
That being said, I believe they will be trained on these specific domains and will become good.
Overall I agree we won't see acceleration like AI 2027 but I am not sure if that will be due to lack of AI being able to do better research than humans by maybe 2030?
My 2nd point was basically if LLM gets good enough at research, we will have enough data points to make reliable progress that we might not need YOLO runs.
Would new Neural Architecture Search results that are actually impressive change your mind about all this? I agree that the failure of NAS a few years ago makes me suspicious of a singularity as well.
1. There is some research suggesting models are already good at idea generation for research than human researchers, why won't they have better intuition for research than us. I understand tinkering and lucky finds but lot of that still can be found by correct ideas and experiments.
2. We need a lot of compute to experiment the ideas, given first point, why won't model be better at finding which experiments are better and have better yolo runs. Plus it has advantage of tracking multiple experiments and selecting right stuff for yolo.
1. Do you have a link? I'm sure AI models can *generate* better ideas than us, but without verifiers we're just sampling a ton from random distributions. As you scale up the N of samples eventually better ideas come. If they cant isolate them, they're useless.
2. I don't really follow. I think yolo runs are far more vibes based than people currently realize, or huge risky bets like GPT-4's 3 month first pretraining run.
I have had a vibe for a long time that LLMs lack deep enough representations to be truly innovative, genius-level, either in ideas or in deciding those yolo runs. Good ideas come from strong intuition, synonymous to vibes here, and not having good enough representations means no strong intuition then.
It's of course hard to say what "deep" or "strong intuition" there means, but your emphasis on operationalism, on feasibility of RL training, made me think that maybe it comes mostly from long-enough experience with real-world-level complexity, which of course is domain-specific in the cases of experts.
Not sure I say anything new there, just clarifying my thinking.
I'm not claiming current models couldn't do some kind of research or help humans on ideas, experimentation, etc. Many kind of research can actually be pretty mundane, more like search by experimentation, and I don't actually know where human-like vibes are essential and where not. (Think of all biomedical science and such where there is little theory and lots of trivia.)
As someone who thinks a rapid (software-only) intelligence explosion is likely, I thought I would respond to this post and try to make the case in favor. I tend to think that AI 2027 is a quite aggressive, but plausible scenario.
---
I interpret the core argument in AI 2027 as:
- We're on track to build AIs which can fully automate research engineering in a few years. (Or at least, this is plausible, like >20%.) AI 2027 calls this level of AI "superhuman coder". (Argument: https://ai-2027.com/research/timelines-forecast)
- Superhuman coders will ~5x accelerate AI R&D because ultra fast and cheap superhuman research engineers would be very helpful. (Argument: https://ai-2027.com/research/takeoff-forecast#sc-would-5x-ai-randd).
- Once you have superhuman coders, unassisted humans would only take a moderate number of years to make AIs which can automate all of AI research ("superhuman AI researcher"), like maybe 5 years. (Argument: https://ai-2027.com/research/takeoff-forecast#human-only-timeline-from-sc-to-sar). And, because of AI R&D acceleration along the way, this will actually happen much faster.
- Superhuman AI researchers can obsolete and outcompete humans at all AI research. This includes messy data research, discovering new paradigms, and whatever humans might be doing which is important. These AIs will also be substantially faster and cheaper than humans, like fast and cheap enough that with 1/5 of our compute we can run 200k parallel copies each at 40x speed (for ~8 million parallel worker equivalents). Because labor is a key input into AI R&D, these AIs will be able to speed things up by ~25x. (Argument: https://ai-2027.com/research/takeoff-forecast#sar-would-25x-ai-randd)
- These speed ups don't stop just above the human range, they continue substantially beyond the human range, allowing for quickly yielding AIs which are much more capable than humans. (AI 2027 doesn't really argue for this, but you can see here: https://ai-2027.com/research/takeoff-forecast#siar-would-250x-ai-randd)
(There are a few other minor components like interpolating the AI R&D multipliers between these milestones and the argument for being able to get these systems to run at effective speeds which are much higher than humans. It's worth noting that these numbers are more aggressive than my actual median view, especially on the timeline to superhuman coder.)
---
Ok, now what is your counterargument?
Sections (1) and (2) don't seem in conflict with the AI 2027 view or the possibility of a software-only intelligence explosion. (I'm not claiming you intended these to be in conflict, just clarifying why I'm not responding. I assume you have these sections as relevant context for later arguments.)
As far as I can tell, section (3) basically says "You won't get large acceleration just via automating implementation and making things more compute efficient. Also, ML research is messy and thus it will be hard to get AI systems which can actually fully automate this."
AI 2027 basically agrees with these literal words:
- The acceleration from superhuman coders which fully automate research engineering (and run at high speeds) is "only" 5x, and to get the higher 25x (or beyond) acceleration, the AIs needed to be as good as the best ML researchers and fully automate ML research.
- AI 2027 thinks it would take ~5 years of unaccelerated human AI R&D progress to get from superhuman coder to superhuman AI researcher, so the forecast incorporates this being pretty hard.
There is presumably a quantitative disagreement: you probably think that the acceleration from superhuman coders is lower and the unassisted research time to go from superhuman coders to superhuman AI researchers is longer (much more than 5 years?).
(FWIW, I think AI 2027 is probably too bullish on the superhuman coder AI R&D multiplier, I expect more like 3x.)
I'll break this down a bit further.
Importantly, I think the claim:
> For machine learning research to accelerate at these rates, it needs to be entirely bottlenecked by compute efficiency and implementation difficulty.
Is probably missing the point: AI 2027 is claiming that we'll get these large multipliers by being able to make AIs which beat humans at the overall task of AI research! So, it just needs to be the case that machine learning research could be greatly accelerated by much faster (and eventually very superhuman) labor. I think the extent of this acceleration and the returns are an open question, but I don't think whether it is entirely bottlenecked by compute efficiency and implementation difficulty is the crux. (The extent to which compute efficiency and implementation difficulty bottleneck AI R&D is the most important factor for the superhuman coder acceleration, but not the acceleration beyond this.)
As far as the difficulty of making a superhuman AI researcher, I think your implicit vibes are something like "we can make AIs better and better at coding (or other easily checkable tasks), but this doesn't transfer to good research taste and intuitions". I think the exact quantitative difficulty of humans making AIs which can automate ML research is certainly an open question (and it would be great to get an overall better understanding), but I think there are good reasons to expect the time frame for unassisted humans (given a fixed compute stock) to be more like 3-10 years than like 30 years:
- ML went from AlexNet to GPT-4 in 10 years! Fast progress has happened historically. To be clear, this was substantially driven by compute scaling (for both training runs and experiments), but nonetheless, research progress can be fast. The gap from superhuman coder to superhuman AI research intuitively feels like a much smaller gap than the gap from AlexNet to GPT-4.
- For success on harder to check tasks, there are hopes for both extending our ability to check and generalization. See [this blog post for more discussion](https://helentoner.substack.com/p/2-big-questions-for-ai-progress-in). Concretely, I think we can measure whether AIs produce some research insight that ended up improving performance and so we can RL on this, at least at small scale (which might transfer). We can also train on things like forecasting the results of training runs etc. Beyond this, we might just be able to get AIs to generalize well: many of current AI capabilities are due to reasonably far generalization and I expect this to continue.
---
As far as section (4), I think the core argument is "solving real-world domains (e.g., things which aren't software only) will be hard". You might also be arguing that RL will struggle to teach AIs to be good ML researchers (it's a bit hard for me to tell how applicable your argument is to this).
I think the biggest response to the concern with real world domains is "the superhuman AI researchers can figure this out even if it would have taken humans a while". As far as RL struggling to teach AIs to be good scientists (really ML researchers in particular), I think this is addressed above, though I agree this is a big unknown. Note that we can (and will) explore approaches that differ some from the current RL paradigm and acceleration from superhuman coders might make this happen a bunch faster.
---
> In reality, with the large sums of capital at play, it is unlikely that labs give free rein to billions of dollars of compute to so called "AI researchers in the datacenter" because of how constrained compute is at all of the top labs.
I find this argument somewhat wild if you assume that the AI researchers in the datacenter are making extremely fast progress. Insofar as the software only singularity works, the AI researchers in the datacenter can rapidly improve efficiency using algorithmic progress saving far more compute than you spent to run them. Currently, inference costs for a given level of performance drop by around 9-40x per year depending on the exact task. (See the graph you included.) So, if AIs can make the progress happen much faster, it will easily pay for itself on inference costs alone. That said, I don't expect this is the largest source of returns...
There will be diminishing returns, but if the software only singularity ends up being viable, then these diminishing returns will be outpaced by ever more effective and smarter AI researchers in the datacenter meaning progress wouldn't slow until quite high limits are reached.
I assume your actual crux is about the returns to AI R&D at the point of full (or substantial) automation by AIs. You think the returns are low enough that companies will (rationally) invest elsewhere.
I agree that the fact that AI companies aren't investing all their compute into AI R&D is some update about the returns to algorithms progress, but the regime might differ substantially after full automation by AI researchers!
(And, I think the world would probably be better if AI companies all avoid doing a rapid intelligence explosion internally!)
Minimally, insofar as AI companies are racing as fast as possible, we'd expect that whatever route the companies take is the one they think is fastest! So, if there is another route than AI accelerating AI R&D which is faster, fair enough companies would do this instead, but if the AI automating AI R&D story is insanely fast, then this route must be even faster.
Nice! Thanks for the nice summary and engagement. You can see my post was a mix of AI 2027 feedback and general commentary on things I view as related. I've skimmed your answer and shared some things that caught my mind.
It is pretty clear to me that where I differ is on very low level assumptions.
Once you assume:
> ~5x accelerate AI R&D because ultra fast and cheap superhuman research engineers would be very helpful.
Rest is downstream. I don't think this is true, at least not additively as research is already much faster.
I also think this:
> AI 2027 thinks it would take ~5 years of unaccelerated human AI R&D progress to get from superhuman coder to superhuman AI researcher, so the forecast incorporates this being pretty hard.
is not well grounded or easy to continue building an arguement on. I think it'll be messier and more outwards than narrow.
Related.
> There is presumably a quantitative disagreement: you probably think that the acceleration from superhuman coders is lower and the unassisted research time to go from superhuman coders to superhuman AI researchers is longer (much more than 5 years?).
yes, but largely delves into "What is AGI" land, which is messy too. I think the areas we are making gains don't translate to necessarily more "intelligent" models with logic, scientific reasoning, and intuition.
> Is probably missing the point: AI 2027 is claiming that we'll get these large multipliers by being able to make AIs which beat humans at the overall task of AI research!
I think the multiples in research speed are from compute not ability, so am saying something different.
> I think there are good reasons to expect the time frame for unassisted humans (given a fixed compute stock) to be more like 3-10 years than like 30 years:
I agree. I dont think its for tasks like AI research, though.
Thanks for the reply!
> Rest is downstream. I don't think this is true, at least not additively as research is already much faster.
Maybe this is an interesting place to have future arguments as well as a good warning sign: how much acceleration do you get from AIs which can fully automate engineering? My view and AI 2027's view is something like "software/algorithmic progress goes 3-5x faster (for an overall progress multiplier of perhaps 2x-4x given that we'll be scaling up training compute in addition to software/algorithmic progress).
> is not well grounded or easy to continue building an arguement on.
I certainly agree this isn't well grounded. It seems very hard to predict, but to understand how the future will go, we have to make a guess at questions like this and I'm not currently aware of a better approach.
> I think the areas we are making gains don't translate to necessarily more "intelligent" models with logic, scientific reasoning, and intuition.
Yeah, this might be a big crux. If you don't think we're making basically any progress on large parts of the AGI problem (and that we couldn't make progress on these parts of the problem given a large focused effort), then we'd presumably get stuck somewhere before AGI, regardless of some acceleration along the way.
> I think the multiples in research speed are from compute not ability, so am saying something different.
I think I probably don't understand what you're saying here, maybe that no level of labor ability / speed (e.g., not even arbitrarily good researchers running 100x faster) would make much of a difference? Feel totally free to ignore (even if you decided to respond to my comment), I'm probably just confused and doesn't see that important.
> I agree. I dont think its for tasks like AI research, though.
I'm not sure I understand again. I was claiming that unassisted humans could advance from superhuman coder AIs to superhuman AI researcher AIs (with a fixed compute stock) in more like 3-10 years. I think probably you mean that what the humans would be doing in this time period is more like acquiring data or doing some other task which isn't well described as AI research. Given that we're assuming a fixed compute stock, I'm not sure what this could be other than acquiring data. As far as acquiring data, fair enough, but I think superhuman coder AIs may be able to accelerate this substantially (though less than normal experiments?), especially if the data isn't coming from humans and can instead be synthetic.
Yeah, thanks for the feedback again. I wouldn't say writing these forecasts is my strength, just trying to get what my reactions are to them in as clear of a way as possible. Will try and keep doing stuff like it every few months!
Yep, good to get your reactions!
As a former ML researcher I broadly agree, but I also vibe with the 'unhobbling' argument of someone like leopold aschbrenner.
It's not impossible that some ML researcher wakes up, has an idea, and the next day ML systems are 1000x more data efficient during training and suddenly a whole avenue of new abilities has opened up for ML systems.
My general sense is we'll have AGI by 2045-ish
Yeah, I mean agents are really working this year now with o3! We'll see which types of problems they solve and we'll see when we get the next step change like inference-time compute (and RL for it)
Reading that Wikipedia text, isn't it obvious that the "intelligence explosion" has been slowly happening since the invention of writing, or machines in general, or a transistor. No super-intelligent agents, but a system where cognition grows faster in the infrastructure than in the human minds, and humans gradually become a margin. And the system still continues to grow, humans being important or not. It's exponential, so an explosion, with all the feedback loops. But the whole system instead of an agent, and in slow motion.
(Kind of a sidetrack, I think there was a lot of points on a technical level that are more interesting.)
I’ve had thoughts along these lines as well, particularly that moores law is the explosion. The singularity is usually framed in terms of the point when humans are no longer participating in research — a threshold I am not sure is that important, given that humans tend to continue participating at higher and higher levels of planning. Even if the humans are still involved, the rate of progress could still increase, and it still feels like an explosion.
We're definitely in the era of most rapid technological improvement, depends where you think it sigmoids out.
The world is way too complicated for any one paradigm to simply take it by storm. We've seen that in self-driving cars, warehouse automation, cancer research, you name it.
It will take a lot of time to make the solutions robust, general, cheap, and easy to use.
Really thought-provoking. Thanks Nathan!
Somewhat unrelated question: you characterized your position as 'overwhelmingly optimistic' (w.r.t. AI). It seems to me that the possible bad outcomes of the intelligence explosion were the main focus of the AI 2027 article. I'm curious about your assessment of these possibilities. Specifically, what predictions do you have concerning AI development with respect to safety measures, human flourishing, and related considerations?
Was talking about https://arxiv.org/abs/2502.18864 and https://arxiv.org/abs/2409.04109
The first one suggests that ideas were verified and they were good although they were in drug discovery.
The second points as you suggested points towards novel idea generation. But there is little proof towards testing those.
That being said, I believe they will be trained on these specific domains and will become good.
Overall I agree we won't see acceleration like AI 2027 but I am not sure if that will be due to lack of AI being able to do better research than humans by maybe 2030?
My 2nd point was basically if LLM gets good enough at research, we will have enough data points to make reliable progress that we might not need YOLO runs.
Would new Neural Architecture Search results that are actually impressive change your mind about all this? I agree that the failure of NAS a few years ago makes me suspicious of a singularity as well.
Wdym? I think they will help improve models but something more fundamental gives step changes in perf
Couple of things I keep debating about
1. There is some research suggesting models are already good at idea generation for research than human researchers, why won't they have better intuition for research than us. I understand tinkering and lucky finds but lot of that still can be found by correct ideas and experiments.
2. We need a lot of compute to experiment the ideas, given first point, why won't model be better at finding which experiments are better and have better yolo runs. Plus it has advantage of tracking multiple experiments and selecting right stuff for yolo.
1. Do you have a link? I'm sure AI models can *generate* better ideas than us, but without verifiers we're just sampling a ton from random distributions. As you scale up the N of samples eventually better ideas come. If they cant isolate them, they're useless.
2. I don't really follow. I think yolo runs are far more vibes based than people currently realize, or huge risky bets like GPT-4's 3 month first pretraining run.
I have had a vibe for a long time that LLMs lack deep enough representations to be truly innovative, genius-level, either in ideas or in deciding those yolo runs. Good ideas come from strong intuition, synonymous to vibes here, and not having good enough representations means no strong intuition then.
It's of course hard to say what "deep" or "strong intuition" there means, but your emphasis on operationalism, on feasibility of RL training, made me think that maybe it comes mostly from long-enough experience with real-world-level complexity, which of course is domain-specific in the cases of experts.
Not sure I say anything new there, just clarifying my thinking.
I'm not claiming current models couldn't do some kind of research or help humans on ideas, experimentation, etc. Many kind of research can actually be pretty mundane, more like search by experimentation, and I don't actually know where human-like vibes are essential and where not. (Think of all biomedical science and such where there is little theory and lots of trivia.)