I appreciate your measured take, and I agree with most of it.
> Diminishing returns of more AI agents in parallel are real
I can't help but feel that we are only in the foothills of scaling agents. I agree that Amdahl's Law is likely in play here -- I think Noam said they don't yet have good heuristics on what number/structure of agents works best. My guess is there is still a lot of capability left to unlock as they iterate on this paradigm. Right now we see the progress in the verifiable domains, but I bet we will see some generalization in other less verifiable domains. When and how much, I'm not sure.
Mainly, I'm blown away that they had 10k agents work towards a problem *at all* without it degenerating into a hallucination spiral. To me that's the biggest surprise out of all of this
thank you for another very helpful distillation of very confusing data points.
and i think your article is a perfect example of how a human, with clear thinking and good articulation, can balance so many conflicting ideas, their biases incentives blind spots etc, and collate them into something that moves the ball forward while keeping the possibilities open. despite LLMs being great at absorbing vastly more information and great at connecting dots, and frequently doing a great job at a similar task, i 'feel' something is missing in their analysis, or more accurately, being more than required confident and therefore preemptively closing some paths in a subtle way. and i think one factor that leads to this difference is humans knack for knowing what they don't know. there are other factors too like humans know how they have been wrong, in small and big ways, in their analysis and predictions so many times in their life, and that brings in a natural self-doubt/humility. an LLM has some notion of to be careful and not being overly confident but it hasn't grown this self-doubt in a natural way and is unsurprisingly artificial and not as effective.
all of this is i guess to say that any research/exploration requires at least 3 things - ability to connect dots, self-doubt, and confidence to own the outcome of decision. LLMs lack the last and self-doubt is not very effective. massive parallel search can compensate for some of the limitations but not sufficiently, especially the lack of ownership/accountability.
massive parallel search approach can hide this limitation, especially when there is a verifiable right answer but because it is almost impossible, even with current compute, to study and learn from the path taken to get there, all failed/dead-end searches do not help with any other pursuit. this is very different from my understanding of how science progresses - people try a much smaller number of paths but learn from many of them, and ideas from one field (even of failed experiments) trigger ideas in other related and unrelated fields. so even though on one particular problem the progress might seem slow (compared to AI), the process is much more organic/natural and therefore more robust to sustain and expand growth in many directions.
for these reasons it seems to me that AI is like a laser that can really lit up one very tiny surface very brightly, but when exploring what is more efficient and effective over long time horizon is a much dimmer torch that sheds diffused light over a much larger area.
I really appreciate your work. Always insightful, always balanced. And I agree that Dwarkesh’s discussion with Noam Brown was one of the best in a while. Dwarkesh seemed to be trying to lure him into accepting the premise of a coming cataclysm. Noam was just unflappable. Perspectives like his and yours will be critical as we enter the policymaking phase of the discussion.
I think Dwarkesh does a good job as an interviewer in taking points of view to challenge his guests, even if it makes himself come off looking a little weird. Eg the Jensen and Dario podcasts.
Dear Nathan, thank you for the most sobering AI piece I've read this year. As a practitioner designing my own house (BIM + AI visualization + agentic supplier analysis), your "lossy self-improvement" maps perfectly onto my experience: AI renders dozens of facade variants in minutes, but never once told me the white corner quoins looked trivial — I had to see it myself. The bottleneck is framing and judging, not generation. Your conclusion is the best vaccine against hype and fear alike: AI won't replace the architect tomorrow; it replaces routine. The scarce assets become taste, systemic thinking, and the right questions. We're not flying into a singularity — we're entering an era where cognitive labor gets cheap and human intuition, taste, and responsibility get priceless. For those of us building real houses with real bricks, that's the best possible division of labor. Thank you for keeping the discourse honest!
The singularity is not a moment it time. It is a process that began when we breached the event horizon and what was is gone. Leaving only what will be.
Thanks for the measured take, Nathan. The point that stayed with me is that experiments may get 10x faster while understanding barely moves. I think the same split is about to show up inside ordinary companies. Agents will produce analysis far faster than teams can agree on what it means, and the slow part of most business decisions was never the analysis anyway. The firms that benefit most will probably be the ones that redesign how they review and decide, not just how they produce.
Your 2023 point is the one that stuck with me. Nobody really paid a price for being wrong about those timelines, and that's probably why the next round sounds just as sure.
A crucial factor for the labs will be increasing margins as revenue could potentially have negative pressure if there’s fierce competition in lowering prices at a fixed intelligence level.
I think margins increase will not happen unless AI companies enter into economic cartel. The pressure to reduce prices and get bigger share of customer base will be more strong than will to keep margins high.
Making the loop faster only helps if it’s the right loop.
Agents can run far more experiments when the objective is clear and the result is measurable. But if post-training can go wrong in ways the benchmark does not catch, more optimisation can just push harder on the wrong signal.
The hard bit is still deciding what to measure and then noticing when the measurement is misleading you.
Appreciate your analysis on this. I find this convincing - that we will certainly see strong AI efficiency gains, likely hugely economically transformative gains, but we haven't yet seen innovative intelligence at the level that can invent & communicate new algorithmic paradigms that would be needed to meaningfully bend the current curves of cost and progress
I appreciate your measured take, and I agree with most of it.
> Diminishing returns of more AI agents in parallel are real
I can't help but feel that we are only in the foothills of scaling agents. I agree that Amdahl's Law is likely in play here -- I think Noam said they don't yet have good heuristics on what number/structure of agents works best. My guess is there is still a lot of capability left to unlock as they iterate on this paradigm. Right now we see the progress in the verifiable domains, but I bet we will see some generalization in other less verifiable domains. When and how much, I'm not sure.
Mainly, I'm blown away that they had 10k agents work towards a problem *at all* without it degenerating into a hallucination spiral. To me that's the biggest surprise out of all of this
thank you for another very helpful distillation of very confusing data points.
and i think your article is a perfect example of how a human, with clear thinking and good articulation, can balance so many conflicting ideas, their biases incentives blind spots etc, and collate them into something that moves the ball forward while keeping the possibilities open. despite LLMs being great at absorbing vastly more information and great at connecting dots, and frequently doing a great job at a similar task, i 'feel' something is missing in their analysis, or more accurately, being more than required confident and therefore preemptively closing some paths in a subtle way. and i think one factor that leads to this difference is humans knack for knowing what they don't know. there are other factors too like humans know how they have been wrong, in small and big ways, in their analysis and predictions so many times in their life, and that brings in a natural self-doubt/humility. an LLM has some notion of to be careful and not being overly confident but it hasn't grown this self-doubt in a natural way and is unsurprisingly artificial and not as effective.
all of this is i guess to say that any research/exploration requires at least 3 things - ability to connect dots, self-doubt, and confidence to own the outcome of decision. LLMs lack the last and self-doubt is not very effective. massive parallel search can compensate for some of the limitations but not sufficiently, especially the lack of ownership/accountability.
massive parallel search approach can hide this limitation, especially when there is a verifiable right answer but because it is almost impossible, even with current compute, to study and learn from the path taken to get there, all failed/dead-end searches do not help with any other pursuit. this is very different from my understanding of how science progresses - people try a much smaller number of paths but learn from many of them, and ideas from one field (even of failed experiments) trigger ideas in other related and unrelated fields. so even though on one particular problem the progress might seem slow (compared to AI), the process is much more organic/natural and therefore more robust to sustain and expand growth in many directions.
for these reasons it seems to me that AI is like a laser that can really lit up one very tiny surface very brightly, but when exploring what is more efficient and effective over long time horizon is a much dimmer torch that sheds diffused light over a much larger area.
"balance so many conflicting ideas" is a good summary of what it feels like to try and stay sane these days in AI
I really appreciate your work. Always insightful, always balanced. And I agree that Dwarkesh’s discussion with Noam Brown was one of the best in a while. Dwarkesh seemed to be trying to lure him into accepting the premise of a coming cataclysm. Noam was just unflappable. Perspectives like his and yours will be critical as we enter the policymaking phase of the discussion.
I think Dwarkesh does a good job as an interviewer in taking points of view to challenge his guests, even if it makes himself come off looking a little weird. Eg the Jensen and Dario podcasts.
Dear Nathan, thank you for the most sobering AI piece I've read this year. As a practitioner designing my own house (BIM + AI visualization + agentic supplier analysis), your "lossy self-improvement" maps perfectly onto my experience: AI renders dozens of facade variants in minutes, but never once told me the white corner quoins looked trivial — I had to see it myself. The bottleneck is framing and judging, not generation. Your conclusion is the best vaccine against hype and fear alike: AI won't replace the architect tomorrow; it replaces routine. The scarce assets become taste, systemic thinking, and the right questions. We're not flying into a singularity — we're entering an era where cognitive labor gets cheap and human intuition, taste, and responsibility get priceless. For those of us building real houses with real bricks, that's the best possible division of labor. Thank you for keeping the discourse honest!
Best regards,
Vladimir Bernadsky
"progress in math is an exception, rather than a rule"
As of yet LLMs have chased upon directions created by humans and have not invented their own directions...
The singularity is not a moment it time. It is a process that began when we breached the event horizon and what was is gone. Leaving only what will be.
We crossed that threshold sometime ago.
What is needed is clarity on what comes next.
Thanks for the measured take, Nathan. The point that stayed with me is that experiments may get 10x faster while understanding barely moves. I think the same split is about to show up inside ordinary companies. Agents will produce analysis far faster than teams can agree on what it means, and the slow part of most business decisions was never the analysis anyway. The firms that benefit most will probably be the ones that redesign how they review and decide, not just how they produce.
Your 2023 point is the one that stuck with me. Nobody really paid a price for being wrong about those timelines, and that's probably why the next round sounds just as sure.
Appreciate the moderate take. RSI discourse is usually all extremes, this is more honest.
A crucial factor for the labs will be increasing margins as revenue could potentially have negative pressure if there’s fierce competition in lowering prices at a fixed intelligence level.
I think margins increase will not happen unless AI companies enter into economic cartel. The pressure to reduce prices and get bigger share of customer base will be more strong than will to keep margins high.
Making the loop faster only helps if it’s the right loop.
Agents can run far more experiments when the objective is clear and the result is measurable. But if post-training can go wrong in ways the benchmark does not catch, more optimisation can just push harder on the wrong signal.
The hard bit is still deciding what to measure and then noticing when the measurement is misleading you.
Appreciate your analysis on this. I find this convincing - that we will certainly see strong AI efficiency gains, likely hugely economically transformative gains, but we haven't yet seen innovative intelligence at the level that can invent & communicate new algorithmic paradigms that would be needed to meaningfully bend the current curves of cost and progress