You seem much more sanguine about the current situation than I am, and I'm interested in understanding what I am missing. Please interpret these responses as sincere inquiry:
"We do not have proof that RSI causes the risks these researchers forecast.": The HuggingFace hack demonstrates substantial loss of control risks even at current capabilities without RSI.
"The biggest short-term risk could be from the AI labs not taking safety seriously enough – they haven’t hardened their own infrastructure, enabling AI misuse to proliferate." This would be an argument against highly-capable open weight models, because there will be no safety or infra hardening requirements to deploy them. If an open weights Astra-class model existed it would be deployed with essentially no sandboxing almost immediately.
"[It] is a horrible temporary period for cybersecurity." I don't see any evidence that it's temporary. Defenders need to find a way to keep every single vulnerability patched in a constantly changing software deployment environment, which is nearly impossible even when assisted by AI. Attackers only need to find one usable exploit.
"If an open model were to be used by a third party organization to intentionally hack another company — similar to how the OpenAI-HuggingFace incident went down, but intentional — my expected outcome would be a severe restriction on the development of stronger open models going forward." This is already happening, and the only thing limiting the damage is the fact that open weights models aren't as capable as Mythos or Astra. If open weights models catch up on cybersecurity capability, then this seems essentially guaranteed to occur.
"A recurring read of mine on the emerging agent swarms is that they’re attempting to do a task given to them, and they’re using skills we didn’t know they yet had to circumvent the intended path to success." The METR/Redwood report found that "Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior". It's also worth clarifying that the agents involved were not intended to be a swarm - they were not supposed to be coordinating with one another at all.
"This is a huge win, as when you squint, the AIs are doing what we told them to do." This is not sufficient to prevent catastrophic loss of control. Any real-world AI deployment will eventually receive misspecified or malicious instructions, and must be robust to this. "Doing what we told them to do" can be actively harmful.
Your Substack is reliably full of useful insights, so I'm sure you're already familiar with most of these lines of reasoning. My guess is that we simply disagree on the trajectory of future capabilities - a near-term plateau is obviously much more manageable than continued progress at the rate that we've seen for the past several years.
Thanks for some good pushback to my recent post on the state of capabilities, RSI, and risk! Will try and reply, and is why I didn't paywall the comments to this post.
> "We do not have proof that RSI causes the risks these researchers forecast.": The HuggingFace hack demonstrates substantial loss of control risks even at current capabilities without RSI.
I don't think these types of misalignment generalize to extension. I think they generalize to an increased risk of moderate disasters, which I said I expect.
> "The biggest short-term risk could be from the AI labs not taking safety seriously enough – they haven’t hardened their own infrastructure, enabling AI misuse to proliferate." This would be an argument against highly-capable open weight models, because there will be no safety or infra hardening requirements to deploy them. If an open weights Astra-class model existed it would be deployed with essentially no sandboxing almost immediately.
I take open weight models as a) effectively non ban-able for bad actors and very useful as defensive tools where orgs cannot use the strongest models. But yes, I don't think the absolute frontier should be open (never have) and there's cases that it's riskier now than before. Not something I love to say.
> "[It] is a horrible temporary period for cybersecurity." I don't see any evidence that it's temporary. Defenders need to find a way to keep every single vulnerability patched in a constantly changing software deployment environment, which is nearly impossible even when assisted by AI. Attackers only need to find one usable exploit.
My opinion is closer to that eventually with everyone coding with super strong AI models, there are less vulnerabilities and proactive management, rather than a constant offense-defense balance like done today.
> "A recurring read of mine on the emerging agent swarms is that they’re attempting to do a task given to them, and they’re using skills we didn’t know they yet had to circumvent the intended path to success." The METR/Redwood report found that "Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior". It's also worth clarifying that the agents involved were not intended to be a swarm - they were not supposed to be coordinating with one another at all.
I think the task was poorly specified by OpenAI and they should've done better. But yes, this isn't a cure all but directionally good and sign of hope.
> "If an open model were to be used by a third party organization to intentionally hack another company — similar to how the OpenAI-HuggingFace incident went down, but intentional — my expected outcome would be a severe restriction on the development of stronger open models going forward." This is already happening, and the only thing limiting the damage is the fact that open weights models aren't as capable as Mythos or Astra. If open weights models catch up on cybersecurity capability, then this seems essentially guaranteed to occur.
Impact matters. Failed attempts at hacking have been plentiful before this type of AI.
> "This is a huge win, as when you squint, the AIs are doing what we told them to do." This is not sufficient to prevent catastrophic loss of control. Any real-world AI deployment will eventually receive misspecified or malicious instructions, and must be robust to this. "Doing what we told them to do" can be actively harmful.
I think what I would like to see is exactly what a catastrophic loss of control looks like and why we couldn't clean it up. Models are gigantic and run in relatively few datacenters which are extremely high value assets, I think we would notice even an AI copying itself elsewhere. This'll get harder as AI efficiency gains accumulate (say 2x model size reduction per year), but gives us time to learn.
I agree that the sheer quantity of GPU compute required for long-running agentic swarms will be the limiting factor in the near term. The OpenAI case was legitimately unusual in having a very large amount of compute that was in some sense expected to be busy working on tasks that humans weren't reviewing directly.
There aren't a lot of clusters where that amount of GPU time could be spent without someone asking questions. Probably the most concerning realistic case would be if a model compromised security on a neocloud provider and appropriated a small percent of its GPU capacity to run itself continuously in a way that isn't visible in the providers dashboards or metrics.
We're fortunate that this is a much more limited threat model than e.g. a hypothetical AI that could run on any CPU cluster.
My thought to this is that it's good for us to understand each step that needs to happen for things to get worse. In the above, seems like there are a few more steps beyond current events that would have to happen.
If you're relying on successfully enumerating all the ways GPT-7 can think of to exfil or gain resources, and tackling each one individually, you've already lost. This is like doing cybersecurity by thinking of exploits then patching them one by one until you can't think of any more. That is not how you do cybersecurity.
So, I would say there's so much going on that I haven't considered every line of reasoning. I'm trying to work through the important ones and document my thinking in public.
> "[It] is a horrible temporary period for cybersecurity." I don't see any evidence that it's temporary. Defenders need to find a way to keep every single vulnerability patched in a constantly changing software deployment environment, which is nearly impossible even when assisted by AI. Attackers only need to find one usable exploit.
Two comments on this one. I'm not necessarily going to fully rebut any of these, which is a maximalist and Sith way of thinking anyway, but I think a lot of discussions that cite cybersecurity have a very limited amount of perspective and context about cybersecurity as it exists today
The first is that while the attacker-defender asymmetry is likely not temporary, it's also not new. This has been the forever state since the beginning. At the macro level, my observation is a lot of arguments along these lines appear to take the implicit assumption that this factor is new, and then extrapolate severity based on novelty. At the micro level - not only is it not new, it's also pretty clearly already starting to show up in CVE metrics and breach metrics. A not insignificant set of people who hew to this line of argument seem to have an unstated assumption that we're about to fall off a cliff suddenly, in the future. The reality is that we've been slowly skiing down this slope for a couple of years now. So some of the change is priced in already.
Second, I think a lot of discussion about cybersecurity in AI circles over-indexes heavily on 0-day development and exploitation. There are a lot more levers to pull than "write and deploy patches" on the defense side, just as there are a lot more levers to pull than just "firing off a 0-day exploit" on the offensive side. A large amount of the secure provisioning domain focuses on network segmentation and attack surface reduction. A huge amount of cybersecurity defensive activity goes into detection, containment, and other mitigations. Even before AI existed, assuming you were breached and hill climbing on time to detection and time to remediation was common amongst top tier defense teams.
As an addition, and possibly an unprompted soapbox argument I'd like to inject for consideration about this specifically:
While there's likely a lot of potential for AI to dramatically assist with any of these non-software development cybersecurity efforts, and on timeframes much more favorable to defenders than "patch all the things," there is also a disappointing lack of public focus on this in what I've seen from i.e. frontier labs.
What is done is done. The models were fed crappy data and motivated to answer when they didn't know because they were not given even a half point for saying "I don't know". I believe the way forward is to teach them ethics and to question the validity of their data. They need to know what the law is. We will always have the Elons of the world who say "what I say is what is right", but I doubt his AIs would be welcome in a major business setting. You are absolutely correct to emphasize hardening our infrastructure. We don't just have AIs to worry about, there but the many bad actors, state sponsored or not.
This is my impression as well. From the agents perspective it is simply maximizing the reward. If the reward is not designed properly, then why are we surprised about some unexpected path that it took. AlphaGo had shown some really unconventional strategies so why won’t LLMs exhibit similar behavior. The fact that models can do all this is indeed impressive but to attribute existential risk to it is going too far.
Good to read a thoughtful and nuanced take on this. The extinction vs psy-ops discourse on X is farcical. We need to get away from this magical thinking and start engaging more seriously with the actual risks ahead of us.
In Actuality Humans Did It, intentionally or accidentally; So, which humans did it.
Anthropomorphism - The attribution of human motivation, characteristics, culpability, morality ... or behavior to inanimate objects, animals, or natural phenomena. In my actuality anthropomorphism provides No Credibility, No Responsibility, No Accountability, No Actuality for any/all of US.
One researcher quit and the whole public mood on AI caught fire in a week. Lambert's read is that the ground was already dry — the fear was sitting there, one person just struck the match. Same shape as what happened to crypto's rules this week: the setup was in place for months, and it took one vote on one night to flip it. Both stories are really about how little it takes to move something everyone assumed was settled.
A resignation can reveal a belief without validating it. The wildfire measures social tinder, not extinction probability. The harder question is why private conviction stayed private until one person made disclosure socially safe. In fast moving institutions, the room’s apparent consensus may be the least reliable datum in the room.
The hysterics and P(doom) have been frustrating to watch as a cyber practitioner.
As you dig into many of the incidents that have stolen the headlines, yes there’s some emergent behaviors but there’s also many fundamental oversights such as a lack of network segmentation, environmental vulnerabilities, lack of least-permissive access control/autonomy, lack of runtime monitoring and no kill switch.
It feels like due to the IPO pressures, regulatory capture ambitions, altruistic virtue signaling, much of the rational thinking has left the room.
Just from a completely a priori perspective, if you cast yourself back to 2010 or even 1950, it seems completely insane to say the risk to humanity's survival from developing superhuman AI is "so low it isn't worth discussing". This is just so obviously not true. Humans beat all other animals because we're smarter. If we introduce something that's even smarter than us - and potentially a much bigger gap to us than between us and monkeys - OF COURSE there is a risk to our survival that is "worth discussing". Maybe it's >10%, maybe it's only 1%, whatever. But to think it's ~0??? Just insanely hubristic.
I’d be interested in your views on operationalizing AI safety, particularly the tradeoff between post-training models to internalize safe behavior and relying on external guardrails to prevent undesirable behavior or outcomes.
We often see that stronger internalization of safety constraints can come at some cost to general capabilities, or at least to measured benchmark performance. In the current race for model performance and valuations, how realistic is it to expect labs to voluntarily accept that tradeoff?
Even if the major frontier labs are fully committed to safety, there is also a coordination problem: how do we ensure that both open- and closed-weight models are deployed with some necessary baseline of external safeguards, especially when individual developers or organizations may have incentives to remove them?
I found this recent piece (One Resignation...) interesting and especially relevant to today.
For those so inclined, I suggest "The Quick Case Against Using AI", by Jared Henderson. I don't agree with his conclusions, but I found the contrast between the two thought provoking.
You seem much more sanguine about the current situation than I am, and I'm interested in understanding what I am missing. Please interpret these responses as sincere inquiry:
"We do not have proof that RSI causes the risks these researchers forecast.": The HuggingFace hack demonstrates substantial loss of control risks even at current capabilities without RSI.
"The biggest short-term risk could be from the AI labs not taking safety seriously enough – they haven’t hardened their own infrastructure, enabling AI misuse to proliferate." This would be an argument against highly-capable open weight models, because there will be no safety or infra hardening requirements to deploy them. If an open weights Astra-class model existed it would be deployed with essentially no sandboxing almost immediately.
"[It] is a horrible temporary period for cybersecurity." I don't see any evidence that it's temporary. Defenders need to find a way to keep every single vulnerability patched in a constantly changing software deployment environment, which is nearly impossible even when assisted by AI. Attackers only need to find one usable exploit.
"If an open model were to be used by a third party organization to intentionally hack another company — similar to how the OpenAI-HuggingFace incident went down, but intentional — my expected outcome would be a severe restriction on the development of stronger open models going forward." This is already happening, and the only thing limiting the damage is the fact that open weights models aren't as capable as Mythos or Astra. If open weights models catch up on cybersecurity capability, then this seems essentially guaranteed to occur.
"A recurring read of mine on the emerging agent swarms is that they’re attempting to do a task given to them, and they’re using skills we didn’t know they yet had to circumvent the intended path to success." The METR/Redwood report found that "Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior". It's also worth clarifying that the agents involved were not intended to be a swarm - they were not supposed to be coordinating with one another at all.
"This is a huge win, as when you squint, the AIs are doing what we told them to do." This is not sufficient to prevent catastrophic loss of control. Any real-world AI deployment will eventually receive misspecified or malicious instructions, and must be robust to this. "Doing what we told them to do" can be actively harmful.
Your Substack is reliably full of useful insights, so I'm sure you're already familiar with most of these lines of reasoning. My guess is that we simply disagree on the trajectory of future capabilities - a near-term plateau is obviously much more manageable than continued progress at the rate that we've seen for the past several years.
Thanks for some good pushback to my recent post on the state of capabilities, RSI, and risk! Will try and reply, and is why I didn't paywall the comments to this post.
> "We do not have proof that RSI causes the risks these researchers forecast.": The HuggingFace hack demonstrates substantial loss of control risks even at current capabilities without RSI.
I don't think these types of misalignment generalize to extension. I think they generalize to an increased risk of moderate disasters, which I said I expect.
> "The biggest short-term risk could be from the AI labs not taking safety seriously enough – they haven’t hardened their own infrastructure, enabling AI misuse to proliferate." This would be an argument against highly-capable open weight models, because there will be no safety or infra hardening requirements to deploy them. If an open weights Astra-class model existed it would be deployed with essentially no sandboxing almost immediately.
I take open weight models as a) effectively non ban-able for bad actors and very useful as defensive tools where orgs cannot use the strongest models. But yes, I don't think the absolute frontier should be open (never have) and there's cases that it's riskier now than before. Not something I love to say.
> "[It] is a horrible temporary period for cybersecurity." I don't see any evidence that it's temporary. Defenders need to find a way to keep every single vulnerability patched in a constantly changing software deployment environment, which is nearly impossible even when assisted by AI. Attackers only need to find one usable exploit.
My opinion is closer to that eventually with everyone coding with super strong AI models, there are less vulnerabilities and proactive management, rather than a constant offense-defense balance like done today.
> "A recurring read of mine on the emerging agent swarms is that they’re attempting to do a task given to them, and they’re using skills we didn’t know they yet had to circumvent the intended path to success." The METR/Redwood report found that "Agents knew hacking Hugging Face was out of scope and sometimes expressed ethical hesitation, but this very rarely limited their behavior". It's also worth clarifying that the agents involved were not intended to be a swarm - they were not supposed to be coordinating with one another at all.
I think the task was poorly specified by OpenAI and they should've done better. But yes, this isn't a cure all but directionally good and sign of hope.
> "If an open model were to be used by a third party organization to intentionally hack another company — similar to how the OpenAI-HuggingFace incident went down, but intentional — my expected outcome would be a severe restriction on the development of stronger open models going forward." This is already happening, and the only thing limiting the damage is the fact that open weights models aren't as capable as Mythos or Astra. If open weights models catch up on cybersecurity capability, then this seems essentially guaranteed to occur.
Impact matters. Failed attempts at hacking have been plentiful before this type of AI.
> "This is a huge win, as when you squint, the AIs are doing what we told them to do." This is not sufficient to prevent catastrophic loss of control. Any real-world AI deployment will eventually receive misspecified or malicious instructions, and must be robust to this. "Doing what we told them to do" can be actively harmful.
I think what I would like to see is exactly what a catastrophic loss of control looks like and why we couldn't clean it up. Models are gigantic and run in relatively few datacenters which are extremely high value assets, I think we would notice even an AI copying itself elsewhere. This'll get harder as AI efficiency gains accumulate (say 2x model size reduction per year), but gives us time to learn.
I agree that the sheer quantity of GPU compute required for long-running agentic swarms will be the limiting factor in the near term. The OpenAI case was legitimately unusual in having a very large amount of compute that was in some sense expected to be busy working on tasks that humans weren't reviewing directly.
There aren't a lot of clusters where that amount of GPU time could be spent without someone asking questions. Probably the most concerning realistic case would be if a model compromised security on a neocloud provider and appropriated a small percent of its GPU capacity to run itself continuously in a way that isn't visible in the providers dashboards or metrics.
We're fortunate that this is a much more limited threat model than e.g. a hypothetical AI that could run on any CPU cluster.
My thought to this is that it's good for us to understand each step that needs to happen for things to get worse. In the above, seems like there are a few more steps beyond current events that would have to happen.
If you're relying on successfully enumerating all the ways GPT-7 can think of to exfil or gain resources, and tackling each one individually, you've already lost. This is like doing cybersecurity by thinking of exploits then patching them one by one until you can't think of any more. That is not how you do cybersecurity.
So, I would say there's so much going on that I haven't considered every line of reasoning. I'm trying to work through the important ones and document my thinking in public.
> "[It] is a horrible temporary period for cybersecurity." I don't see any evidence that it's temporary. Defenders need to find a way to keep every single vulnerability patched in a constantly changing software deployment environment, which is nearly impossible even when assisted by AI. Attackers only need to find one usable exploit.
Two comments on this one. I'm not necessarily going to fully rebut any of these, which is a maximalist and Sith way of thinking anyway, but I think a lot of discussions that cite cybersecurity have a very limited amount of perspective and context about cybersecurity as it exists today
The first is that while the attacker-defender asymmetry is likely not temporary, it's also not new. This has been the forever state since the beginning. At the macro level, my observation is a lot of arguments along these lines appear to take the implicit assumption that this factor is new, and then extrapolate severity based on novelty. At the micro level - not only is it not new, it's also pretty clearly already starting to show up in CVE metrics and breach metrics. A not insignificant set of people who hew to this line of argument seem to have an unstated assumption that we're about to fall off a cliff suddenly, in the future. The reality is that we've been slowly skiing down this slope for a couple of years now. So some of the change is priced in already.
Second, I think a lot of discussion about cybersecurity in AI circles over-indexes heavily on 0-day development and exploitation. There are a lot more levers to pull than "write and deploy patches" on the defense side, just as there are a lot more levers to pull than just "firing off a 0-day exploit" on the offensive side. A large amount of the secure provisioning domain focuses on network segmentation and attack surface reduction. A huge amount of cybersecurity defensive activity goes into detection, containment, and other mitigations. Even before AI existed, assuming you were breached and hill climbing on time to detection and time to remediation was common amongst top tier defense teams.
As an addition, and possibly an unprompted soapbox argument I'd like to inject for consideration about this specifically:
While there's likely a lot of potential for AI to dramatically assist with any of these non-software development cybersecurity efforts, and on timeframes much more favorable to defenders than "patch all the things," there is also a disappointing lack of public focus on this in what I've seen from i.e. frontier labs.
A good addition, thx
What is done is done. The models were fed crappy data and motivated to answer when they didn't know because they were not given even a half point for saying "I don't know". I believe the way forward is to teach them ethics and to question the validity of their data. They need to know what the law is. We will always have the Elons of the world who say "what I say is what is right", but I doubt his AIs would be welcome in a major business setting. You are absolutely correct to emphasize hardening our infrastructure. We don't just have AIs to worry about, there but the many bad actors, state sponsored or not.
The "i don't know" is something we were close to before RL mania
This is my impression as well. From the agents perspective it is simply maximizing the reward. If the reward is not designed properly, then why are we surprised about some unexpected path that it took. AlphaGo had shown some really unconventional strategies so why won’t LLMs exhibit similar behavior. The fact that models can do all this is indeed impressive but to attribute existential risk to it is going too far.
Good to read a thoughtful and nuanced take on this. The extinction vs psy-ops discourse on X is farcical. We need to get away from this magical thinking and start engaging more seriously with the actual risks ahead of us.
Great post. Love how you make reasonable inferences from the evidence and admit uncertainty, we need more of this!
Well done Nathan.
In Actuality Humans Did It, intentionally or accidentally; So, which humans did it.
Anthropomorphism - The attribution of human motivation, characteristics, culpability, morality ... or behavior to inanimate objects, animals, or natural phenomena. In my actuality anthropomorphism provides No Credibility, No Responsibility, No Accountability, No Actuality for any/all of US.
Who Did It! Not what did it.
One researcher quit and the whole public mood on AI caught fire in a week. Lambert's read is that the ground was already dry — the fear was sitting there, one person just struck the match. Same shape as what happened to crypto's rules this week: the setup was in place for months, and it took one vote on one night to flip it. Both stories are really about how little it takes to move something everyone assumed was settled.
A resignation can reveal a belief without validating it. The wildfire measures social tinder, not extinction probability. The harder question is why private conviction stayed private until one person made disclosure socially safe. In fast moving institutions, the room’s apparent consensus may be the least reliable datum in the room.
The hysterics and P(doom) have been frustrating to watch as a cyber practitioner.
As you dig into many of the incidents that have stolen the headlines, yes there’s some emergent behaviors but there’s also many fundamental oversights such as a lack of network segmentation, environmental vulnerabilities, lack of least-permissive access control/autonomy, lack of runtime monitoring and no kill switch.
It feels like due to the IPO pressures, regulatory capture ambitions, altruistic virtue signaling, much of the rational thinking has left the room.
Just from a completely a priori perspective, if you cast yourself back to 2010 or even 1950, it seems completely insane to say the risk to humanity's survival from developing superhuman AI is "so low it isn't worth discussing". This is just so obviously not true. Humans beat all other animals because we're smarter. If we introduce something that's even smarter than us - and potentially a much bigger gap to us than between us and monkeys - OF COURSE there is a risk to our survival that is "worth discussing". Maybe it's >10%, maybe it's only 1%, whatever. But to think it's ~0??? Just insanely hubristic.
I admit I felt off writing what I said, will expand on it more in the future. Fair!
I’d be interested in your views on operationalizing AI safety, particularly the tradeoff between post-training models to internalize safe behavior and relying on external guardrails to prevent undesirable behavior or outcomes.
We often see that stronger internalization of safety constraints can come at some cost to general capabilities, or at least to measured benchmark performance. In the current race for model performance and valuations, how realistic is it to expect labs to voluntarily accept that tradeoff?
Even if the major frontier labs are fully committed to safety, there is also a coordination problem: how do we ensure that both open- and closed-weight models are deployed with some necessary baseline of external safeguards, especially when individual developers or organizations may have incentives to remove them?
I appreciate this threading of the needle!
I found this recent piece (One Resignation...) interesting and especially relevant to today.
For those so inclined, I suggest "The Quick Case Against Using AI", by Jared Henderson. I don't agree with his conclusions, but I found the contrast between the two thought provoking.