• Why did Kimi K3.1 Pro have a mixed reaction with such incredible benchmarks? Will it be adopted?

• Were we wrong about the risks of open-weight models and cybersecurity?

• Why do cyber experts discuss recent events so differently than AI experts?

• What comes next in the few months? What will change our minds?

We will write more about this subject, but wanted to bring you behind the curtain as we aspire to quickly learn more about the cybersecurity industry and trends across the last few years that have been leading us today. We conclude the podcast with some lighter topics, from Muse-maxing to excitement in still hiring junior technical talent. Please send us questions and topics, as we plan to record more of these podcasts going forward.

Share

Listen on Apple Podcasts, Spotify, and wherever you get your podcasts. For other Interconnects podcasts, go here. YouTube ↓

Chapters

• 00:00 Kimi K3.1 Reaction

• 08:15 Kimi’s open-weight cyber principles

• 11:23 You can hack everything era

• 15:54 PHP2Shell, LiteLLM, and the new internet

• 24:00 Abliterated APIs, safeguards, and trusted access

• 31:41 Anthropic not hiring juniors employees

• 36:16 Florian’s Muse addiction

• 45:30 Wrap-up

Links we discuss

• My thread with Dean Ball about the “boy who cried wolf” story of the frontier labs

• Kimi K3.1 Pro announcement blog

• Kimi K3.1 Pro on Artificial Analysis Intelligence Index

• Kimi K3.1 on Artificial Analysis defensive cyber benchmarks

• Program Bench

• Open-weight Mythos would’ve been fine take

• PHP “hasn’t caused a disaster yet” update

• Abliteration.ai serving abliterated models API

• Morph serving abliterated DeepSeek via API

• Joshua Saxe’s Substack on AI and cybersecurity

• Azeem Azhar, “The internet’s Wuhan moment”

• Anthropic, “How employment is changing at Anthropic”

• LiteLLM security incident report

Transcript

00:00:00 Nathan Lambert: We are back with another Interconnects podcast. We are with Florian, eval guy, recently moved to SF. Florian, is Kimi K3.1 Pro, name is a mouthful, going to live up to the benchmark scores, for example on Artificial Analysis, or where is it going to land?

00:00:17 Florian Brand: In the past, the Kimi models always lived up to the expectations, but this release is kind of weird compared to the others. I think it is also partially because they are becoming more professional, and the previous model drops were kind of out of nowhere. This time, which also got some criticism, they launched a bigger social media campaign, which allegedly also involved paying people to post about the model without disclosing.

00:00:56 Nathan Lambert: I think a lot of companies do this and tell you not to mention it, but it’s a mess. It’s bad form. No company should do it. Essentially, one person tweeted a screenshot of the press instructions for Kimi that said, “Don’t say this is a paid promotion.” And to my knowledge, no one has corroborated it after that.

00:01:15 Florian Brand: Yeah, and there are right now a few of these model launches, including by American companies, that are just very obviously astroturfed. We have seen launches where it’s thousands of retweets and a few hundred likes, because it works so well to get the attention. Some of them disclose that it is a paid ad while others don’t. But in general, we have seen the professionalization of model launches over time, and pretty quickly. A year ago, everyone just dropped the model, then patched vLLM afterwards, so no one was actually able to run any of the open models for a while. Now you get all the inference frameworks on day one, you get all the inference providers on day one. Or, which is also what we see more and more, the model creator has a few weeks of exclusive access before they release the weights. And that’s also what’s happening with K3.1 Pro. I think it’s two weeks of exclusive access before they release the model weights.

00:02:38 Nathan Lambert: They said the weights would come out by November 4th. And the headline things... I don’t know if I’m going to be able to share screen. I don’t have this set up on this computer. But the headline is this Artificial Analysis number where it’s ahead of GPT-6 Astra. And then if you look at the Kimi K3.1 blog, they have their headline numbers, which are really strong, and then they also have some numbers later on which seem a bit more jagged. I think it was Terminal-Bench Science, which is 20 points behind GPT-6 Astra. I think there were a few others, like ProgramBench. Do you know Terminal-Bench Science and ProgramBench? Are any of these things that people think are important benchmarks for frontier models? Or how do you explain the fact that on Artificial Analysis the model is top, what, three, four? It’s the two latest Claude 5.5 models, Muse Wonder Full, and then Kimi K3.1, which looks crazy. It just looks like totally crazy scores, and I think everybody is still surprised by it. But then there’s this long tail of details that you use to evaluate a model now. I had posted a thread on Twitter of, “What are people’s impressions?” And there’s a lot of people that are like, “This seems not great,” and then some people that are like, “Yeah, it’s fine.” I don’t know, there’s a mixed bag. Relative to K3, which I think was pretty unanimously, this feels like a big model and the post-training seemed a little bit undercooked for that. Maybe this is just a tiny bit RL fried. We’ll see where we land.

00:04:26 Florian Brand: For the benchmarks, one of the things is that the benchmarks that are right now used by Artificial Analysis, they will post their new version probably very soon after we release this podcast.

00:04:43 Nathan Lambert: We can’t tell you the exact date, but both of us know.

00:04:48 Florian Brand: In general, the Artificial Analysis benchmarks are rather short in terms of time horizon, whereas something like ProgramBench is really long and also very explorative. Other benchmarks are telling the model, “All right, this is the goal, and I want you to reach it,” whereas ProgramBench is more like, “All right, here’s what I want, but you need to explore a lot.” And I think that might be the reason that we see some differences: their RL training is still more towards the exploitative side of model training versus the explorative side, where you need to find the solution somehow before you actually know what you’re doing.

00:05:41 Nathan Lambert: But if you’re using a three trillion parameter model, I feel like you kind of want it to be pretty robust at exploration. Because if you’re going to want to use a model for pretty known coding tasks, like using DeepSeek-V4.1-Flash-1017, which is not that far behind on these scores, it’s just going to be way faster. It might not be wildly different on cost per task with the actual token numbers. That’s why I think it’s pretty reasonable criticism of this Kimi release: a big model has got to be... Because it’s going to be slower, it’s got to be like Fable, which is just... I still think Fable is really nice if you’re doing very blurry things. I revert to it when checking my slides over sometimes, where I just feel like it kind of gets me a bit better, where Opus is really, really grindy. I think the open-weight models are better in this kind of grindy perspective, because you can just get them to be faster and cheaper.

00:06:44 Nathan Lambert: It seems like Kimi is gearing up for this Flash-style release with their naming change. Previous Kimi models were just K2, K2.6, K3, and now this is K3.1 Pro. So a lot of the company forecasting, investment-y side of interest on Twitter was, a lot hinges on this for Kimi, because it’s kind of true. We’re not the inference experts, but serving the model seems like you’ve got to commit way more resources to serve this three trillion versus this 700 billion parameter model. You’re going to need NVL72 racks, I’m guessing, to do it well.

00:07:21 Florian Brand: To fit the model, you need B300s, one node per model replica, which is kind of awkward, versus other models which you can even fit on H100s, or H200s rather, which are more readily available. Splitting models is always awkward compared to having one model replica per node, or using existing nodes to fit the full weights. So in that regard, three trillion is just out of, or just in the range of what you can host somewhat reasonably, but it is kind of annoying compared to all the Flash models and all the smaller models, where you can use so many of the existing GPUs that people have.

00:08:15 Nathan Lambert: Do you think Kimi’s shooting itself in the foot a little bit by saying it’s slowing its own cyber capabilities? In the blog post, I can read some of this, because it’s really good for people to say. “Open-weight principles. These evaluations frame our open-weight release decision. First, we aim not to release open weights that would extend the absolute frontier of cyber capability. K3.1 Pro’s offensive cyber capability is, by design, below what our pipeline can produce, and remains behind that of frontier proprietary models.” And they go on to say... I don’t know if the mid-training part is in this, but essentially they manipulated the mid-training data to down-weight cyber capabilities and see that that can last into the post-training. Do you think this would reflect upon the other benchmarks that they have, and maybe some of this long-horizon weakness? Or do you think that people could actually separate this?

00:09:08 Florian Brand: I think people can very well separate these things. I talked to a bunch of other people, and I think that style of release is really good. You want very broad access for everyone to find all the bugs and to patch all the bugs, which is net positive for the world. And Kimi saying they still release the capable model for the defensive side, but they don’t want to advance the offensive side. What I’ve heard from cybersecurity professionals is that exploiting and being able to go through or wreck a whole organization is way harder, because you need very long-horizon capabilities, and you need to persist through all the stages that you penetrate. That’s where closed models, especially GPT-5.6 Sol, a model that’s named often in this regard, are just way, way better than open models. And I think that is important. We want all our code to be secure, and that should happen pretty fast. But for this, you don’t need long horizon. You don’t need all the CTF-style challenges. So throwing these out of the training mix is a reasonable decision, and it helps them position against the fearmongering that once an open model with Mythos-level capabilities, which we arguably have in some regards, is released, the world will end. Which we haven’t seen so far, and I don’t think will happen in the near future. So open models get way more scrutiny than closed models in this regard, and them positioning themselves as, “We are the responsible actors,” is a smart move.

00:11:23 Nathan Lambert: Okay, we can transition into the “you can hack everything” part of this segment. There’s a lot going on here, and Florian and I are both not cyber experts, but it is worth discussing, because I’m kind of of the opinion that the cyber experts that I read talk about this from an ecosystem perspective. All measurements like vulnerability detection, time to exploitation, have been on these exponentials before ChatGPT came out, and there wasn’t a step change as AI models came out, and they focus on this ecosystem perspective. But when you come to the AI people, which is our corner of Twitter, people love to debate open versus closed models and Mythos and things like this. And then the messy middle is, what is the effect of the Daybreak programs and Mythos and things like this on broader society? I feel like I’m trying to muddle through this without getting too stuck into any one point of view. But I really don’t think it is that much of an open versus closed and a capability model thing.

00:12:33 Nathan Lambert: I wish I could go relive the Mythos moment, because my impression was that they made it quickly seem like a cyber weapon. I can go revisit their communications about what they actually said, because it might have been more that they were like, “This is a step change in cyber,” and then the government was very scared of it and took it very, very seriously, and then that pipeline spiraled into this communications of the AI safety community potentially forecasting more problems. It’s hard to hold people accountable for what the media ecosystem does, but also if you are the most prominent company in the world, you are partially responsible for how all of this works. So I’ve gotten pushback from people in the program that are like, “We didn’t say that the world would end when Mythos came out. We just said that there’d be a ton of problems.” And in some ways they are right that there are a ton of vulnerabilities being found. But I do distinctly remember the boy-who-cried-wolf vibe of a lot of the AI safety companies over the last few years. I had this exchange with Dean Ball where I was trying to point this out to him on Twitter, and I was like, “Dog, we’ve been doing this for years.” The boy who cried wolf is a classic parable for a reason, and it seems like a bit of a blind spot, and I just can’t understand it.

00:13:50 Nathan Lambert: But we’re in it, and the vulnerabilities are being found en masse, and I would say there’s a slight uptick in visible hackings. It might be that there’s more hackings within the Twitter community. A lot of cyber people I’ve talked to use reported cyber damages, which a lot of them are through ransomware, as the trend to look at. Historically, AI has been attributed to a very, very low percentage of cyber damages. So if cyber damages do skyrocket with attribution to AI, that is a very difficult world. But the vulnerability finding is very, very unsettling, but we don’t know exactly what it’ll translate into. That’s where I stand, and the reason as to why that translation might not happen. For example, I like Joshua Saxe’s work a lot, and he talks about how the supply chain of cyber hackers is very weak. These people tend to have competition. The population are people that self-select because they cannot get better jobs, and this population of people might also not be people that are willing to do cyber attacks that I would describe as loud and very high damage. So I think there’s a very high likelihood that nation-state actors are using AI to collect information and infiltrate at a large scale. But it’s not clear who the people are in the world that are willing to use AI models to cause mass damage in the physical world. And I don’t like that being... That’s a horrible thing to be like, “Okay, we’re going to figure out whether or not that’s true in the next four months,” but it literally might be where we’re at with the number of vulnerabilities out there. And I think you’ve been following the PHP issue, which I don’t know if you want to summarize for people, and then also some of these things that you’ve been following in the commentary after that.

00:15:54 Florian Brand: So the PHP issue is basically that for years there was a bug which allows you, in PHP, to execute code remotely. So anyone can use any PHP website, which are a lot of the websites on the web. WordPress is one of the biggest ecosystems which runs on PHP. And the OpenAI Patch the Planet program, together with some other AI security researchers or cybersecurity researchers, found this bug and then reported it to the PHP Foundation, worked with other actors to patch this, and then released the patch all at once. Because PHP powers so many parts of the web, and it is a really, really strong exploit, really high risk, a lot of websites even went offline for a while just to patch this, which is the right response. And obviously because it was so high profile, and people think about Mythos and think about open model releases, that has kicked off a number of takes that now everyone who can run an open model basically can hack the world. But the interesting thing is that there are numerous reports that the mass exploitation hasn’t happened and isn’t happening yet.

00:17:37 Nathan Lambert: Wait, did you see the tweet about the DeepSeek swarm fixing it?

00:17:40 Florian Brand: Yeah.

00:17:42 Nathan Lambert: I don’t know if it’s real, but somebody that works at a big cloud company was like, “We’re literally seeing mass fixing at scale by somebody, or a DeepSeek swarm.” Great meme. Coin flip on if it’s real.

00:17:59 Florian Brand: But in general, the expectation of some people seems to be that the second it drops, everyone is able to write code and exploit and is able to scan all the websites and do mass damage, and that doesn’t seem to have happened, which is great for the current state of the world. And I think we will see similar things in the future, so very quick fixes by responsible actors, which is the vast majority. Like you have said, the people who work in cybersecurity are experts at their job. They get those jobs, and there is so much money in fixing things. All the cybersecurity firms, Anthropic and OpenAI, all the AI players as well make a lot of money with these programs, so single nefarious actors are the minority. And nation states, I would say, are completely out of that question, because if you are a nation state, you have completely different capabilities. You might run completely different models and abuse these things for a long time. So I wouldn’t say it’s fair to say that now that open models are becoming better at these things, nation states didn’t have these capabilities for a while, or can’t train models to advance their own capabilities. We are basically seeing that the overall dynamics are staying the same, but are accelerated by a lot.

00:19:49 Florian Brand: The other interesting thing, especially if you think about things like ransomware, is that people still haven’t woken up that they need to patch their systems. I’ve had a few friends who work at very small or medium businesses, and they are like, “Oh, yeah, that sounds horrible. Let me check my PHP version.” And it is so far out of life that they don’t get the security patches, because they didn’t update their PHP versions or their systems for years. So they are vulnerable to a bunch of other security bugs, which still weren’t exploited, which is also very interesting. Another thing that plays into the argument that right now the amount of nefarious actors is very small, and things can stay accessible for way longer than people think, is the LiteLLM hack. LiteLLM is a software library where you can use one endpoint and then ping every provider, which basically allows you to save a lot of lines of code. And they got compromised again this week, which, with all the PHP stuff and other security vulnerabilities, just contributed to the impression that everything is insecure. They did a preliminary report that the reason why they got hacked again is that they had an API key committed to GitHub for five months. So it was accessible in some way, or it was readable for five months, and five months later, someone exploited that and used that to spread malware.

00:21:44 Nathan Lambert: I don’t think we’re in a world where we’re defense dominant, but I don’t think that offensive capabilities being higher is necessarily a lot of the bottleneck, which is very messy. But that’s probably why it’s not changing that much, because yes, the cost and ability to do cyberattacks is going way up, but the institutions that are doing them, that doesn’t necessarily 10X their output in what they’re doing. But the defensive side is worrisome. Azeem Azhar had a good piece on Exponential View, which was just, this is day zero of the new internet, and potentially things will never be as secure and open again, because people are writing exponentially more code with coding agents. There’s just so much more code going out there, and I don’t think flaws and vulnerabilities work in a way where the model writing the code can prevent all the vulnerabilities, because it’s a lot of interactions between software and how you stack these things on top of each other, and I think you need to give the models more context they won’t always have. So this is people’s view, where there’s going to be way more vulnerabilities going forwards, and this transient period of a few years is where we have the backlog. We have the problem of the backlog and exponentially more code being written, so it does seem pretty likely that there’s just going to be way more vulnerability for the next few years, which is not good.

00:23:17 Nathan Lambert: But none of this seems like an open-closed model story. How do you bisect the timing and the forecasting of these things with the fact that open models are going to keep getting better, and that closed models are going to keep getting better while having imperfect safeguards? I kind of feel like the communities actually aren’t doing a good job of saying how the other one informs them. Some people ignore the cyber community, and some people in the cyber community will be like, “This is on trend.” I still don’t know how to fully parse it, because it’s all so overwhelming.

00:23:54 Florian Brand: I would broadly agree with that take.

00:24:00 Nathan Lambert: I’ve been getting a lot of pushback for my open-weight Mythos in April would have been fine take, which was essentially... This is a few weeks ago when I posted this, buried at the end of a cyber article, and at that time there was so much fearmongering that open-weight models were going to cause all these problems and cause all these cyberattacks. A lot of people are essentially saying, “Look, we were right. There’s all these cyber exploits due to open-weight models.” But causality is so hard. I think it would be good to talk to people that are pals with big tech company security leadership and understand, are companies actually deploying open models for defense? I don’t think there are that many companies that are going to be proactive on cyber defense. I get the sense that it’s a pretty reactive industry. So I don’t know. But at the same time, there are these API companies that are serving abliterated models over the API, which seems like just asking to get sued by the government. I don’t know why you would do this. I think the abliterated models for defense is kind of overblown, but it is very real that people want to have models on their own infrastructure so they don’t need to send anything over an API. So to have an abliterated model that you need to send information to over an API in order to secure your systems... I think one of the companies is literally Abliteration.ai, and the other one is Morph. I think it is a bad judgment call for these companies to host these models at this time.

00:25:43 Florian Brand: I also think it remains to be seen how much the abliteration really helps versus it just being very effective marketing. I think the whole thing has shown one thing, and that might play into your open-source Mythos would have been fine take, which is that the diffusion isn’t happening fast enough. People are struggling to get access to, and especially back then were struggling to get access to, the cyber programs from Anthropic and from OpenAI. So they had to use open models. It is obvious that the closed models are just better at these things, but if you can’t get access and get blocked with every request, what should you do? You need to use the less capable open models. And it seems like that has now changed for Anthropic as well. Getting access to the defensive cybersecurity program from OpenAI is rather easy. You submit your ID, you have to secure a system with hardware keys, and then you basically get it, just for the defensive side, for securing your code. And I think that’s amazing. If someone was to abuse it, OpenAI can just look up who the real person behind that is and, in the worst case, go after them or talk to law enforcement. The offensive program is still very hard to get into. And from what I’ve heard, Anthropic is also broadening the access now to more and more people. That’s just positive, and the thing that will change the landscape, because more security researchers and more people being able to sift through so much software and the whole backlog to secure our code is the thing that is needed in the end.

00:27:49 Nathan Lambert: I do think that these programs are a big reason why we’re seeing so many vulnerability disclosures. The PHP disclosure was in the works, and people were in the process of fixing it when it got leaked. I would put more likelihood on people using these closed models easily with trusted access programs to do a lot of those, and I think that companies deserve credit for this. The open-weight model stuff is just really messy. We’re only going to see it when there’s a disaster, and it’s really hard to see usage. I still do think if there was a DeepSeek swarm that hacked an American company, the political fallout would be insane. Maybe some of the politics of this are just going to be slower due to the midterms coming up, where everybody’s focusing on midterms and they’re not focusing on AI right now. I wanted to revisit my open-weight Mythos would be fine piece, and I think the thing that I didn’t do a good job of articulating in it is just what the trend lines forecast in terms of vulnerabilities accelerating, because those trend lines existed. I could have done more learning about cybersecurity ahead of time to be like, “Those are going to come,” and to price that in. It seems very, very ecosystem dependent, rather than dependent on when we get a certain capability diffused widely, at this point. I don’t love the balance, but that’s what the evidence I’m seeing is looking at. So I do think there’s still going to be political challenges ahead for open models, even though a lot of the facts don’t support the evidence right now.

00:29:28 Nathan Lambert: I made a slide in my talk that was just screen grabs of closed models hacking companies and stuff, and there’s so many of them. I think the closed model safety notion is really, really suppressed right now, with models in training being able to hack people, people getting around the safeguards very easily and hacking companies. If you had taken us back a year ago and asked especially people at labs about that, they would have been like, “No, that’s not going to happen,” I would think. And I don’t think that is fully updated across the community very well.

00:30:07 Florian Brand: One thing that I like is that OpenAI and Anthropic seem to have updated their safeguards, because they can update them at a moment’s notice. It seems like they have rolled out an update to their safeguards that basically blocks any try to get the real exploit for PHP2Shell. If you mention, “All right, I want you to look through this code,” the safeguard will just block you off, to stop people that are just trying to get the exploit very easily in their tracks. I think that is a useful showcase for why safeguards for closed models might work better, or are working at all.

00:30:56 Nathan Lambert: I hear these things, and I hear things like researchers who are trying to do cyber research that are like, “Yeah, I haven’t been able to use any agents after Opus 4.8, because I get rolled back immediately.” You hear these, but you also do see the news that’s like, “I used Claude Code to hack OpenAI.” I think there’s still a gray area in there, where maybe the least motivated hacker is blocked by the guardrails, which is the teenager being like, “Can I steal my friend’s parents’ money out of their bank account?” That stuff’s getting stopped, blocked, which is good. But I do think open models, to a large extent, also block that type of thing.

00:31:41 Nathan Lambert: I have another rant. What do you think about Anthropic saying they’re not going to hire junior employees?

00:31:48 Florian Brand: Well, I mean, that’s great for everyone else. So they can hire all the junior employees that have very bright ideas and are very hungry to work. When we visited China, one of our takeaways was that if you just hire all the young PhD students who work their ass off all the time and are super happy to get hands on a cluster, you catch up really fast, and those people also have very bright ideas. So yeah, it’s pretty good for everyone else.

00:32:24 Nathan Lambert: That’s pretty funny. My take is, Anthropic does all this work to talk about how they want to do all this ambitious stuff to prepare society for the transition to super powerful AI, and I’m like, the number one thing you need to do is keep training people so that they are fulfilled and have things to do with their life. Yeah, taking risk on a bunch of junior employees isn’t great, but I think it’s more about the culture of too-short timelines in terms of building technology, where they’re like, “We just need to build this. I don’t have time to talk to people.” It’s another red flag on their safety culture: safety is a much broader thing. You’ve got to have your chips covering a lot of issues that matter, and a lot of these things will come back to bite them, I think, in the future. We’ve seen a lot of pushback to the labs and how they communicate things in the past, and I think this is just the next brewing issue, which is when there is a job correction or slowing jobs due to AI, and the companies, they look back six months and they’re like, “Anthropic decided not to hire any junior employees six months before a bunch of news on youth unemployment.” It’s not going to look good, and I feel like they definitely have the money. So many scientists I know that end up there like doing this. It’s a fun part of the job to have these relationships that you build over time. So I thought it was another sign of strange culture at the labs, and they need to touch grass and realize that it’s going to take a long time for things to change, and I think that RSI has limits.

00:34:05 Florian Brand: I think we haven’t seen mass job displacements, and not even mass job displacements in the software sector. Everyone is just hiring more and more, because you can do so much more with coding agents, and everyone seems to be working more and more. So it remains to be seen how fast things diffuse in the real world. But again, for everyone else, it’s fine that the juniors that might apply to Anthropic are now looking elsewhere, because there’s a lot of talent that is just hungry for the right opportunity.

00:34:50 Nathan Lambert: Very curious to see, what do you think is coming in the next few weeks? There’s rumors of more open models, and we’re going to be riding out these waves of vulnerability announcements, with more sporadic hacks. Are OpenAI and Anthropic going to keep releasing models? I guess the IPO could potentially be soon. It’s just very weird to have all of these happening at the same time.

00:35:23 Florian Brand: I think we will see the trends continue, so more disclosures, more high-scale things that everyone needs to patch. A lot more open model releases, especially from the US, which is great for us. We’ve been following that for a long time, and now we seemingly have a month or two ahead of us where everyone will just drop one open model after another. The Chinese players are also gearing up for new model releases, which again is just how we operated for a while, but I felt like the past month or two have been relatively quiet, and it seems like now things will pick up steam by a lot.

00:36:16 Nathan Lambert: Where does Muse fit into this? Florian doesn’t grocery shop, doesn’t order his own food, and has no budget.

00:36:29 Florian Brand: So now that I moved to the US, I can actually use Muse and all the other AI things that are blocked in the EU, so it is kind of interesting to get everything at once and all the functionality in my operating systems. One of the things I tried very hard to... I’ve read a lot on Twitter, so I needed to finally try this, is Muse. I could set up OpenClaw, Hermes, all these things myself, but I didn’t want to tinker around with it necessarily, especially because these things need constant updates, and I don’t trust myself with setting up the secure infrastructure, especially if we talk about money. There are a bunch of other alternatives, but I trust Facebook enough because they have so much experience with these things, and as soon as something happens with Muse, like something gets leaked, their stock would just crater and they would get sued into oblivion. So they have a higher risk, which makes things very secure. So I set up Muse. I also set up Dots at the same time, to just play around with them.

00:37:53 Florian Brand: I am somewhat annoyed how good Muse is. Although it’s not the most frontier model, it is good enough for a lot of things, and it is very helpful with its proactive messaging. I have to deregister a lot of things in Germany. I have to sign up for a lot of things in the US, like Social Security number, bank accounts, all that jazz, which also means I need to have a lot of different papers, different forms. Muse messages me proactively with, “Oh, I saw that you have that appointment. You need these forms. You probably have them because you have an O-1, and that form is where you got it. That’s how you download that form.” That’s been super, super helpful, whereas Dots is like, “Oh, yeah, so two days ago you messaged that guy on Slack, and I have resolved that two days ago. So maybe you want to check in again.” And I’m like, “No, I don’t want to.” The other nice thing from Muse is, I obviously also need to buy a lot of stuff because my apartment is empty, and I don’t know which websites are useful, which brands in the US are good. So I just ask it, “All right, I need to stock my whole kitchen. I need pants and so on, so get me those things.” I gave it my credit card, and it just runs with it.

00:39:29 Florian Brand: It gets the whole shopping cart going, not on Amazon because Amazon blocks them, and then I check out with a Stripe link, which creates a one-time credit card, which is also amazing. If Muse fucks something up or the shop fucks something up, it’s not my credit card. It’s a one-time credit card which doesn’t exist after the transaction. Then I get the things after a few days to my home, and that’s been very helpful. One thing that is annoying is it is kind of slow, so if you do time-sensitive things, it’s just faster to visit a website yourself. And also, it has some issues with the browser. Sometimes its browser crashes, and then it has to rebuild everything, and because the model is slow, you lose 20 minutes, which is fine if you are doing things asynchronously. Some websites start to block the data centers of Muse or try to throw things in its way. CAPTCHAs are not a problem. Muse very happily circumvents any CAPTCHA it gets thrown in its way. But that’s been a lot of fun, and the more I trust it and the more context it has, the more useful it becomes. So I am kind of shocked how useful I think it is.

00:40:54 Nathan Lambert: How much do you need to give it to get out? I’m guessing you need to give it email, calendar, and email that has a whole bunch of this life crap in it, and you just kind of need to let it do shit and see what happens.

00:41:08 Florian Brand: First is that you can import your memories from any system. Everyone tries to lock you into their memory system, and Muse basically circumvents that by giving you a prompt that has the other memory system, so in my case ChatGPT: “All right, tell me everything you know about me.” Then you get the export from ChatGPT, and that is then fed into Muse. But the more things you give it... I have used iCloud as my mail provider for over a decade, and Muse and Dots don’t have an integration with iCloud, so I asked, “All right, can I set it up?” Both said no, but Muse also immediately suggested, “Hey, you can just log into a browser, and I can monitor that instead,” which works. But on the other hand, Muse always asks for permission to access iCloud because it’s a sensitive website, which, again, is great, because I trust Facebook to do these things properly. It was kind of annoying, so I set up email forwarding to Gmail and used the connector. And now, because I’m buying stuff and I’m telling it stuff, it gets a bunch of emails it can search through, and it remembers, “All right, you bought these things. You have signed up for these things. You have confirmation about these appointments.” And then it becomes more and more useful over time.

00:42:37 Nathan Lambert: Maybe I’m a boomer, but I feel bad having an AI agent call my prospective new dentist.

00:42:47 Florian Brand: It cannot call things. And the thing I do is, as soon as it would interact with real people, like support or whatever, I take over and I write things myself, because I think it’s a sign of mutual respect to each other that any human connection is still genuine. But dealing with support forms or other chatbots or all these crappy systems, whatever. I don’t want to click around these things. I just find if an agent does this, I want to save time.

00:43:24 Nathan Lambert: I would rather, if somebody was going to send me an AI-written email, it go to an AI before going to me, because I now signed up for the Pangram thing where you pay $20 a month and it checks every email if it’s written by AI. It gives me free reason to just delete more emails which were crap. The most fun part is that there are people that I know closely, and I’m like, “Huh, you’re using AI to write your emails? I thought you’ve been emailing me for years,” which is really funny. But it is nice metadata to have. I don’t know how we get from... Is it Muse, from people just getting used to sending each other’s Muses things? It just seems like a really hard cold start problem. Instead of listing Gmail on my website as a scientist, can I list my agent’s contact for other agents to find? Maybe it’s not even visible, but I think that’s the thing we have to get to: how do I set up the front door for agents for all of my web and so on? I guess some of the payment stuff is there. It could be like, “Pay me a dollar for your agent to go send a message to my Gmail,” and set that up and see what happens. So it’s good that Muse is breaking down these boundaries, but I’m like, “I don’t know if I need it.” I’m busy, and I like when I get to go grocery shopping. At some point, I will surely dive much more fully into what I’m wearing.

00:44:58 Florian Brand: My situation is also kind of unique in the respect that I’m in a new country. I don’t know anything about anything. And having AI to help me with those things has saved me so much time and so many annoyances that I’m very happy about that. Every time I cross-checked things, because I found that things are kind of weird, Muse was always spot on. So that’s been great and really helpful for me.

00:45:30 Nathan Lambert: I think for refunds and dealing with crap... I got married, and I’m trying to extract a domain from my wedding website to Porkbun, where I keep my domains, and you have to go through customer service. I think this is the task where I’m going to be like, “Muse, you’ve got to go get me my domain back from this cruddy website that I used.” So I’m getting closer. Maybe I’ll try that right now. Do you have any final hot takes? I am going to see You Can See Everything pretty soon after this, so I’m pretty excited about this.

00:46:11 Florian Brand: Nice. I think with regards to hot takes, especially all our cybersecurity stuff, we are set for a while.

00:46:23 Nathan Lambert: We shall see. Okay. Godspeed, everybody. I hope you don’t get hacked. Keep disclosing your vulnerabilities, and whatever you do, don’t blame DeepSeek. We’ll end on that. Good to see you.

00:46:40 Florian Brand: Bye.

1. MOFCOM, the Ministry of Commerce of the People’s Republic of China, released a policy proposal (it’s not yet a law) that would require frontier AI models and some other technologies to get pre-approval from the government before they can be shared internationally. We found the reaction to this to be overblown – as the document states that open-source technologies will be set up for automatic approvals – but many in geopolitical circles say this is a big deal because it proposes a regulatory framework that could be used in the future to regulate China’s AI industry.
   
   Others we trust on this issue say this proposal could be more about preventing issues like the sale of Chinese startup Manus to Western companies, rather than being about frontier AI models. Either way, the wheels of control are moving around AI, and open-weight models are not free from any influence.

2. The Department of Homeland Security (DHS) released an order on DeepSeek, which restricts DeepSeek itself, as a company, and DeepSeek’s models, as a technology platform, from being a vendor in federal civilian procurement. It is not clear how this will relate to finetuned models or models trained on the outputs of DeepSeek (e.g., Nvidia’s Nemotron models have long relied heavily on DeepSeek models in their training recipes – will they be banned?), but this is an expected action and reinforces the need for a domestic open model ecosystem.
   
   Other model makers in China, such as Zhipu AI, makers of GLM, are already on the Entity List under Commerce/BIS, which restricts their access to American technology.

DeepSeek followed the above news up by releasing their internal inference stack publicly, DeepInfer, which is likely far more efficient than existing open-source tools like vLLM/SGLang for their models. Estimates point to DeepSeek making up to 90% gross margin when selling tokens on this inference stack. It is well-known that the top AI labs have much more inference-efficient tools internally, and this is the first time we’ve seen a leading lab share them. Inference companies will be quick to adopt these tools, which will drive prices for DeepSeek’s models down across the ecosystem (or margins up). While not an open model or dataset, we consider this release to be one of the bigger open-source AI artifacts in the last few months.

Share

As of writing this, Kimi K3.1 is rumored to be released very soon, with metadata associated with it starting to appear in public libraries. Exciting times!

You can view the models in this email, and all previous issues, on our Artifacts Hub. It includes commentary and other ecosystem metadata.

Browse the Artifacts Hub

Our Picks

• MiMo-V2.6-Pro-RL by XiaomiMiMo: an update for Mimo, which reuses the same bases as MiMo V2.5, but used a lot more resources on RL training. They also release an in-depth tech report, as well as some of the used environments, which is rare at this scale. The final model from this recipe, V2.6-Pro, is the top open model on Artificial Analysis’s ranking today, and the scaled RL runs came with a fun live dashboard of training. We’ve been fans of Xiaomi for months now, and their efforts are bearing fruit.

  

• DeepSeek-V4.1-Flash-1017 by deepseek-ai: Since the latest Artifacts, the whale has not released one, but two new model iterations with a new size (552B-A16B) which also support image inputs. The model has gained a lot of (well-deserved) attention for its performance at a small price. It also carries the MIT license, continuing the trend established by DeepSeek themselves. This is notable in times where licenses seem to become more bespoke and exclusionary. Below is the Pareto of performance vs price on DeepSWE, a favorite agentic software engineering benchmark today, which DeepSeek dominates.

  

• clef by Cloudflare: Decision models are all the rage after being introduced by the startup Typesafe. In short, they are general classification models that output few tokens with their probabilities. Its usefulness has led to quick adoption and also a lot of open model finetunes such as Clef.

• NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 by nvidia: The mid-sized model of the Nemotron family got its long awaited update. It is strong for its size and can be run at home (given you own a beefy laptop or RTX Spark). As usual, NVIDIA releases the vast majority of the data and an in-depth tech report.

  

• Beam-1.0-501B-A23B by reflection: The long awaited open model by Reflection, making it the strongest open American model released under a permissive license (Apache 2.0). While not the smartest open model available, Beam is very impressively token efficient (second to only GPT-6-Astra below). This is a sign of a well-developed post-training stack.

  

Models

General Purpose

• Ternary-Bonsai-2-27B-gguf by prism-ml: A ternary version of Qwen3.8-27B for on-device deployments.

• Ornith-1.5-397B by ornith-ai: A series of Qwen3.5 and Gemma 4 finetunes that gained attention in the local AI community for its strong performance over the respective (strong) baselines.

  

• Ling-3.0-flash-VL-fp4 by inclusionAI: A 124B-A5.5B model by Ant Ling (within Ant Group), which continues to churn out models consistently.

• Muse-Spark-Open-816B-A42B by meta-models: META has released their open version of Muse, which we have written about in more detail!

Read more

In many ways, the gap between the best American and Chinese models has been reduced roughly by half. We’ve gone from the best American open models being largely irrelevant to consumers who require the cutting edge of performance, to them being behind but relevant.

This paints the picture of the cut-throat, competitive nature of the frontier model business. Reflection and Mistral announced their model scores and Meta dropped the weights of a smarter model before these other two labs could get their own weights out. All of these models are competing with Chinese models released months ago, where the industry is collectively asking — where have Kimi and Zhipu AI been since the releases of GLM-5.3 and Kimi K3 on Aug. 18th and Jul. 16th, respectively?

That is a very nuanced question, which I’ll dig into another time. Today’s piece is focused on Meta’s turnaround, what Muse Spark Open means for the ecosystem, and the balance of power across Western frontier labs.

Share

Why Muse Spark Open should take off — and why it may not

One thing we see again and again at Interconnects through our open model analysis is that Western open-weight models get astonishingly more adoption than Chinese models with the same performance-to-size footprint (even though the Chinese models usually come out first)! Muse Spark Open is landing into this environment, at the same time as when open-ended LLM agents are finally real (~GLM 5.2 capabilities and on).

Riding off a rapid increase in performance of Chinese models, the open ecosystem has been living through an acceleration in demand. The inference providers are rapidly scaling, the open post-training economy is forming, and countless enterprises are looking to own their intelligence. Muse Spark Open’s performance falls somewhere above GLM 5.2, near DeepSeek V4.1 Flash, and behind GLM 5.3 or Kimi K3. It should be squarely “good enough” to slot right into this.

For some background, from the Muse Spark Open model card on Hugging Face:

Muse Spark Open is an 816B total/42B active parameter open weights model with a context window of 1M tokens. Built for coding and agentic work, Muse Spark Open is natively multimodal. This open weights version has additional safety post-training to make it more robust against misuse as a standalone model without system guardrails. We recommend deploying Muse Spark Open as part of a broader AI system, with additional guardrails suited to the specific use case.

Here’s the distribution of the various Muse models on the Artificial Analysis Intelligence Index.

This is included to loop back to the original model card snippet. It was shared that this open-weight model underwent additional safety training to prepare for this open release. Many observers saw this release as potentially a model that was “broken” during the safety training, as it came out after Muse Spark 1.3.

In reality, this was much more likely a safety tune of Muse Spark 1.2, which would only be a 1 point drop on Artificial Analysis to shore up safety behaviors. When Meta released Muse Glimmer, the 30B agentic model, Alex Wang said that they were preparing to release the Muse Spark 1.2 model as open weights, which fits with the above.

If anything will hold Muse Spark Open back, it’ll be the specific license they designed for it. In summary, this custom license has pretty much all of the restrictive license behaviors we’ve seen (and criticized here) in the last few years of building with open models. They are as follows:

1. Users of the models in products must display “Built with Muse” on their products. Meta did this with Llama, and Kimi has tried it, but almost no one complies.

2. There is a revenue-gating factor, where companies that make more than $20M per month across their entire business deploying the model need a separate license with Meta. Other prominent open model providers are doing this too, in order to set up revenue share agreements for re-selling inference.

3. Muse Spark Open makes you agree to an acceptable use policy, which in the true “open source software” corner of the world makes this model extremely not open-source.

4. All of the above applies to outputs from the model as well, and derivatives, so there is legal ambiguity if you can safely use the model for synthetic data.

The license does permit using Muse Spark Open for internal research and applications.

We’re at a weird equilibrium where this license is complex and unproven in court, but many players will likely ignore it. Many, many people will build on this model from various inference providers, and Meta will only likely go after egregious violations.

For now, licenses haven’t been the determining factor of AI adoption, but this could change in the future. This is why we think it’s an unnecessary strategic risk for labs trying to compete and regain adoption to have complex, non-standard open licenses for their models.

Is Meta a tier 1 (OpenAI/Anthropic) or tier 2 (Gemini/xAI) frontier lab?

Just a few days before Muse Spark Open, Meta unveiled their closed frontier models, Muse Wonder and Muse Wonder Full.1 On traditional benchmarks, these are very strong models only behind Anthropic’s Opus 5.5 and similar to OpenAI’s GPT-6-Astra or Google’s Gemini-4-Argon.

These major releases are coming just over a year out from Meta’s massive AI redirection, characterized by CEO Mark Zuckerberg’s Personal Superintelligence post, which got a lot of criticism at the time. Meta is the only American company with frontier open and closed models. Alibaba’s Qwen has followed a similar strategy, but with a far more open-leaning approach.

The vibe checks across all of these models have been solid, but the expectations were for them to not immediately challenge the level of refinement of the most recent Anthropic and OpenAI models in real-world usage. I’m excited to try them more — time to sign up for another AI subscription.

I like this recent run of form from Meta, driving their own products and directions in the popular Muse agent, taking wins like the leading US open-weight crown when useful. Whether or not it’s financially successful, it’s a unique perspective that fits their history as a business.

It often feels like there’s a revolving door of which lab is nipping at the heels of the Anthropic-OpenAI leading duo. It’s not clear that in the long run Gemini and xAI are out of it, but those two are much more muddled in their offerings than Meta today. If you listen to Zuck and Alexandr Wang on The Cheeky Pint, they clearly had a very specific vision of a high-ownership, high-compute per researcher lab that I would expect to pay off. Many labs die due to layers of bureaucracy and misaligned incentives. Meanwhile Alex Wang has over 200 direct reports — i.e. it’s a very flat org with minimal management — and they’re releasing very consistent model and product improvements.

Leave a comment

Much of my worldview of building models is that you need the resources, the compute, the data, and the talent (all downstream of large sums of money), but organizing them is very hard. Gemini is sometimes described as a place where it’s impossible to get any work done or grow — which is embellishing the issue, as many people love it there, but there are kernels of truth to it. Layers of management easily get in the way of making a great model.

While I’m praising Meta today, the Llama failure has often been described as that sort of management going too far, with model decisions being made out of politics rather than performance. xAI is in its own lane, where it may be closer to having too much talent turnover or not enough organization to sustain great models for a long period of time.

When we look back at the last few years, all of Meta, Gemini, and xAI have produced models pretty much at the frontier of performance, but what has distinguished OpenAI and Anthropic is their consistency in maintaining frontier status. Anthropic seems the most poised and organized, with a plan and vision for how these organizations should work, and they’ve gained a lead from it. OpenAI has been more hot and cold, but manages to keep up at the frontier of deployed models while being the leader in public scientific breakthroughs (from Navier-Stokes to o1’s inference time scaling).

Here’s the state of play among the companies that this post focused on, plus Xiaomi V2.6 Pro representing the leading open-weight model overall. The ranking would be Anthropic > OpenAI > Meta ~ Gemini > xAI ~ Chinese Open Weights > U.S. Open Weights.

Share

Open questions for the AI industry

After a long time of predicting the frontier model game getting too expensive for many to keep up, we may be seeing this play out among Western models. We seem very far from “one person wins” with RSI given how close all the models are to each other, but we’re narrowing down the number of labs who can compete in the US from a much wider net with surprises, to the 5 of Anthropic, OpenAI, Meta, Gemini, and xAI. If Microsoft starts releasing models, as they’re posting strong scaling science results, they could join this list — given their financial backing with a hyper-scale business and direct access to OpenAI’s intellectual property for training models.

The best models we’ve seen from the “tier 3” players like Mistral, Nemotron, Reflection, Arcee, etc. have had an increasingly hard time impacting either the open or closed frontiers. These models still can get substantial adoption, enough to start growing a business, but the costs of building true frontier models are now measured in billions per major model and only huge companies can sustain that.

Despite this massive scaling in resources, the technical core of the industry still doesn’t have a high hit rate on predicting how successful a model will be in real-world usage. Yes, evaluations are a good proxy, but why do the leading labs like Anthropic go from hit models like Claude Fable 5 to total duds in Claude Opus 5 and then back to hits with Opus 5.5? There are billions of dollars to be made in predicting this accurately, and the labs are surely trying, but the models fundamentally still are largely unknowable in the long-tail.

Of all of the players left in the race, betting on Meta, Gemini, and Microsoft financially seems much easier from a long-term perspective due to their rich balance sheets. This is the “AI will take a while and OpenAI/Anthropic do not have secret sauce” perspective. After all, it’s the fastest revenue growth industry ever — competition will naturally be immense.

Currently, Meta is valued under $2T and Anthropic is approaching their barnstorming IPO at the same price. Meta seems half a step behind on the same technology, but in the long-term view these seem to be starting from almost the same point and Meta has an extremely profitable core business. One of the two valuations needs to correct massively. A fun thought experiment I’ve been pondering with friends is: What do you expect Anthropic’s public valuation to be in November of 2027? How about 2028?

Leave a comment

The longer Meta and the “competitors” to OpenAI and Anthropic stick around, the more negative valuation pressure will exist around the AI IPOs (and that’s not to say anything about pricing in safety risks). Being at the absolute frontier could be financial tumultuous, while valuable. Time will tell — we know markets can stay irrational a very long time.

In some ways, Ilya was early in his declaration of AI being in the era of research. Research is much easier today now that we have coding agents. This ease is through a different form of work that is flexible and engaging, which signals the start of an era where good ideas can be much more valuable than good execution in software.

Subscribe now

We’re very early on this path. It’s also not the first transition the field has seen across the balance in if AI is more “research” or “engineering.” Before deep learning took off, AI was also far more of a research endeavor. Today, the best researchers are well-known to be judged by their ability to implement and scale their ideas in complex infrastructure. We’re about to go through a massive acceleration over a few years where the bottleneck is less and less engineering again. Some call this RSI, but a more grounded form is simply “parallelized, AI-assisted language modeling.” You don’t need to buy into takeoff scenarios to see just how much infrastructure improvements are coming, and how this’ll change the nature of working in AI.

The goal of this piece is to give an overview on why this engineering acceleration will happen, and touch on what it won’t quite fix. This’ll be a massive boon for diffusion of the technology into the economy, even if we don’t get economically valuable superhuman traits outside of math and coding. A lot of this near-term progress comes from scaling the inference-time compute we have with current tools, rather than dramatic step-changes in how good the models are at research.

Share

Many pieces of the AI training and inference stack are very verifiable. Training metrics like tokens per second per GPU, which is just training speed. Inference metrics look like tokens per prompt, FLOPs per token, or really just cost per answer. Both of these are very optimizable, especially as there are established sub-problems and architecture trade-offs which can improve them. I expect AI agents to help optimize this process end-to-end in a few years, where our inference capabilities get very close to the underlying maximum compute possible on our accelerators like GPUs. Over the previous few years, the gains companies can make on inference efficiency were already giant, e.g. saving like 10-30% on cost to serve a model after announcing it at a certain price point. This entire stack will compound, and I expect the effective cost of model intelligence to decline near-exponentially over the coming years (potentially faster than recent trends).

This low-hanging fruit of efficiency should take only a few years to capture. The longer term will be co-design of accelerators and models on longer timelines, which give extra orders of magnitude in efficiency on top of the flexible GPU platform. During this time of rapid efficiency gains, the flexibility of the GPU should be king, as the exploration space in architecture feels important and like a major lever of automated research. A prediction of pretraining research, at least in architecture and data selection to serve our current class of models, being automated in 2-3 years feels reasonable to me.

All of this will be a massive trigger of Jevons paradox for agentic models. I expect demand to only increase, as the industry is largely bottlenecked on figuring out better ways to orient and deliver the agents. Meta’s Muse agent is an early indicator of this and we should expect more Muse-like experiences for different audiences and target use-cases. It’s value created by understanding how agents work, rather than pushing the frontier of performance.

The equivalent phase in the broader scientific literature like biology and chemistry will be an era defined by models finding numerous cross-sub-field findings. The AI models are superhuman at crawling literature and making connections across sparse networks which are currently stewarded by small communities of scientists who rarely interacted. There will be a fine line between this heralding a new era of scientific discovery, e.g. with cures of most cancers, or an acceleration of the arc science was already on.

Another industrial-scale piece of low-hanging fruit is improving the general quality of RL environments. You don’t have to look far to see another new RL data company that’s crossed $100M or $1B in revenue. There are way more than I expected. The average outputs from this sector are remarkably low-quality. Countless researchers agree that lots of what they buy is frankly crap. At the same time, the leading labs see clear return on investment from buying the data.

The way that many RL environments are cruddy is clearly fixable.

Read more

Much has been written about the views of various players in this debate, which I summarize as follows:

1. Open weight models pose an untenable risk to society’s functionality, occupied by frontier lab leadership and the national security community in the U.S.

2. Western voices saying open weight models are necessary for defense and banning them will make the world less safe, occupied by AI risk moderates to different extremes. I include myself in this bucket, along with Hugging Face’s perspective after the OpenAI incident, and others like Joshua Saxe.

3. Chinese companies continuing to release open-weight models with strong cyber capabilities, who are making decisions based on the risk assessments of their society and government.

This list is also organized by volume or clarity of message in the Western AI ecosystem. The risk-focused discussions have been disproportionately visible, including pieces of work which I view as very detrimental to potential of engaging across assumed party lines on this issue. There’s also very little substantive engagement trying to understand how Chinese companies assess risk — rather, mostly arguments that border on ad hominem attacks like “Chinese labs don’t care about safety.”

The delusions of the anti open-weight alliance

The most recent publication from the “open-weights are dangerous” crowd was the report by Anthropic on the risks of GLM-5.3 as an offensive cyber tool. The problem with this is not the technical research they did, which is largely reasonable, but the failure to engage on more cross cutting questions like: What happens if we ban open models due to cyber risks or why do Chinese companies deem these models safe to release?

These questions, which I’ll come back to, are the ones we need to answer to understand the ecosystem perspective on current AI risks. All policy actions happen in a multi-piece puzzle, where the shape of risk can be changed, but it is rare that you get a universally better outcome. They are all trade-offs.

This sort of solipsistic writing by Anthropic fits very closely to another type of discourse I’m hearing about regularly that is determining the fate of open models and AI broadly — classified briefings. I’ve had many discussions with people who say something like “Look, I’m pro open models, but if you’re seeing what I’m seeing, it’s an onslaught out there due to [XYZ Chinese open-weight model] and we need to act.”

This perspective flies in the face of public information, where to date closed models have been documented as the cause of most existing cyber attacks. As someone who tries to make grounded predictions, the evidence of numerous attacks from OpenAI models is the only data I have on the shape of cyber risks (or, you can broaden the scope to include all of FelonyBench). There are a few possible explanations, and it’ll take years to get to the true answer:

• Maybe open model weights and closed model APIs (with some safe guards) are both far closer to being easy to mis-use, rather than API models being closer to safe. The trope "Open Dangerous, Closed Safe" may be closer to "Open Unsafe, Closed Unsafe."

• Maybe there are far fewer bad actors who are willing to use AI models for “loud” cyber attacks that target crucial infrastructure in the US or powerful and underprepared institutions. Closed models have stronger capabilities and stronger safeguards (in theory), but the stronger capabilities part could matter more in net harm if the safeguards on both are porous.

• I have a lot more to learn about how the cybersecurity ecosystem works, so I won’t support this full-heartedly as one of my current recommendations, but I think it’s at least a reasonable argument that open weight models as a diffusion tool for cyber defense could be the best tool we have to prevent harm over the next few years. There are many domains, e.g. sensitive government agencies, where open-weight models are the only tools that can be deployed in the near-term on air-gapped networks.

Lots of this situation seems like the messy reality being hard to pin down, where the logical limits are clearer. I may agree with the simple reality that closed models are far safer in the limit, as a technical tool with more layers of protection, but they may actually be causing more harm in the near term because of how accessible model APIs are. Still, much of the impact of the tools comes down to how they’re used and controlled. A duopoly on such a crucial tool may be the defining, risky part. This is the debate we’re having, and over time, as intelligence diffuses, many things may change.

These together make me come to the conclusion that if you think the latest open-weight models need to be banned to slow the diffusion of cyber risks, you probably also need to make public-facing APIs for the frontier closed-models illegal. The current stack of safeguards on closed models is stronger than open-weight models, but far from perfect. It is very likely that cyber capabilities of closed models increase much faster than guardrail performance, and a world where open models are banned while closed models continue to progress would be increasing the offense-defense cyber gap. It will take meaningful time to build mitigations that allow defenders to use strong AI models on their private infrastructure if we remove access to strong open-weight models.

Share

Explaining China’s AI risk posture

China cares about AI safety. They have come to this discourse in a complete, endogenous manner relating to Chinese culture and power structures. There are some big picture pieces of this puzzle that reflect in other ways, such as Chinese society’s techno-optimism that extends into AI and the Chinese government’s complete focus on political stability. These things do, of course, interface with emerging AI risks like cybersecurity.

They are in their own AI risk dynamic, which hasn’t risen to the prominence of the political dance between the White House and the frontier labs which we have followed in the U.S. for a few months. My understanding is that the Chinese companies have to register every major model release with the Chinese government. This includes evaluations, which originated around information control for banned government information. It is unclear if this framework has expanded meaningfully into any risks like cyber or bio. The Chinese government is very decentralized and information from the labs needs to go through existing structures to make it to the top.

At the same time, there is a much stronger air of social risk in China, where leading AI researchers aren’t allowed to leave the country and the industry is closing off to foreign investment. On balance I would guess the personal risks facing people by unleashing any domestic harms would be much higher than in the U.S., where the worst case is that your company gets deleted.

Overall, the political oversight seems much earlier than in the U.S. The Western AI companies’ core principle is close to “the best way to stop a bad guy with AI is to be a good guy with AI.” In China, these companies are much more practical about building a wonderful tool, and in many cases building a tool that complements their existing businesses.

I do think that the companies consider the global implications of their work. These labs also clearly are incentivized, and often pushed via the government (at least in public messaging) to compete as much as they can. Running comprehensive safety evaluations on a frontier model like Kimi K3 could cost tens of millions of dollars in compute. These labs would much rather leave that for training.

The correct debate is “what is the correct minimum amount of compute a lab should spend on safety testing before releasing each model?” There’s surely a case to be made that it’s more than the Chinese labs do, or at least they should be more transparent on what they do, but I would be hard pressed to agree if you said the minimum amount of compute would match what Anthropic or OpenAI does. It is not clear that, for all the effort Anthropic and OpenAI make on creating an institution that prioritizes understanding risks, they put it before business value and economic success in their priority stack. This “hands off the wheel” vibe from incidents like the HuggingFace—OpenAI incident was one of my lasting takeaways. It’s exemplified in the Hacktron hacking of OpenAI — it’s hard to be the world’s trust cyber partner if your own house is teeming with leaks. I trust many of the individuals at the labs to work hard to ensure safety, but I’m not trustworthy of the institutions overall.

Open-weight Mythos in April would’ve been fine?

When Claude Mythos was announced, it was previewed as if it was a new class of cyber weapon, where if it ended up in the wrong hands it would’ve caused mass societal destabilization.

It seems like this was wrong.

By all measures, GLM-5.3 is the model that crosses that threshold of capability, and there’s little public evidence that much has changed, and we’re over a month out of the model weight release.

In fact, I would go further. If Claude Mythos was accidentally released as open-weight, it seems like the world would have been more or less fine. Yes, there would be a clear increase in cybersecurity incidents and it would be bad for society if the model leaked, but it would be a scenario that looks more like an acceleration than a step change in risk.

The leading proponents of this new wave of open-weight fear mongering have effectively been making a falsifiable prediction — that the current open-weight models are going to cause substantial, not seen before AI harms by crippling our cyber infrastructure.

Based on all the evidence I have, which I’ve unpacked above, they’re set up to be wrong. At least now, after so many years of debating open weight risks, we will start to get some real answers.

Thanks to Rohit Krishnan and Joshua Saxe for some feedback on this post.

A big takeaway of this podcast is how JS and I both have so much uncertainty with exactly where we are heading, and this was our best effort at stating our observations today.

Chapters / topics include:

• 00:00 Predictions for RSI

• 18:15 The role of robotics in an AI acceleration

• 24:20 How far behind are Chinese models?

• 27:39 Does distillation explain the gap?

• 40:58 What Chinese job postings reveal about their labs

• 48:13 Are open or closed models safer?

• 58:10 How Epoch AI ticks

• 1:00:55 What a frontier post-training recipe looks like

Enjoy!

More from JS: Epoch AI profile and writing, X, LinkedIn

Share

Listen on Apple Podcasts, Spotify, and where ever you get your podcasts. For other Interconnects interviews, go here.

Transcript

00:00:00 Nathan Lambert: I’m here with JS Denain, who is a senior researcher at Epoch AI. He leads the insights team. He is one of the people who I feel like I get the best feedback on my writing from, whether it’s from US-China AI capabilities, now RSI. And I just wanted to open this discussion and honestly go deeper with him, trying to understand how he thinks about these various things. And I think you have a very useful, moderate point of view, which I feel like you’re probably a step further into what would be called faster scenarios for AI progress. But let’s get into this, and it’s like, what measurements do you think OpenAI and Anthropic are seeing when we get all these proclamations on RSI happening very imminently?

00:00:47 JS Denain: Yeah. I think, so there’s the measurements they’ve published, right? So, OpenAI and Anthropic both had blog posts, I mean, Anthropic two at least, on the effect AI has on accelerating AI progress. I think at least the things they publish, I don’t think are super strong evidence of imminent self-sustaining acceleration AI capabilities, or full automation of the job of AI researcher. But I think the kinds of things we see are, I think probably the most striking thing I saw in the OpenAI blog post was increasing usage of AI systems in model deployment, like the increase in spending on Codex that we saw. And it’s kind of unclear how exactly to interpret this, because maybe it’s a measurement artifact where they’re only looking at Codex, but in fact, there was a bunch of ChatGPT usage before from the researchers. But overall, that plot, for example, just shows a 2X a month increase in Codex spending by researchers, and that does seem to me to be some evidence of they’re getting a lot of value out of this probably. I don’t think this is strong evidence that in six months we have a software intelligence explosion.

00:01:57 Nathan Lambert: Do you think this is the same? So, what is the information they have internally relative to what we have? And this is obviously hypothetical. We don’t have this internal information. Because I get the sense that a lot of people are more scared in their updates from the labs than the information we have. And I try to take this very seriously of, what will they be seeing that is making the acceleration of risk comments go faster, and how much of this is material evidence versus how cultures evolve over time? And I’m much more interested in evidence.

00:02:29 JS Denain: Yeah. So two things. I think, first of all, I guess I don’t think, I don’t know, right? I don’t have full information here. I don’t currently think that either there’s some specific thing that people at OpenAI or Anthropic are seeing right now that we don’t have access to that warrants being way more freaked out about this. I also don’t think that... I think the public evidence we have right now, more general on AI progress and just a priori case for this being an important dynamic, I think is enough. I think to care about this particular dynamic of AI accelerating AI progress, that being a big deal and worth tracking. And then it’s kind of unclear what the urgency is of when the feedback loop really kicks in. So, basically on the what is there on the inside that people have access to, I could give examples of kinds of metrics, right, they could be looking at. It’s plausible that we have access to the capabilities of AI systems, but internal teams have their KPIs, and maybe they’re seeing compute multipliers in the pre-training team or other kinds of metrics that people are tracking going crazy. And then the combination of this plus some intuitions of how the different outputs of different teams combine yields a prediction on the trend in actual performance of the end AI systems. So that could be an early warning sign. It’s unclear to me that the recent discourse we’ve seen is evidence of things going crazy on those metrics.

00:04:09 Nathan Lambert: And how do you think of the link between RSI and existential risk? So I would posit that you agree. I think that there are very real risks of AI, and I’m curious on how you think these, what I would describe as very, very early measurements change anything on the scope of risk. Because I don’t think if you had asked people six months ago, it would be as immediate to x-risk among people are very reasonable. I think there’s more people that are reasonable talking about x-risk again, which was a little surprising to me.

00:04:43 JS Denain: Yeah. So, okay, my sense is something like... So personally, I feel very uncertain about this, but I do feel, yeah, basically bought into there’s, I know Evan Hubinger was like, at least 10% of x-risk within I don’t know what timeframe. I think I’m like, yeah, I know, and this seems pretty reasonable over a decade-long timeframe. I just feel extremely uncertain about it, but I’m definitely very worried about this. Now, why am I worried about this, and where do I think the disagreements come from? And then how do I relate this to the early sense of RSI? My sense, I’m kind of a capabilities theory of everything person. I think, and I think some people disagree here, but I really think that principal component of disagreement between everyone is how huge do the capabilities get, how soon, of AI systems? And I sort of agree that there’s other factors that come in play for how big your economic growth gets, also depend on the diffusion you get. And you could possibly you could think that capabilities are going to get crazy, but the AI system’s going to be just aligned and benign and stuff, and so there’s no huge risk. But my sense is concretely, when I look at the main kinds of disagreements between people, most of the people who I see who are very skeptical of those most extreme scenarios... I think just expect capabilities to not be as huge or as I think folks who are—

00:06:18 Nathan Lambert: What does being a capabilities maximalist look like in a few years? Because I think I’m probably on the skeptical side, so please continue.

00:06:28 JS Denain: I think it looks, for example, something like the AI 2027 scenario, right? I think it looks like the mechanism for this is AI is automating the AI research process, I think, and that’s a reason to pay attention to it. But in terms of effect on the real world, I think it’s like massive progress on robotics. I think a huge industrial explosion, AI systems are just managing factories. You have this kind of self-sustaining economy that just is able to make a large scientific progress much faster than you would have expected. And so I think concretely, the kinds of disagreements I would expect are on, yeah, if you have AI systems that are both very intelligent in the book smart sense, but also have been trained to have more affordances and use them astutely, have been trained to kind of manage projects in efficient ways and stuff like that. How big are the real-world bottlenecks to making very fast R&D progress or getting hard power over humans?

00:07:26 Nathan Lambert: Yeah. Can we go into some of these in detail? Did you listen to the Dwarkesh podcast with Charlie, Beren, and John?

00:07:32 JS Denain: Yes. Yeah.

00:07:33 Nathan Lambert: Yeah, because they had at the end, they had this section on various capability levels and timelines for getting them. And I feel like I agreed with most... I was very in agreement on the distribution they had up to this, and then was surprised by the timelines. And one of them was the 10X productivity for the AI researchers. And I think Beren and John were faster than I think. And my kind of statement is that I think the cycle from of having an idea and doing the experimentation to test it, I agree will be 10X faster very soon. But I don’t necessarily agree that I would say that AI researchers will be 10X more productive in net, which I would describe as the pace of the field’s complete understanding. And understanding is a different axis from just continuing to scale models. I think that’s one of my core confusions on the AI research side. So I’m just kind of curious how you think about this type of thing and how you might specify a 10X improvement in AI research into subcategories.

00:08:37 JS Denain: Yeah. So maybe there’s a scale you could have here, which is the end thing that you might care about is how much faster is AI research overall? Or how much faster is Anthropic’s overall output? And then Anthropic as a company is producing some things, and it’s doing in one year what it would have taken it 10 years to do. And that’s pretty different from individual researcher productivities, where I think you could... So if most of what AI researchers right now are doing is this loop that you were describing, then it’s possible that you get a 10X productivity improvement for the median researcher based on the tasks they’re doing right now. But first of all, that doesn’t mean you get a 10X productivity improvement for all researchers. And even if you did, right, there’s other bottlenecks that hit such that that needn’t convert into a 10X productivity improvement for Anthropic as a whole, right? You could have all the researchers be 10X more productive, but because of compute or other things, the company itself still doesn’t move as fast.

00:09:38 Nathan Lambert: I think an analogy I have is, I think that junior PhD students will be 10X as productive, but from the advisor’s perspective, their research agenda will not proceed 10X as fast. And it’s like to the extent that that contributes to Anthropic’s progress is another hard thing to jump on AI capabilities, where I think that listening to the Noam podcast. This is me, I’m just thinking, was thinking about this when writing about this, is there’s such an amount of inference compute coming online, and I think Dwarkesh highlights this very well, that it’s very hard for me to disambiguate massive speed-up in AI research from the fact that we have way more compute and can now much more effectively spend it on related problems. And I think we’re going to get all of this at once.

00:10:27 JS Denain: Yeah. So definitely I think this question of... So when you look at the OpenAI blog post they had on their acceleration, right, they do point out huge surge in Codex spending from researchers. Interestingly, actually, Codex spending in other parts of the company kind of had a huge surge in the spring and then kind of plateaued in the summer. But for researchers or the data team or engineers, it keeps growing, even accelerates sometimes. And so they point this out, and then they try to look at where there was an increase, which kinds of tasks had an increase in usage. And a lot of them are engineering tasks. There’s an increase in troubleshooting tasks. But they definitely point out that for a lot of the high-level strategic decision-making, they both anecdotally and also when they look at sessions, don’t seem to find a huge uplift in making better compute allocation decisions or deciding on research directions. And so to me, that’s pretty similar to the PI case. And so I think there’s a first question, which is, how much of an improvement... Imagine you just didn’t get that much AI uplift on that component of AI research, but the rest of AI research really went crazy. Then how much faster do things go? And the separate question is, I don’t know, how hard is this strategic decision-making? Can’t you just have a bit longer horizon RL? Or maybe you can bet on decent transfer from other fields where those kind of decisions are important, and then you do actually get that kind of uplift at the end.

00:11:59 Nathan Lambert: I think part of my intuition is that science will look so fundamentally different that it’s almost hard to put a number on it. And it’s the pre and post-AI era, and we’re just in the rapid transition to what is a new method, new way of doing science, because I think all the conferences are ready to burn down and struggle through the next few years. I hope that they collectively figure out a way to like add AI oversight into reviewing and things that are scalable, because they have so many slop papers that they need this type of gate. So I don’t really know. And I think on the capability side, I’m like, what an AI progress is like clearly translated into new capabilities. There are two things. One is like pre-training scaling laws. Our loss is proportional to like an exponential increase in compute. And on the research side, what we are doing is we’re shifting the line so that it has a better offset and potentially a better slope. And then on the other side is the RL environments, and I think the RL environments we’re building now are very comparable to valuable work. So I expect the AI models to get much, much better at like knowledge work that can be scoped. But I don’t know if we have a good process for like churning out an order of magnitude harder environments, which would be closer to like cure cancer, solve these open math problems. I think math is a case that we could talk about. But that’s kind of like, I think there are unknowns on scaling the like raw intelligence more than efficiency. So I’m very optimistic in scaling efficiency.

00:13:29 JS Denain: Yeah. I agree that like in some sense, right, like inference efficiency is like a very like hill-climbing task. It’s like pretty well-scoped. And yeah, so I mean, it’s already something that has very, very fast trends, but I could imagine those trends. Yeah, I imagine those trends will go even faster. It seems really like the kind of... I mean, indeed, we have evidence from OpenAI, right? Like, saving on like serving costs, et cetera, through like building better kernels and like if you’re new to this. They don’t give that many details, but that’s already happening. One thing I’m curious about actually in your case is like, so here’s one way of defining like Anthropic, for example, like accelerates overall, which is you could look at the like ECI trend in like, the ECI of the best Anthropic quality point in time. And you can like look at the current trend line and you can ask the question, like over the next year, will we see a 5X acceleration? Like, will the slope be like 5X larger than it was, say, in like 2025? And I think it’s like pretty likely we see like a huge increase in this because it is like kind of a legible like KPI that like, I mean, it’s not literally KPI, but it is a KPI that like the company is aiming for, modulo like safety considerations, et cetera. And I’m curious about whether you think it’s very unlikely we get this or whether it’s more like we might get this, but like if we do get this, it’s mostly that like ECI has been Goodharted as a metric and like the implications for like real-world capabilities aren’t that huge.

00:15:03 Nathan Lambert: I wouldn’t be surprised if we got this, but I think that it’s going to be like we’re on a slope and then we could get an uptick in slope of hill climbing, but then we like saturate what we know how to hill climb and then it goes to be lower. So it’s like all the things that we could measure I think are going to be getting pulled up very quickly by being measurable. And then we’re in the domain of like, how do we measure it? Because I think, like I talk to people that are trying... Like evals are so expensive to build now, and I do think that evaluations are going to be like how good and efficient it is at coding, how good and efficient it is at ML research, how good and efficient it is at knowledge work. But I don’t know how to like... Building those evals all seems tractable but hard. But then like how to make a breakthrough in fundamental chemistry seems really, really, really hard to measure. I was going to draw on like maybe frontier math as an example, but I think math is such an exception as like one of the most jagged pieces of AI. I think especially like open problems in mathematics are like the perfect target for rapidly improving AI because it’s like a falsifiable thing. And it’s like if we were to, say, see that in something that’s much more open-ended, I think I would update a lot. Or if the labs were like to come out and say, “Using Claude, we have a very, very big change in what our architecture of AI is,” to like there’s the famous like Jonathan Frankle–Sasha Rush bet, and it’s like, and the transformer is no longer like the lineage we are on. I think any of those things being very AI-driven would make me update a lot. But seeing more math, like I think I was surprised by the pace of math, but like not astonished.

00:16:49 JS Denain: That’s interesting to me. I definitely agree with this general sense. So like I think METR folks looking at like your nanoGPT results from autoresearch-style things compared to like what the humans were doing, it does seem like there’s this, I think Tom Cunningham calls this like the apple-picking model where AI is like much more efficient at the start, but then doesn’t actually like uncover as many new ideas. And you see this in this kind of optimizer research. Yeah, I mean, one thing I will say on this like verifiability point is like, I think a pretty common trend is like you’ll have some task that’s like not verifiable and you’re like, maybe you struggle to build an environment for it. But actually it’s like it’s a subset of a larger task that is itself like verifiable. It’s just like longer range. An example of this is like there are many like hard to verify tasks out of like companies. But in some sense, like revenue or like other like metrics, like valuations are like pretty legible. So that’s like one thing. I mean, the other thing is like expect things to be pretty jagged. But I think a big question is like, yeah, can you get, for a crazy world, can you get like a large, like self-sustaining industrial kind of explosion?

00:18:05 Nathan Lambert: Yeah. Well, can we talk about robotics and industry? Because I have a background in physical robots and like I think the robotics trends will look much closer to self-driving cars than LLMs. And I think that a lot of the singularity arguments are based on robotics being able to look much closer to LLMs than the self-driving cars roll out. So, why would you disagree? Or, what is the argument that mass industrialization and robotic expansion is doable? Because my prior is so suspicious that I maybe even haven’t given it enough justice, but I’m very suspicious of this being a viability, and mostly in terms of being a relative timeline. I think it could happen over decades, but I don’t think it’s a two to five-year concern.

00:19:01 JS Denain: Two to five years seems rough, to be clear. I think I just don’t know as much about robotics here. I think is your main concern just reliability is really rough to get right in the same way that it was just a long tail of scenarios where things are, or was it more like a real-world thing where there’s much more regulation that comes up?

00:19:21 Nathan Lambert: I think it’s building things is hard. I think that, let’s see. I’ll talk us through some of this. For example, I know places like Amazon, they build new factories to be robotic first, and those are more effective for them. And what this would take then is building a robotics factory. In the case of the US, it’s like you have to build a robotics factory that builds robots very efficiently in the US and then transition that or make a new one that is built by said robots. And I think the re-industrialization of the US is something that I think is like, there’s a lot of reasons why it is not happening. I think potentially in China it is more likely, but I also just haven’t been convinced by AI results on visual and action models that they’re progressing fast enough. I think I’ve had discussions with people in the multimodal field have described the techniques as being much more rudimentary and less developed than the text language models, and in need of much more fundamental innovation, where something like code plus RL is a very natural match that the hill climbing is very predictable. So—

00:20:35 JS Denain: So, it seems like there’s two things. There’s the trends in robot capabilities is not as fast as you would expect for LLMs, and also even if robot capabilities were huge, it takes a while to build factories. I think I’m sort of skeptical of the second one. I’m just like, if robot capabilities are sufficient, the total addressable market for this is massive. And if you look at data centers in the US, there has been extremely fast build-out. If you had robots that were just literally able to substitute for blue-collar human workers, I feel like the financial incentives would be huge. And I think a lot of the reason why in some cases, the US doesn’t have huge build-out is just a demand thing. I think that’s the case for power, for example. So I think in that case, I’m just like, yeah, I feel like we just, what is the Tyler Cowen thing? Don’t underestimate the elasticity of supply is the main thing I would point to. I think on the capabilities front, I’m more uncertain. In particular, I’m sort of still confused and haven’t really looked into the, how much do you get directly actually from LLMs and foundation models for robotic capabilities? In particular, the other uncertainty I have is, it’s not clear to me that extremely fine-grained, extremely dexterous capabilities are the main thing you need for massive industrial explosions. And so, this longer tail of the hardest part of robotics, I’m not sure if that’s the biggest blocker for massive industrial explosion. Overall, robotics is something I have less expertise in. I’m interested in how many of the scenarios for doom ultimately kind of route through hard power acquired through robotics. I think part of my uncertainty also comes from, is it plausible to me that the minimum abilities that you need to acquire a lot of hard power and pose pretty catastrophic possibly extinction risks is more like, have access to nuclear codes or something like that? I don’t feel like I have great thoughts on this. I’m interested in more threat modeling, but I think that’s part of the thing is, what are the capabilities trends is something that people have disagreements about, and so what’s the minimum capability that’s necessary to cause these extinction-level harms, or harms that are sufficiently catastrophic, they just permanently alter the human trajectory?

00:22:46 Nathan Lambert: Yeah. The last point I would make on—

00:22:47 JS Denain: I think that’s the kind of questions I want to see a bit more thinking on, but yeah.

00:22:50 Nathan Lambert: The last point I would make on robotics is that I think the robots will be very good in constrained and repetitive environments, like manufacturing robots, and I think much longer until there are robots walking around the street cohabitating with humans. And if I were to go deep on this, I would want studies and discussions with people that are building multiple different data centers to how much variety is in their job versus how much of it is you take box off of truck and you put box in location. And I don’t have any good signal on how clearly repetitive that is now versus more human dexterous and ingenuity in problem-solving.

00:23:34 JS Denain: Yeah. Although also currently, the environments have been designed around humans who are pretty flexible in those ways and have other constraints. You could imagine designing factories to be robot first is a thing that I think you’re mentioning Amazon, right? Sort of does that more now. And so you could also... Yeah, I think that’s the kind of stuff that jaggedness would get you, which is you might just have massive accelerations, including in the physical world of some industries, even if capabilities for fully being as dexterous or flexible as a human might not be there. Mostly, yeah. I think mostly, yeah, robot capabilities seem very important to track to me. We had some piece about this at Epoch, but we don’t claim robotics expertise.

00:24:21 Nathan Lambert: Yeah. I agree. I think we could shift to another capabilities topic, which is how far behind do you think the top Chinese labs are of OpenAI and Anthropic? And you can define how you want to measure it, whether it’s like public models or like internal models. I think doing both is actually pretty interesting to think about. Like, how would you describe the gap? Like, choose your... This is one of the few things I’ll push you for a number on.

00:24:47 JS Denain: Yeah. I think I would go with like, I don’t know, six to eight months or something, roughly.

00:24:53 Nathan Lambert: From public to public?

00:24:56 JS Denain: Yeah. Like, release date to release dates. I don’t have a great catch numbers on like how long the internal to public deployment. Yeah, I’m sorry if it increases a bit if you’re counting when the model was built, because I would expect the delay to be larger for OpenAI and Anthropic than it is for Chinese labs. But yeah, I don’t know, just looking at ECI and there’s a few different methods you can do there, and there’s a good amount of noise between. Those methods are kind of reasonable, and my sense is they give you something on the order of like six to eight months-ish.

00:25:28 Nathan Lambert: I would say that like Kimi K3 and GLM 5.2 are closer to like two to three or four. So do you think those are anomalies down to measurement or overfitting? So I have this discussion a lot with Florian. And Florian, who helps me with Interconnects, is constantly badgering me down. And I think I intuitively have landed something closer to you. And I think those two models, and in particular Artificial Analysis, were very close. I think Kimi might have been even under two in the Artificial Analysis Index at the time. And I’m just putting this out there, and I go back and forth all the time on it.

00:26:12 JS Denain: So I don’t remember the especially most up-to-date details on the Artificial Analysis Index. My sense is that it might understate the gap for curation reasons or something, but I don’t want to be too confident there because also they update methodology and stuff like that. I think even in ECI, though, there are some cases where it was more like four months. Yeah, I don’t know. Four to eight months seems reasonable to me. I think like two months, I would be like, “Nah, that seems like a bit more overfitting-y.” We did internally look into a bit how much overfitting explains the gap. Like, just doing the kind of like, what’s the ECI lag analysis for, if you just take an ECI based on private benchmarks versus not, or private benchmarks plus results were from models that were from benchmarks that were published after a model was released or something like that, where there’s no risk of contamination. I think overall, that effect was basically not statistically significant, which was interesting. I think if you did rely just results in model cards as opposed to the results being put in ECI, we probably get some contamination or overfitting effects. I still think there’s some amount of, yeah, something like ECI might underestimate the gap just because I do think OpenAI and Anthropic are probably like... There’s probably other capabilities just would appear less in benchmarks and where there’s been more optimization or serving a broader set of users. But yeah, I don’t know. That’s my overall number.

00:27:39 Nathan Lambert: If we were to ban distillation effectively, not even just ban, but if distillation were effectively be stopped, where do you think the number would be in six to 12 months? I think without RSI being super crazy. I think RSI going super crazy, the labs pull ahead by a lot more.

00:28:01 JS Denain: Yeah. On current trends, I think I’ve actually updated towards distillation is actually a really big factor. We can talk about the different factors. And so, okay, so currently if I’m saying six months gap, then in six months, it’s probably not literally 12 months, right? So, my guess is, yeah, you are at like eight or nine, something like that. You go from like six to nine maybe.

00:28:31 Nathan Lambert: Why have you updated on distillation being effective?

00:28:35 JS Denain: Yeah. So I think it’s like, yeah, there’s two reasons. I think, a lame reason is, I don’t know, more people seem to be saying it’s this.

00:28:45 Nathan Lambert: They have been. I’m surprised by it.

00:28:47 JS Denain: Yeah. The Stolen Thoughts paper did suggest it was actually relatively easy to get the stuff. Yeah, random gossip. I think also a big one is, we can talk about this, thinking about the other explanations for the lag and them not seeming as compelling as when I first thought of them. I do think the Claude routers thing also just seems like a pretty big deal. And initially when I thought of distillation, I hadn’t considered that. But I do think Claude routers giving the kind of right prompt distribution for realistic usages, use cases and stuff like that just seems quite useful.

00:29:26 Nathan Lambert: But it’s also like these companies have their own usage at this point. And it’s also like we have the benchmarks. So the benchmark distribution we already have. So I don’t necessarily think distillation is helping with... I think the routers could be very helpful, but I was confused by that prompt distribution thing because once you have the benchmark distribution, you just make similar examples or find similar examples.

00:29:50 JS Denain: Yeah, so I agree. So I think the Claude routers just give you a bunch of useful data to train on, and that will generally improve capabilities. I think that’s one explanation. I agree that I think giving you the right prompt distribution is, I think, going to be useful for real-world capabilities. I agree that purely if I’m discussing this ECI lag or something, then yeah, you can just use the prompt distribution for benchmarks. And so I think it’s not going to be that good of an explanation there. Yeah. I think just... Yeah. Also, so as an example, right, people saying that mid-training is a really important thing or something. I think actually in that conversation, right, Beren was like, “Mid-training is actually getting you 80% of the way there.” And more claims of this form and generally of RL being very useful but not doing that much of the exploration work, I think is also evidence that doing SFT on language trajectories is really useful.

00:30:48 Nathan Lambert: Yeah. I want to go through—

00:30:50 JS Denain: I think he mentioned on a post on this is, it’s a good initialization, but I guess I’m updated that this initialization is really, really important. And yeah.

00:30:59 Nathan Lambert: Yeah, because I’m still on the side that I think doing RL well and doing RL faster so you can go bigger is the way people are getting capabilities. But I have been hearing more about this repeated... The framework would be it’s a very repeated cycle between mid-training SFT and RL, and then your peak RL helps you feed into mid-training and less of a very big RL run, which I never thought... The RL runs are a week probably. I don’t think they’re insane, especially like Zhipu or Kimi. And that’s... So I could see this, and I was like, “I don’t really know.” I generally thought that because the models are getting so useful, I don’t know how... It just seems like the scale of distillation would need to be so big to really do that for mid-training. And the counterargument is Kimi and GLM 5.3 are also strong models, where it’s like, if the gap is that small, it just doesn’t make sense to me that the help would be so big. That’s why I’m still on the not convinced, but looking for more evidence. And then some of the other re... I have this blog post that I wrote in front of me. If you want to talk about distillation more, you can get that comment in before I switch to other topics.

00:32:21 JS Denain: I guess I’m pretty interested in talking about the alternative explanations or something for... I think there’s this general phenomenon of Chinese labs have way less capital and particularly way less compute than US labs. And that delay is huge. And then this, whether it’s four months or eight months, it’s a much shorter delay in capabilities than in capital. And so, this requires some explanation.

00:32:48 Nathan Lambert: Yeah. Do we think the Chinese labs are still releasing meaningfully faster from time that RL is done to public gets the API and evaluation scores? In the past, I thought that the time to release was much faster, and I think the labs are releasing intermediate versions faster now, so I don’t know if that explains as much of it. But I would say a year ago, I would put a lot more to Chinese models get model out within days, American labs could take months, and that would artificially squeeze the gap very substantially.

00:33:26 JS Denain: Why would it squeeze the gap that much?

00:33:28 Nathan Lambert: Because you have the trajectory of capabilities, and higher one is OpenAI and Anthropic, and they stop training here in time, and then the Chinese labs are here. But then it’s just like the model is stagnant before it gets released.

00:33:44 JS Denain: Yeah, I think this depends on the method that... This both depends on the method that you use for computing the lag or something, right? So my sense, I think when I was saying four to eight months, I think a lot of that uncertainty interval also comes from, are you measuring the method by looking forward or backward or something? Or how are you resolving the uncertainty? Are you taking this kind of staircase or this kind of staircase, or interpolating or something like that? So I think that’s kind of still accounted for in the uncertainty. So my guess is that’s not a huge... Or that’s still, even with this, I think there’s still a large effect to explain. And I think, yeah, these are other potential explanations, right? So one is maybe there’s a big delay in compute, but in labor or potentially data, there’s less of a delay, and those are important factors. And so even if you’re two years behind in compute, if you’re six months behind in data and in fact ahead in number of researchers or something, or not that far behind in number of researchers, maybe that helps catch up, and that’s not even a spillover effect, right? That’s just a reason. Yeah, there’s distillation. Other explanations people have are just ideas might leak, and I don’t suspect that’s a huge effect. People can play with a model, and through playing with the model, they kind of infer maybe what it was trained on or what’s useful to push on is another explanation. Those things are not quite like distillation, but using Claude as your reward model or using Claude to clean data or something, maybe have some spillovers.

00:35:23 Nathan Lambert: Let’s see. I’m trying to quantify some. I would say that I think the Chinese labs care about benchmarks a bit more for financial... Some of them are public companies, and showing close benchmarks helps them a lot. So I do think they care about it more. I don’t know if that’s going to give you a month or two. I don’t know what are you going to give back, like a month or two. I think there’s a good chance that the Chinese labs are better at organizing talent on just doing really mundane hill-climbing data work. I don’t think that’s a huge effect, but it’s just like they have a ton of talent, and culturally through the way that... I think that there’s a good chance that it’s just more grind. They might actually grind more effectively than the American labs, which is... I don’t know how. I wouldn’t put a lot to this because I think it’s fairly close, but I think that’s a chance. But that’s a very marginal... That’s not a gigantic lead cause. I would put more of it to compute difference than to distillation. But maybe distillation is very related as a compute difference because it’s a way to turn... It is a definition of way to turn money into very high-quality data, which is something you would otherwise need compute for.

00:36:35 JS Denain: Yeah. Another theory that I’ve vetted about, and I’m not sure how big of a deal it is, is there’s this data market in the US, and maybe one thing that happens is the best RL environments or evals, it kind of takes a lot of compute actually to figure out which ones they are. And so maybe there’s kind of a collaboration between data providers and AI labs to actually test and figure out what RL environment really worked. And then maybe that data provider, once they’ve done this iteration, which implied a lot of R&D compute spending from a frontier lab, then they’ll build a bunch of similar RL environments because they’ve learned the lesson, and then they’ll sell them with some delay, but still sell them to other people, including in China.

00:37:21 Nathan Lambert: I’ve heard this from people.

00:37:22 JS Denain: And so there’s kind of this implicit R&D compute that was spent and that it gets saved. I’m not sure how big of a deal that is.

00:37:29 Nathan Lambert: I’ve heard this multiple times, including from people in China that have... This is like multi-hop type of rumor that I’ve heard a few times is like, we just have to wait a certain amount of time and then we buy the RL environments that Anthropic bought for a 10th of the price. And I think this could contribute a lot to capital efficiency, but also as somebody who buys into human factors being very important in model progress, which is just like competition, I think the analog to the competition that OpenAI and Anthropic feel right now is proof of concept as a way to make something way easier. And I do think the OpenAIs and Anthropics of the world are much more likely to innovate in open-ended things like Navier-Stokes, multi-agent mega-scaling than the Chinese labs. And having talked to many of the labs, I don’t think they would contest this, but they probably won’t put it on the record. But they’re just like, trying to keep up is so much easier.

00:38:27 JS Denain: Yeah. So that’s more like the... Yeah, so the way I describe this was like four-minute mile style effects of like, you show that something is possible at all and then people have the conviction to go down that route, don’t waste a ton of resources exploring a bunch of other things, is the thing you’re pointing at here.

00:38:43 Nathan Lambert: Yeah. And I think this—

00:38:44 JS Denain: I guess I’m looking for examples where you think this... What are examples? Because for example, reasoning models is not this, right? Like, DeepSeek Math or whatever came way before o1. Yeah, I’m kind of curious if... Yeah, I agree this seems kind of plausible, and I’m not sure I can come up with examples of big innovations that a frontier lab has—

00:39:04 Nathan Lambert: I think it’s more in org structure. So it’s like there’s R&D and modeling compute at the labs, which I think kind of get lumped together. And realistically, make-model-better compute of the next generation model versus the two or three generations down the line is very different. And I would think that the Chinese labs are just spending on a shorter time horizon. And then if there’s a major innovation in architecture or something that gets bubbled around the US ecosystem, I would assume the Chinese labs will figure out as well. I don’t think that an architecture innovation is kept very well between Anthropic, OpenAI, Meta, and Google. I think that these are all leaked very quickly through personnel turnover.

00:39:50 JS Denain: I see. So you think that actually has a big effect. So basically, I think it seems like you are just like, you think that with the other explanations, you can just get pretty far, such that you don’t really need distillation to explain a large fraction of the gap.

00:40:03 Nathan Lambert: Yeah. So, I think without distillation, the gap would be fairly similar. It’s like also just building these models, there’s so much low-hanging fruit, and I don’t think distillation plugs that much of the low-hanging fruit that you are doing. It’s very unclear. It’s like once you establish your reasoning trace as being... This isn’t proven, but I think in mid-training and SFT, you need to have a pretty distinctive style in your reasoning trace to kind of stack it and have a defined reasoning strategy for the model. And Kimi and GLM have this, and I think they could modify it from where they are. And it’s just like, how does that compound over time if Anthropic and OpenAI change way faster, maybe it matters more. I don’t know. Doesn’t sound like we’re getting somewhere. I did like the post you had on understanding Chinese job postings, which I thought was a very clever way to try to peek behind the curtain of the Chinese AI industry. So I don’t know if you have anything you thought was particularly fun about this. It’s like, it was so hard to understand the industry that is going on there from outside.

00:41:22 JS Denain: Yeah. So this is work by Cheryl Wu and I. I think it’s, as a source of evidence, it’s more like you can get interesting anecdotes or interesting facts. I think it’s not so much the kind of Epoch-y thing where you can get a trend line that you really follow really well. I think some job posting trends are good for this, but I think especially for China, it is a bit more like, “Oh, well, you can see they’re hiring in this region for data center roles, and it seems like that suggests that at least some of these labs are basically constructing their own data centers as opposed to renting compute from Alibaba or whatever.” So you get a few kind of nice facts like this. I think you also do understand a bit more what kinds of product strategies the different companies have and what they’re aiming for. So I think it’s pretty interesting for that. This post was now published in June, so things move fast and things may have changed. But yeah, I think there are interesting takeaways on company strategies, on how they—

00:42:31 Nathan Lambert: Can you say more about company strategies? Because I think that a lot has been cast on, say, like... The different company strategies in China. It’s like, what is DeepSeek’s strategy? I think MiniMax is one of the clear ones, and then Zhipu. And Zhipu has some on-premises deployment stuff, but at the high level, I think Zhipu can be much closer to OpenAI and Anthropic. But do you agree with this, or do you think any of them are more defined or have more interesting different things?

00:43:03 JS Denain: Yeah. I mean, one thing is just, I think we have this plot of how many job postings there are for B2B sales. As an example, it really does seem like Z.ai, like Zhipu, has a lot more of them than MiniMax or Moonshot. I think also, they have different strategies in terms of how much they want to expand internationally. I think MiniMax is a decent amount of that, more so than Z.ai, I think was the thing we found. I think there’s a few things like this. I think for DeepSeek, we didn’t see as much of that, for example.

00:43:40 Nathan Lambert: Having done this analysis, do you think you could hand an AI model your taxonomy and have it redo it every three months? Do you think this is now a thing that Epoch could almost automate?

00:43:53 JS Denain: So I think we do have some upcoming project on job postings. I think specifically for this kind of thing, I feel like the takeaways are kind of ad hoc or something. And so I think it relies a bit more on you have a bunch of this data, what actually seems interesting? And I feel pretty often that LLM’s assessment of what are the top five most interesting things is not that great. But one example of a thing where I think... So I actually have this really crappy vibe-coded job listings tracker, which looks a bit more like the quality’s better for an OpenAI Anthropic. And I think Epoch might have something on this in a bit. But one example where AI is pretty useful is just, I don’t know, you just want to reasonably classify jobs as are they basically research jobs or are they basically engineering jobs? Or maybe research engineering is actually hard, but are they research jobs or are they GTM jobs? And the structure of the teams are going to be not coherent across different companies, but you can use AI to automatically track this over time and have some coherent categories. So then you do get trends like, “Oh, it seems like there’s been a surge in hiring from this frontier AI lab, and it’s basically mostly been sales, or it’s been mostly hardware infrastructure.” I think that’s pretty cool, interesting, and automatable.

00:45:14 Nathan Lambert: Yeah. I think if you do this again, I’m interested in if you find things on data market expansion, which would be like your job is buy data. Because I’ve heard from many people that the Chinese data market is exploding, and I have poked various, whether it’s people I know at labs or people who I know follow Chinese industry closely. And it’s like, they’re like, “Yeah, but I can’t get anyone on the phone to know what this manifests as and what their budgets might be,” because Anthropic famously has billion-dollar budgets. And it’s just like I think it very quickly can become very important, like compute export control discussions stuff. By the end of the year, I suspect there will be some article talking about this and going deep reporting it. But from my perspective, getting that reporting information is near impossible.

00:46:05 JS Denain: Yeah. I think data is just really rough. I think for Epoch, data has been a tricky thing. It’s obviously an enormously important factor, but it’s harder to quantify or commoditize, and it’s also quite secretive. And so it’s like, I think a lot of the information is gossip. It’s hard to know how much to trust. My sense on Chinese, I think that article actually has something on this. Just it does seem like there’s a lot more in-housing in general by Chinese companies, and I think this also applies to data. And kind of relatedly, actually, there’s a lot of hiring for intern roles. I think if you go to the jobs listings pages for a lot of these companies, there’s specifically two tabs. They have summer internship tab or the internship tab and the normal hiring tab. And there’s a lot of, I think, hiring fairly junior people, and some of that might be for data purposes. But you did also see a surge in in-housing for data in the frontier AI companies. I’m not sure if this is still the case, but xAI was the only company that had this, but it was kind of nice because if you went to the careers page, there was actually the human data category, and you could just see which data roles they were hiring for. So there’s stuff to do on job postings on the data front as well, both xAI. I did this with Mercor and a few others previously. But I think it’s similarly, I think the most interesting things have been like, “Oh, well, that’s a crazy-ass role.” xAI is hiring for meme specialists or whatever, and it’s more ad hoc fun things than great trends, at least from what I’ve seen.

00:47:35 Nathan Lambert: Yeah. I think that this is not going to change, unfortunately. It seems very much like individual teams at labs have contacts at data companies, and all of this is done privately, and it’s not... I guess even tracking compute is fairly hard, but now data centers are a manifestation that is somewhat trackable. And NVIDIA is a public company, and that is what is done. If the data companies were public, it would probably be easier to tease some things apart.

00:48:13 JS Denain: Agree.

00:48:13 Nathan Lambert: Okay, we have another timely topic. How do you feel about frontier model safeguards? Given that in the last 24 hours, we learned that three people used public Claude models to hack OpenAI and get internal access, which is just crazy to me. Did you see this? It’s genuinely surprising to me that this is the leaky world we are in. At the same time, we are discussing RSI things much more seriously.

00:48:46 JS Denain: Yeah. I briefly saw this incident. I guess I’m not sure how much of an update it should be specifically on like model safeguards versus like you know security quality of the companies, or probably both. I think it’s a big question, right? Like, how good are the model safeguards? Seems like quite important, especially when people are talking about open models where it’s clearly open model safeguards are worse because there’s way more affordances that someone can have, but also open models are less powerful. And so just in terms of the you know misuse implications, it seems there’s just like slightly safeguarded, very powerful model versus like not at all safeguarded you know less powerful model. And I feel it was a priori kind of unclear to me, like which one is going to have the largest effect. Yeah, I don’t know. I think this is an interesting question. I think I feel also like people feel differently about this or something. It seems like some people feel pretty strongly that it’s hard to at least persistently do really bad things using the frontier models. But I’m like, yeah, there’s definitely clearly examples like this one, or really the Stolen Thoughts paper or something. It just seems like the safeguards weren’t good enough to prevent that from happening, even though there’s strong incentives for the companies to prevent it.

00:50:06 Nathan Lambert: Yeah, the Stolen Thoughts—

00:50:07 JS Denain: I’m kind of confused about this.

00:50:09 Nathan Lambert: Stolen Thoughts I put into like a serving issue, which is just like you were able to kind of jailbreak the API functionality to get the reasoning traces out in a reliable manner. Which I think the Chinese labs almost certainly had been doing for some time. And that potentially has a way to show more about what the effect of distillation was. I would guess that there are other ways that they get those tokens out, unfortunately, if there was one such prominent one. And then there’s the models themselves, which is the safeguards they have on the models, which is like the classifiers and prompt distributions and things. And I would expect the models to be somewhat permanently leaky. I just think that it’s hard to imagine. If you have a less aligned model and the task is, “Help me figure out how to get around said aligned model’s safeguards,” I would guess that that’s in scope of a strong cyber model’s abilities to figure out some sort of jailbreak for another model. It seems hard to permanently squash these things with how leaky they are now, despite the supposed launch, like higher false positive rate and being so general, and people get upset when they launch and they’re getting downgraded all the time, and people’s accounts get banned for doing bio research and things. So it’s just like the whole thing seems like a very big mess. I would be on the opinion of like, I guess maybe this is somewhat why the whole Fable freakout went down, which is like Anthropic was like, “Mythos is a powerful weapon.” And then somebody told the government and was like, “It’s actually easy to get around it,” and then they freaked out. But I think of it as like some of the distillation story has been around distillation lets them scale, take dangerous capabilities. But I think if you’re hosting an API at all, you have to assume in the current and near future that you can extract the capabilities out of that model, even if it’s slightly safeguarded.

00:52:14 JS Denain: Yeah. I guess one thing is I’m not sure how they prioritize between different threat models, but they sure seem to have some incentive to prevent distillation. It’s also hard to know what the counterfactual is, right? We have these examples of people doing those bad things. There’s a lot of usage of those models. It’s also compatible with, in most cases, it is actually really hard, and those people figured it out, or there was some random blip that allowed them to do it. So I feel I don’t really know how to update on that evidence. But yeah, at least in some cases, people just do manage to get around it. Maybe one distinction is also like, how long do you have to do this for? So there’s the model’s alignment that’s in the weights of the model, and then there’s your synchronous monitoring that you can do in the moment. And surely that’s going to be way less effective than how good you’re going to be at detecting things where asynchronously able to spend a ton of compute to parse through trajectories and analyze group trajectories together and stuff like that. But that takes a longer time, and so if someone needs to do an attack very quickly, then it’s just too hard for someone to detect it in a moment. But then if someone wants to pursue a very long campaign, maybe it’s harder. I’m not sure how long those campaigns take or how effective those KYC things are. But that also seems relevant for other cases that are not about leaking capabilities, but about other cases of misuse.

00:53:38 Nathan Lambert: I guess to make it more specific, how much of, say, risk proliferating from open models do you think is down to the fact that people will serve the open models or can serve themselves without the additional oversight and classifiers, versus what I think was talked about more, which is just like fine-tune the safety away, which I think very few people are going to fine-tune Kimi K3. But if these classifiers that squash 99% of people at inference time just kind of blanket a lot of the risk, I think that’s almost more plausible. It’s a bit of a sequence to get there. You have to assume people could get around model safeguards, assume a lot of usage.

00:54:20 JS Denain: Yeah. I guess the fine-tuning thing right now is not a big deal because not that many people have the combination of motivation to do it and competence and willingness to do it. The bar for that also falls as AI research becomes more accessible. But yeah, I think right now, the fact that there’s basically no safeguards, or my sense is the inference providers aren’t trying as hard as Anthropic or something at this. I could be wrong about that, but that’s my impression. And also you can pick whichever one is least secured. So I think that’s a bigger deal, at least currently, yeah, I would expect. But yeah, I think it also depends. I think the main thing that I care about for this is actual threat modeling. I just feel kind of confused about which kind of group I should have in mind as the most likely person to do a bad thing with AI. And then I feel like if someone gives you an actual scenario, then I feel like you and I will have decent takes about will it be easier for them to use an open model or to try to jailbreak a closed model? And I just don’t currently feel like I have a good sense of this. But I think—

00:55:24 Nathan Lambert: When this is so ambiguous, I think it makes me feel a bit better about the near term, which is just like these safety tools are, it’s to some capacity inadequate now, and there are strong open models, and it’s not like all hell is breaking loose. And you have to keep reevaluating this as things change. But I think it doesn’t seem like things are going to change. I don’t think there’s that much of step function changes. I personally need to learn more about biorisk that people very often talk about, and I think need to... My question is, what is the actual manufacturing pathway and where is the model intelligence at now? Where I think cybersecurity, we’re in it. We’re in what it’s going to be like for a bit, and it’s not good, but it’s not like nothing horrible has happened yet, and it’s okay. I don’t think there’s going to be more.

00:56:25 JS Denain: I don’t know what the sense with the evidence is, right? Companies don’t love to admit. I think it’s famously kind of hard to estimate cybersecurity costs and stuff. I know we want to look at insurance costs. I don’t think they’ve changed that much, but it would be interesting to look at right now. Like cybersecurity inference prices or something is like what percentages you can look at, but companies don’t love to disclose they’ve been hacked, and so they sometimes just pay. But yeah, definitely, yeah.

00:56:50 Nathan Lambert: That’s a good point as well.

00:56:51 JS Denain: I don’t feel like I have a ton of expertise there, but I’m pretty interested in those questions.

00:56:57 Nathan Lambert: I’m glad that this was a kind of like, “We don’t know, it could be worse,” and not a, “This is the list of very bad things.” Which is, it’s hard to motivate people when it’s like, “I think we need to keep preparing, and you need to assume that the Hugging Face thing might happen to you.” But it is not a super, it’s not an avalanche of compounding problems right now.

00:57:20 JS Denain: Yeah. I do expect Hugging Face-style things to happen kind of in the wild regularly and outside of OpenAI. And if it’s just that, then that’s not massively bad. I mean, it’s still a kind of screwed-up world. But I do expect more of this just happening with a variety of models and neoclouds and stuff over the next year. Just a bunch of open source agents that people can’t control.

00:57:44 Nathan Lambert: Yeah. I do think your threat model thing is a good understanding, and it’s like, who are the people that would try to take down an energy grid in a domestic area? And I think the list is pretty small. And it’s like, what are they doing? And I’m not an expert in this at all.

00:58:02 JS Denain: Yeah. I feel like the biggest safeguard is just there’s not that many people who want to do horrible things. Fingers crossed.

00:58:10 Nathan Lambert: Yeah. Okay, I have a fun question. I think it’s like Epoch AI creates so much of this kind of area setting information for different parts of the AI discourse. One thing is, is there a fun process on how you choose problems that you think are important? Are you guys constantly underwater and wanting to do different things, or is sometimes it’s like, “Now’s our moment?” Just behind the curtain of Epoch and what you guys do culturally is fun to me.

00:58:41 JS Denain: I think what determines project selection, I do think it is very what seem like the important questions to us are quite curiosity-driven overall. I think that’s probably the main thing is just what seems like a huge deal, I think has been the main determinant of where efforts go. More specifically, yeah, another framing is the Epoch was created as this thing where it seemed like neither the industry nor academia would be able to provide a good public resource is also well-maintained on the most important trends in AI. So it seemed like there was something missing. Another framing is, I know, what do people have misconceptions about, is sometimes, “Oh, people are wrong on the internet,” kind of things. Some articles can be explained by this. But yeah, I think it’s very curiosity-driven. And also it’s just like, oh, what’s evidence that’s leaking information that’s laying out there that we can use? Like, the job postings is this. Data center tracking is very much this as well. It’s like, man, these are big objects. Surely you can see them and track them.

00:59:53 Nathan Lambert: Do you think the information needed to understand the AI industry is stable in your access, like getting less accessible, getting more accessible over time?

01:00:03 JS Denain: I mean, it really depends, right? I think data center is, so far at least, it’s pretty good, and they’re getting bigger, which makes it easier. I could imagine at some point it becomes a really strategic asset, and there’s way more adversarial pressure to hide them or at least hide key information. For now, it’s pretty good, right? In contrast, Epoch is an effort to track model training compute, and at the time, you could just look at the paper. It’s much harder to do this with Astra. So yeah, I think it depends on the verticals. One thing that I mentioned before is, yeah, data is pretty rough. We’ve had some stuff on this, but it’s much tougher than compute, and it’s pretty secretive. So yeah, I think it depends on the vertical. Oh, yeah. One thing is, as the economy grows and companies go public, that’s better information, right? So that’s true for compute companies. As IPOs happen, that also helps get more data.

01:00:55 Nathan Lambert: Yeah. I find it very interesting. It’s like I’ve long wanted to know what a post-training... Like, what is a flowchart of a post-training recipe at a frontier lab? And I think it would be wild, and I think those flowcharts exist. And even some old ones, I think, would give people a lot of thought, because I do not think it looks remotely like the OLMo three-stage approximation type thing. And we have some from like Nemotron and some of the Chinese labs release stuff on it. But when you hear John Schulman talk about post-training and other people, it seems like the total Wild West and just stacking everything together. That framing for like that framing for what is the complexity of putting a model together, I think would be very useful for people as a mental model for what is being scaled and what are you scaling through. Like, yes, you are scaling, but if you’re scaling on top of this kind of house of cards thing, it’s not perfect.

01:01:55 JS Denain: Yeah. I’m actually curious. Yeah. I have this like... Okay, can I test my current mental model of this, which is pretty uncertain? My sense is this, okay, there’s a big post-training team, and it just has sub-teams that are kind of specialized. Like, some people are going to be doing math, and some people are going to be doing biology or finance or something. And the job of those teams is to ask for data and curate that data and make the RL environments good, and also figure out what kinds of hyperparameters work well for their own domain. And they do this by buying very high-quality evals that they use as a north star to iterate on and buying those RL environments. And their main output is basically like, “Here’s a curated set of environments, and here’s the hyperparameters that we want.” And then my sense is indeed now things are basically all those teams bid to put some fraction of their data into a final big RL training run. And they also bid to maybe for the hyperparameters of that final RL run to be close to what they want, question mark. That’s at least for the RL process. And then there’s in fact a big RL run. Another alternative would be, of course, they all do their experts, they distill them back into... Does that roughly what things are like? This looks—

01:03:16 Nathan Lambert: I think the org side, I agree a lot. I think the org side, there’s definitely per domain expertise, and it’s a lot of why I didn’t want to go get a frontier lab job, because most likely I would just be given, like, “You’re going to work on this data and you turn the crank.” I don’t know if the complexity is a lot higher on how you put it together. I think it’s interesting that the open weight models are split between multi-teacher on-policy distillation, and some are just like do sequential RL. I think the diversity there would be very, very interesting to the industry of whether you have multiple RL runs, how do you decide which environment to do and which of the stages? How do you manage mid-training and SFT? I think mid-training and SFT, I think, are one stage in the frontier lab, and the base model is just before mid-training, and then you do a whole bunch of stuff at the end there. And I just think there’s a lot of things you can do, and I feel like if OpenAI and Anthropic’s recipes were closer to mid-training and then big RL run, if that’s actually getting them to where they are, I think I would expect progress to go faster than if it’s mid-training and fork and model merge and bunch of experts and MOPD and then more RL. I think that just is a bit of a bottleneck on progress, and it would be very useful to have more information on what does model training look like.

01:04:44 JS Denain: My vague sense, but I don’t have super hard evidence on this, is that it has gone more in the direction of, yeah, away from MOPD with multiple experts and a ton of checkpoints, and a bit more in the direction of like, okay, there’s in fact one big run towards the end, but of course, much federation before, but I don’t know that.

01:05:01 Nathan Lambert: Yeah, maybe the rumors we’re hearing about mid-training being important and distillation is from that style of loop rather than the expert style loop. I don’t think we’ll know. I think we could continue that another time. This was fun. I think you should keep giving your incisive responses when I write something. I think this will be very useful to people and good to finally chat a bit longer.

01:05:24 JS Denain: Likewise, yeah. Thanks a lot.

01:05:26 Nathan Lambert: See you soon. Bye.

Recap: What is an open source v. open-weight vs. closed model?

Open language models are AI models where their weights are publicly available for inspection or downstream use. These are most often contrasted to so-called “closed” AI models. Closed models offer access only through Application Programming Interfaces (APIs) that developers can use to directly query a model, like GPT-4 or Claude Opus 4.5, or through products, like ChatGPT and Claude Code.

Open language models primarily are bucketed into two categories, open-weight and open-source models. Open-weight models are the most common form, such as popular models like Meta’s Llama, Alibaba’s Qwen, Google’s Gemma, or DeepSeek’s models. These models are governed by licenses, governing documents dictating what is allowed with downstream use, and are often accompanied by inference code in libraries such as Transformers, VLLM, SGLANG, etc. Since about April 2025, Chinese AI companies have been the clear leader in open-weight models.

True “open-source” models are similar to these, as they include the weights, licenses, and inference code, but they also include the complete information needed to reproduce the model – the training code and training data. The most prominent open-source models have been built in the United States, led recently by the Allen Institute for AI’s Olmo models that I helped build in my recent 2.5 years there. The other prominent open-source models are also built by American non-profit organizations, including OpenAthena’s Marin models and EleutherAI’s Pythia models.

Open-weight, open-source, and every other label for a model – including closed models primarily offered via an API – exist on a spectrum. For example, Nvidia’s Nemotron models are far more open than most open-weight models, releasing large quantities of their training data under permissive licenses, but they’re not fully open-source because they do not release all of the data. Closed models also exist on a spectrum based on what information the API reveals and the terms of use.

The state of competition between American and Chinese open-weight models (unit economics, technical capabilities, etc.)

We are living in a world where GLM-5.2 and Kimi K3, some of the latest, leading Chinese models, have enacted a step change in the commercial viability of open models — crossing a similar threshold in agentic capabilities that Anthropic’s Claude Code crossed in December of 2025.

America was the early leader in open language models, primarily through Meta’s Llama models, which were used extensively across research and commercial tasks. Chinese open-weight models surpassed American open-weight models in these two key areas about 18 months ago. The simple metric showing this is Hugging Face Downloads, where China took the lead in July of 2025 primarily through the success of Alibaba’s Qwen models. I personally maintain tools to track this data, and since I first published the American Truly Open Models (ATOM) Project in August of 2025, China’s download lead has grown to about 1.6B – with a total of 3.2B downloads, twice that of America’s total.

On popular capabilities benchmarks, such as the Artificial Analysis Intelligence Index (AAII), the Chinese open-weight models have a clear lead over American counterparts. The top three Chinese models as of writing this on September 14, 2026 are Z.ai’s GLM-5.3 and GLM-5.3-Flash and Moonshot AI’s Kimi K3 with scores of 45, 42, and 44 respectively. By comparison, the leading American models are Thinking Machines’ Inkling and Inkling Small, both with a score of 26, and Nvidia’s Nemotron 3 Ultra, with a score of 23. The top American models were released in June and July of 2026, and are updated less frequently than their Chinese counterparts. For example, Chinese labs released models with scores above these American models 2-6 months before the American companies got there (e.g. GLM-5 or DeepSeek V4 Pro). There is a trend of more American companies releasing models, including names like Arcee AI, Poolside and IBM, but they are not rapidly closing this performance gap. Other benchmarks tell a similar story.

Together, Chinese open-weight models are approximately 2-5 months behind the closed American frontier, with the open-weight American models being approximately 6-9 months behind the likes of OpenAI and Anthropic. The Chinese labs are closest in tasks with clear user demand, such as agentic coding, and further behind on more open-ended scientific tasks, such as physics or biology.

The reasons why Chinese labs can produce these strong models, despite having fewer resources than American counterparts, is still an open debate and heavily influenced by different work cultures, but is also influenced by a few key technical factors. The Chinese labs release their models faster and focus on a slightly narrower distribution of tasks, flattering them slightly on public benchmarks. Releasing faster helps them score higher because all the labs are making consistent progress, so once you “finish” a model to be released, it is a snapshot of performance at that given time — labs where that time is later tend to score higher. Still, the models built by the Chinese labs are genuinely strong and represent real competition to the American industry. This competition will not decrease meaningfully as the closed labs patch vulnerabilities in their API offerings which enable distillation.

Distillation is most impactful in new domains and does not make it trivial to create a universally strong final model. I estimate that if distillation was fully prevented, e.g. with know-your-customer (KYC) tools at Anthropic and OpenAI, the gap from the strongest American models to Chinese open-weight models would only increase by 1-2 months.

For example, the Chinese labs are rapidly changing their posture towards paying for training data in 2026. Earlier in the year, the top Chinese labs including Moonshot AI and Z.ai had a strong preference towards building data workflows in-house, but by the summer they had begun to buy the cutting edge data – challenging RL environments for agentic tasks – from both established American companies and new Chinese startups.

With the advance of open weight models in China towards the frontier of capabilities, and the recent documentation of growing risks around frontier models in areas such as cybersecurity (e.g. the OpenAI-HuggingFace incident), there’s growing regulatory uncertainty on how continued releases can enable a safer ecosystem?

A structural challenge in open-weight models is that there are few effective methods for stopping pieces of open software from reaching bad actors. If an attempt was made to restrict access to the strongest open-weight models from China because they amplify risks, the parties who would be set back are American businesses. We have an example of this – HuggingFace used a Chinese open-weight model to understand the cyberattack because closed models would not answer their requests. Thus, managing the risks of open-weight models often comes down to ecosystem preparation.

Open-weight models are becoming an essential tool for AI diffusion, and the best path to get ahead of these risks and unbalanced relationships where American companies rely on models built in China is to continue to enable investment in open models in the US. Ownership of open models allows better coordination and preparation of risks that are global in their nature while accelerating diffusion of AI services throughout the domestic economy.

Leave a comment

The state of open model adoption: How is open-source being used by academia, businesses, and other countries?

Open-weight language models have grown substantially in general interest and economic viability in 2026, allowing early glimpses of more direct ways to compare adoption of models from the US, China, or elsewhere on top of Hugging Face metrics. One example is OpenRouter usage. OpenRouter is a popular LLM inference platform that supplies a single interface to switch between models, open and closed, from the US and China. This platform is primarily known for trying different open-weight models. The platform has shared usage data for the top models since Jan. 1, 2025, and shown growth in usage from ~1T tokens processed from open models in a week of September 2025 to ~80T tokens per week today. In that time, Chinese models have grown from ~70% market share to over 80% of usage. Other platforms that are designed to commercialize open models show similar data, such as the open-source coding agent OpenCode, which shows an inference volume of ~95% or higher with Chinese models.

These open platforms are the best approximation of open model usage we have – a large proportion of open model usage is on platforms that do not disclose per-model breakdowns, such as Together AI or Fireworks AI, and in private deployments for enterprise applications.

Many prominent technology companies and startups have been building on Chinese open-weight models for their AI features, such as Harvey, the legal agent, Cursor, the coding agent, and DoorDash’s use of Kimi models, Airbnb’s use of Qwen, or Perplexity’s use of DeepSeek. These prominent companies are the tip of the iceberg, where a large swath of younger Silicon Valley startups are building on Chinese models in order to have low-cost, flexible options. There is a growing trend of American startups and companies entering enterprise agreements with Chinese model labs in order to get permission to use their models in their products – a new form of cross-border technology collaboration I have not witnessed in my career.

The foundation of innovation on Chinese models extends further into the AI ecosystem. To a first order approximation, most of academic research is conducted on Alibaba’s Qwen family of models. Having met multiple members of the Qwen leadership team during my trip to China, they are very invested in and intentional about this type of adoption, which will not be easy to claw back to American models.

To quantify the adoption of open models across academia, I scanned every paper in the 5 most popular ML categories of arXiv (cs.AI, cs.CL, cs.CV, cs.LG, stat.ML), the preprint platform popular in AI research. The results clearly track my understanding of the evolving leadership in AI research, showing LLMs becoming a foundational layer of ML research – mentions of any open model were 2% in January of 2023 and 50% in September of 2026 – and the leading role shift from the U.S. to China in the same time period.

For example, in April to May of 2023, a few months after Meta’s original Llama (a backronym, Large Language Model Meta AI, first released in Feb. of 2023), about 2,600 of 12,000 new AI/ML papers on arXiv mentioned at least one prominent open model family. Of all those scanned papers, ~5.5% mentioned Llama and ~1% mentioned a Chinese model. In the fall of 2024, during Llama’s peak, about 23% of papers mentioned Llama with about 7.5% mentioning Qwen, the most direct Chinese competition. Today, Llama has lost its lead in academia, being mentioned in about 21% of papers still, which is remarkable longevity, but Qwen’s share has risen to 30% of papers. Overall, any Chinese open weight model is mentioned in over 40% of papers, over the U.S.’s 30%, with China’s share continuing to grow.

This shows that we clearly have a lot of work to do in order to re-establish the U.S. as the home of AI research in the era of open-weight language models. There are signs of hope.

In our research, we find that American models of comparable capabilities-to-size regions to their Chinese counterparts get adopted at disproportionate rates. In the last year we’ve seen OpenAI’s first open-weight models since ChatGPT, gpt-oss, become one of the most adopted open-weight models of all time. Since then, Google’s Gemma 4 models have been some of the only ones ever to show similar adoption numbers to Qwen’s most popular small models, and Nvidia’s Nemotron models have modest adoption despite numerous more capable models at the same size point.

Share

Summary

The story of open models in 2026 is one of establishing economic relevance. This is the convergence of many stories across the AI ecosystem, summarized as:

1. The capabilities gap from open to closed models available to users has been decreasing over the last 3 years. This varies by task, but can be estimated as a 2-5 month gap in capabilities. With capabilities overall progressing so fast, this has seen open-weight AI models unlock substantial markets in 2026 and points to more inflection points in the near future.

2. Open model usage is exploding in high-value industries (e.g. software engineering, legal services, financial services), indicating an emergence of an alternative ecosystem to the best closed models. Platforms offering inference primarily on open models, from Together, OpenRouter, Fireworks, Baseten, etc., are seeing incredible growth as the first winners of an open model post-training economy (other layers include finetuning APIs such as Thinking Machines’ Tinker). This is combined with numerous anecdotes from technical staff in the AI industry that uses open-weight models such as GLM-5.3 as an alternative to Claude or GPT due to a combination of speed, lower prices, customizable offerings, and privacy.

3. Chinese AI companies are the clear leaders in open weight models. Relative to 2025, where Chinese models like DeepSeek R1 shook the AI world with surprise, the American AI labs have been recovering in their positions with open-weight models, but despite more substantial investment in the US, the Chinese labs regularly are producing notably stronger models adored by many types of users.

4. Distillation of American AI models by Chinese labs does not explain the entire story of their success. Distillation is an industry standard technique of training another AI model on the outputs from a usually stronger model. The technique is most prevalent in the Chinese AI industry, which has used basic exploits to extract reasoning traces and additional data from American companies’ products that are not fully secured. The best estimates are that distillation helps reduce the performance gap of Chinese companies relative to the American frontier by 1-2 months.

5. Chinese models, particularly Alibaba’s Qwen family, are established as a foundational layer of research and development across academia and local model users. In recent months, Chinese open weight models were mentioned in 38% of AI papers, above the U.S.’s 28% – and the Chinese share is growing much faster than its American counterparts. This, along with other political factors and the closed nature of leading American AI companies, is contributing to an accelerated decline in America’s lead as the preeminent AI research hub in the world.

6. Open weight models are entering the capability levels where new risks, e.g. cybersecurity, can be enabled by numerous open-weight models being available, necessitating an ecosystem level response in preparation. This new era of risks is also enabling a period of political uncertainty, where there is regulatory attention on the strongest AI models, but massive uncertainty on how policy would be legally enacted. At the same time, many researchers and engineers rely on open models due to more permissive safeguards, where the closed models such as Claude and GPT often refuse critical cybersecurity defensive work or biology research.

For more data, view the Interconnects Dashboard.

Conclusions

In 2026 the Chinese labs are clearly maintaining their status as the leaders of the open-weight AI ecosystem. This comes as open-weight models have passed an inflection point in economic viability and in the face of increased activity from American labs as model competition. The leading Chinese labs do not appear to be meaningfully challenged, as they expand their enterprise and research adoption globally.

This landscape of open models comes at a crucial time in the broader AI ecosystem. We’re seeing OpenAI and Anthropic take massive steps forward with their latest public models, and at the same time call for coordinated care on how we manage the next stage of AI progress. What is happening in the confines of a few AI labs today, especially with extreme talent and compute density, is a precursor to what will soon emerge in the open model ecosystem. Open models are going to be the substrate for everyone else in the world outside of the few true frontier AI labs, to harness an acceleration in software engineering and other computational practices. This represents a substantial source of soft power, influence, and potential for the organizations that enable this broad access to transformative intelligence.

With this future coming soon, we need to collectively stay humble about the exact path open models will take. There are a lot of unknowns with open models – e.g. we don’t have good data on how they’re used in countries other than the U.S. and China. With the distribution of ML training expertise being broad, i.e. tens of organizations and thousands of people that are within a year of the frontier of capabilities, it is a matter of when, not if, open models cross the performance thresholds that enable new workflows. The collective approach should be to understand how to use this broadly accessible, open intelligence for good while proactively mitigating the potential harms.

Thank you to Florian Brand and Kevin Xu for feedback and/or suggestions for this work. For more research informing this post, see the open-source AI reading list.

A core perspective I have is that the frontier labs and broader frenetic, competitive culture in the San Francisco AI scene set up an environment that amplifies any AI concern. This has some benefits in causing more general audience awareness of AI, as fear sells, but exaggerating risk timelines or severity will have negative second-order effects. I remember many loud AI safety debates, and their associated clouds over the viability of open-source AI, in 2023 and 2024 — the primary risks then did not arrive in the forecasted timelines.

The general populace of these two key labs was very anxious about AI risks and the rate of progress even a year ago, and especially as agents got stronger product-market fit at the start of 2026. This cultural precondition, when exposed to the reality that thousands of agents will constantly be working fairly productively in your business, will only increase this anxiety. The step from this anxiety, and incidents like OpenAI-HuggingFace, to extinction risks feels very religious.

Share

Richard Ngo had an apt summary of the situation:

Now a large proportion of the AI safety community is implicitly or explicitly orienting to futures where an intelligence explosion occurs within a few years. My default expectation (absent an extensive pause) is that a similar thing will happen: they’ll turn out to be directionally correct (relative to the expectations of almost anyone not linked to the community) but factually wrong. Specifically, we won’t have superintelligence within the next 8 years, but things will still be moving so fast that it’ll *feel* like the people who argued for short timelines were right.

… I wanted to say something now because it feels like the level of bandwagoning towards “singularity soon” is getting pretty wild.

Personally, I think this view aligns closely to what I outlined in my alternate scenario to true recursive self-improvement (RSI), which I called lossy self-improvement. A summary of this view is that:

1. Automatable research is too narrow to achieve a massive net acceleration in progress, in the face of scaling laws’ exponential costs,

2. Diminishing returns of more AI agents in parallel are real, &

3. Resource bottlenecks and politics are a major factor in building strong LLMs (and AI can do much less to accelerate this).

So, I’m left balancing the above, latent increase in the cultural temperature with the potential that the labs have seen genuinely scary, specific breakthroughs that are not public yet. My expectation is that more of the current AI safety concern is on the former – scaled agents working – but I hold high levels of uncertainty here. Foundational, imagination-based AI breakthroughs are the sort of thing that would make me update my RSI timelines from closer to a tool to sustain progress in the face of exponential costs (scaling laws), to something more unpredictable and/or unstable.

Some of the best recent resources on RSI have been Dwarkesh’s podcasts with Noam Brown and the trio of John Schulman, Beren Millidge and Charlie O’Neill. I have a few important reflections from both of them.

First, the podcast with Noam Brown made me internalize how big of a short-term acceleration mass inference capacity is. These labs will throw thousands of agents at important, measurable problems. At the same time, compute capacity available to them is going to continue to scale. I have my doubts that the labs can afford to spend a constant portion of this compute on internal R&D as the total volume goes up, especially with plans to IPO, as they face increased scrutiny on basic economics. It is important to not confuse massive steps in inference-time scaling, a dynamic which should be fairly predictable, with being the outputs of RSI, which is highly uncertain.

Second, the trio podcast debating the state of the art in technical capacities induced more of a surprising reaction that I haven’t fully settled. Through the first hour or so of this podcast, where they debate the role of RL, distillation, scaling, inference-time compute, etc., I found myself strongly agreeing with the distribution of claims. A TLDR would be that our current techniques work and let us solve problems we know how to state, but they don’t result in a magical level of generalization to unknown, harder problems in most partially verifiable domains (i.e. progress in math is an exception, rather than a rule).

The surprise of this podcast was the end, where they were predicting timelines for various thresholds of AI. I had GPT-6-Astra summarize the answers provided to three questions from Dwarkesh, of the form “when will AI reach X ability”:

All timelines are relative to the interview date.

• Drop-in remote worker for broad white-collar work over a month

  • Charlie O’Neill: ~1 year with programmatic access to workplace tools; ~2 years if it must operate through a browser. Means ordinary white-collar work, not highly creative research.

  • Beren Millidge: ~3 years for full generality; 80–90% coverage sooner. Main uncertainties: online learning and the long tail of tasks.

  • John Schulman: ~1 year for an “okay” version, with uneven capabilities that improve over time.

• 10× productivity uplift for AI researchers

  • Charlie O’Neill: 5–10 years. Bottleneck: absorbing information and deciding which experiment to run next.

  • Beren Millidge: Finds John’s ~2-year estimate plausible, but gives no independent timeline. Assumes AI can run successive experiments and learn from feedback; other bottlenecks would remain.

  • John Schulman: ~2 years.

• AI surpassing top human experts across all computer-based work, including multiyear projects (“ASI”)

  • Charlie O’Neill: 5–10 years. Highlights limitations in memory and context length.

  • Beren Millidge: ~5 years for areas labs focus on; potentially longer for literally every domain. Gives no firm timeline for the universal version.

  • John Schulman: 3–4 years. Spatial/physical fields may take longer; requires onboarding and solving longer-horizon learning.

Roughly, a recurring problem when discussing RSI is a lack of specification in intelligence. The jaggedness of intelligence means that we need to discuss thresholds in specific, measurable tasks. The nature of LLMs’ intelligence is shaped very differently than humans, and the roles we forecast are human-shaped. AIs, therefore, do not cross these thresholds like remote worker or AI researcher discretely. It’s a slow diffusion, and a form of long tail will always exist.

Take the case of productivity of AI researchers. Many people under-index how much of science is communication and standard setting with colleagues. I do buy the cycle of experiment design and testing being 10x faster in the near future, but not hypothesis generation and intuition building. Accelerating understanding will be the key bottleneck – and it is one that despite all of the AI tools getting massively improved, humans will only improve marginally in their capability. A big improvement in the nature of science will be enabling humans to invest more time here, not them becoming exponentially better at it.

Leave a comment

This links back to the Noam podcast. Agent swarms in the near future will be effective at solving clear, open problems with verifiable answers. In this vein, when it comes to improving AI models, RSI is much more helpful at efficiency rather than expanding peak intelligence. This is due to the fact that LLM serving has clear metrics you want to improve that are measurable and malleable. This’ll enable better inference-time scaling and more efficient multi-agent systems.

Still, I cannot get past the fact that all of our scaling laws show that you need exponential compute and resources to make linear improvements in intelligence. RSI is poised to make modern LLMs vastly cheaper. Trends that have shown LLMs get exponentially cheaper at a given intelligence are likely to accelerate. A crucial factor for the labs will be increasing margins as revenue could potentially have negative pressure if there’s fierce competition in lowering prices at a fixed intelligence level — Jevons paradox will likely prevail, resulting in strong businesses.

RSI factors will have a much harder time improving pieces of the LLM puzzle like managing complex post-training recipes. There were a few quotes from John Schulman that I strongly agree with on the state of post-training at the labs:

If I think about a post-training team and why you need a lot of people on the team, it’s just because there are a lot of different areas where you have to figure out how the model should behave. It would be very hard to automate the whole thing, just because someone has to think about how the model should behave in this area.

and later:

It’s really easy to screw up post-training in some way that doesn’t show up in benchmarks.

These tasks are uniquely hard for current LLMs. Yes, they’ll get better as the industry is still rapidly scaling RL environments related to these domains, but this paradigm does not last forever. In the near future, it could become exponentially harder to conceive, build, and test new environments that meaningfully challenge the leading LLMs – these hard environments are the ones that are crucial as a learning signal in RL.

OpenAI and Anthropic have shared a good amount of internal measurements related to RSI, and my current read is that the biggest takeoff in automation within the labs is in tasks like software engineering, monitoring logs, managing planned experiments, and other fairly routine (but not always easy) tasks. For example, I was surprised by this language in the recent Claude Fable 5.1 & Mythos 5.1 System Card:

We believe that internal usage of recent AI models has been a key factor in maintaining the current rate of progress, but we do not yet see clear signs of dramatic acceleration beyond that rate.

Altogether, I think the hardest exponential we are fighting is on peak intelligence. That is the hardest one to budge or even accelerate. Still, my mental model for the very early innings of RSI is more of massively scaling and diffusing inference-time compute to AI research and related activities, which has a large amount of low-hanging fruit available. This, on its own, is still poised to be economically transformative. It may also unlock more resources to push on AI diffusion, which is the crucial bottleneck in unlocking much of the potential benefits of AI.

For now and until more evidence emerges, lossy self-improvement remains my baseline on the trajectory of progress, and the increased discussion of extinction risk seems very misplaced. As always, things can change fast in AI.

This is my list of the best writing on open models in the last few years. If someone decides they want to get up to speed on the area, reading this will be a comprehensive overview of the state of affairs. Please comment pieces to consider adding below, and I’ll update this over time.

List last updated: 26 Oct. 2026

Share

Foundation

What open models are, why people release them, how they relate to business strategy, and what the risks are.

• On open source AI strategy, a walkthrough of how open-source software has been used by businesses and early signs of what that means for AI — From Open Source Software to Open Source Strategy, Bill Gurley (May 2026).

• One of the clearest articulations is Mark Zuckerberg’s comments around Llama 3’s release as to why Meta releases open models — Open Source AI is the Path Forward, Mark Zuckerberg (Jul. 2024)

• Why you should view open models on a gradient, rather than binary open/closed, based on factors such as licenses, cost of running the model, data access, etc. — The Gradient of Generative AI Release: Methods and Considerations, Irene Solaiman (Feb. 2023).

• The role open models will play in the economy of the future, as a complement to strong closed models. Why open models will be used to create custom agentic workflows in enterprises across the world – What comes next with open models, Nathan Lambert / Interconnects (Mar. 2026)

  • A position on how open models will capture value by providing a complementary tool to large swaths of the existing economy, drawing on the history of IP and current debates on open vs. closed models (e.g. distillation) — Some Simple Economics of Open versus Closed AI, Christian Catalini (Aug. 2026)

  • Why open models will constantly be behind closed models in performance — Open models in perpetual catch-up, Nathan Lambert / Interconnects (Feb. 2026)

  • Where adoption differs for open and closed models — Open and closed models are on different exponentials, Nathan Lambert / Interconnects (Jun. 2026)

• A clear articulation on how to balance releasing powerful open-weight models while taking safety seriously — A Safe Path to Open Weights, Thinking Machines Lab (Jul. 2026).

  • Early paper on marginal risks that showed text-focused LLMs very marginally increased documented potential risks of models — On the Societal Impact of Open Foundation Models, Sayash Kapoor, Rishi Bommasani et al. (Feb. 2024).

  • Closed models safety guardrails are regularly bypassed causing a plethora of real AI-risks before hypothetical risks of open weight models have emerged — The Myth of unsafe Open Source AI, Florian Brand (Jun. 2026).

• The mass reduction in open data, which is a crucial factor that has hampered truly open AI research — Consent in Crisis: The Rapid Decline of the AI Data Commons, Shayne Longpre et al. (Jul. 2024).

• Recent examples on how strong Chinese models impact the AI ecosystem — Kimi K3: The open-weights escalation, Nathan Lambert / Interconnects (Jul. 2026) / GLM-5.2 is the step change for open agents, Nathan Lambert / Interconnects (Jun. 2026).

  • A summary of the story of open models in 2025: Nathan Lambert on China’s AI Ecosystem and the Open Model Gap | The Curve 2025, Golden Gate Institute for AI (Nov. 2025).

• A lecture surveying all of the above topics: Open-source history, licenses, open model history, adoption numbers, open risks, and balancing the benefits of openness. Nathan Lambert — YouTube (Oct. 2026)

• [Optional] Latest data on open model adoption:
  A general summary on US vs. China model adoption — The ATOM Report (Apr. 2026),
  The latest data on model downloads, derivatives, and research adoption by region — Interconnects Adoption Dashboard, and
  The most important models to know about in the ecosystem — Interconnects Artifacts Hub

  Interconnects AI is a reader-supported publication. Consider becoming a subscriber.

US-China Competition

Who is leading in open models, how this has changed over time, how China maintains its leading position, and relevant history.

• Why the U.S. needs to invest in open models for fundamental R&D / innovation in the face of growing competition from China – The ATOM Project, Nathan Lambert (Aug. 2025)

  • The lens as to why open models help spur research innovation and beneficial outcomes for AI — Why I build open language models, Nathan Lambert / Interconnects (Oct. 2024)

  • Why open models foster education, innovation and competition, three core American values — Banning Open Source AI Would Be A Mistake, Nathan Lambert & Kevin Xu (Jun. 2026)

  • Why the recent “vibe regulation” / vague federal oversight mechanisms set us up for a clash and-or ban of frontier open models in the near future — 6 months to live for open models, Nathan Lambert / Interconnects (Jul. 2026)

  • [Optional] Fully open language model technical reports to illustrate the start of the art in understanding: Pythia (EleutherAI, 2023), Olmo (2024), Olmo 2 (2024), Olmo 3 (2025)

• Chinese open-source history leading up to AI — Chinese Open Source: A Definitive History, Kevin Xu (Mar. 2026).

  • China’s structural advantages in open-source — China’s Structural Advantage in Open Source AI, Kevin Xu (Jun. 2025).

  • How Chinese labs themselves discuss building models, and how the Chinese industry differs from the U.S. — Notes from inside China’s AI labs, Nathan Lambert / Interconnects (May 2026).

  • Why Chinese labs are so good at keeping up with American competition (e.g. American open weight labs struggle to compete with Chinese labs on fair performance comparisons) — GLM-5.3: How Chinese labs keep stride with the frontier, Nathan Lambert / Interconnects (Aug. 2026).

• Prominent uses of Chinese models by Western companies have prompted meaningful regulatory attention (more discussion)

  • Lawmakers have probed the following companies over using Chinese models: DoorDash (CNBC, Jul. 31 2026), Airbnb (Bloomberg, Apr. 29 2026; Semafor, Apr. 29 2026), Anysphere / Cursor (Bloomberg, Apr. 29 2026; Semafor, Apr. 29 2026), Apple (Reuters, May 17 2025), Harvey (Aug. 2026)

  • Other western companies have very publicly shifted the models they use from American, closed labs to Chinese open models to save costs. Examples include Perplexity prominently and rapidly adopted DeepSeek R1 (Forbes, Jan. 28 2025) and Thomson Reuters building on Qwen to move off Claude (Business Insider, Aug. 24 2026)

Leave a comment

Technical Details

What is distillation and how much does it help Chinese labs, how do open models impact frontier AI risks like cybersecurity, and how far are open models behind the closed frontier?

• The open-closed model gap has reduced in recent years, and is now at roughly 4-6 months. The leading open models have all come from Chinese labs since ~2024.

  • SemiAnalysis article which ran independent evaluations, concluding that open models have been getting closer to the closer frontier of performance over time — Are Open Models Catching Up?, SemiAnalysis (Aug. 2026)

  • Open models are on the Pareto cost frontier, while not at the absolute performance frontier. E.g. DeepSeek V4 Flash, see evaluation and cost on Artificial Analysis.

  • Data sources from Epoch AI and Artificial Analysis (and U.S. v China, related) showing the open-closed gap over time.

  • An independent analysis of the open-closed gap across a mix of public and private evaluations — How far behind are open models?, Håvard Tveit Ihle (May 2026)

  • E.g. in 2025, the product lead of Z.ai said with respect to their release time “Get it out fast. We open source it within a few hours.” — The Z.ai Playbook, ChinaTalk (Nov. 21, 2025)

• Cyber, risks & open models (I plan to develop this further)

  • Why we cannot effectively ban open models as used by bad actors for cyber capabilities (they will always have access) — The OpenAI/Huggingface incident; how we should manage the imminent arrival of autonomous hacking too cheap to meter, Joshua Saxe (Jul. 2026)

  • What the government should do to observe, orient, decide, and act with respect to emerging cyber threats (versus blocking models based on in-house capability assessments) — We urgently need a coherent national AI cybersecurity policy, Joshua Saxe (Aug. 2026)

  • Why you cannot expect to control access to AI at a certain threshold (e.g. open weight models) and need to prepare society to tackle risks downstream of available intelligence — Nonproliferation is the wrong approach to AI misuse, Helen Toner (Apr. 2025)

• Distillation – the process of training on output tokens from another model – is the single most eventful debate around open models in 2026.

  • For basic background, see a textbook chapter on synthetic data & distillation generally, from Reinforcement Learning from Human Feedback (post-training textbook published in 2026)

  • How distillation helps the Chinese labs, but doesn’t take away from their innovation — How much does distillation really matter for Chinese LLMs?, Nathan Lambert / Interconnects (Feb. 2026)

  • A very transparent documentation of how Chinese company use Anthropic’s products and circumvent the terms of service or intended use. The report details at-scale usage of Anthropic’s products by banned parties, as a mix of technical distillation (mentioned via SFT data) and extensive routing of Claude into their products and services without telling users —Detecting and countering misuse of AI: September 2026.

  • A recent paper that showed that the frontier labs had implementations in their APIs that made systematic extraction of reasoning traces (the crucial part of modern training) through clever tricks. Recent distillation paper, my writing on it — Stealing Reasoning Traces from Proprietary LLM APIs, Panfilov, Schmotz, Shumailov et. al 2026 (more on X). Anthropic confirmed this technique was used by Chinese labs.

  • Why the political panic over distillation, claiming that distillation is the only reason Chinese models are close to the frontier, is not grounded in the evidence — The distillation panic, Nathan Lambert / Interconnects (May 2026)

  • How labs can use distillation to improve models in an era of scaling RL environments across agentic behaviors — How distillation is used today and what performance uplift it gives to open models, Nathan Lambert (Jul. 2026)

  • [Optional] More history: In 2024, I wrote Frontiers in synthetic data where the key points were that synthetic data, primarily in “distilling” models by training with SFT on outputs from a stronger model, was the dominant form of distillation. Frontier labs had been shifting the logit-based, knowledge distillation, confirmed earliest in Gemini and continuing to this day. In early 2025, there was substantial debate on if DeepSeek-R1 was distilled from OpenAI’s o1 model. There is no clear evidence suggesting that they did, and in Apr. of 2025 I wrote confidently that DeepSeek did not distill. At the time of R1, it is more possible than I gave it credit to that DeepSeek did distill some o1 traces to make it easier for them to train their R1 model – based on the above reasoning trace extraction methods. This does not take away from the innovation of it, but it’s worth being realistic and is a way that distillation could accelerate China closing the gap to American labs.

How did we get here? Why did this quitting announcement reach so far? In many ways, the rest of the world’s views around AI in the past was a dampening factor. You can think about this like the damp ground around a fire. Many people were striking matches for years about AI risk – they’d smolder in their community and largely burn out, going unnoticed. As the stakes of AI have risen this year, from the OpenAI-HuggingFace incident and breakthroughs like the Navier-Stokes result (also from OpenAI), the ground has dried out and the latent energy around the AI discourse has increased. More people not in the industry have thought, “huh, maybe I should care about this AI thing.” The ambient temperature and stakes have been obviously rising.

Then, some basic factors of human nature apply, with the most crucial being that fear sells. Fear is the simplest story, the one people cannot look away from. Jacob Coxon was the one who stumbled into this new powder keg, totally unaware of what was going to come. What looked like a fairly innocuous event – another AI researcher quitting citing safety risks – landed into a very different environment and it caught like wildfire. The discussion of existential risk, mass extinction, and the trajectory of AI has traveled further than even the most seasoned AI commentariat would ever predict.

Share

There are a set of facts we need to get clear, which paint the picture of the situation. The key Tweets to reference are from Jacob Coxon, the resignation thread, and Evan Hubinger, the source of the >10% extinction risk figure .

1. There are plenty of AI risks which are likely to cause harm, even if estimating annihilation is useless. It is important to weigh these with respect to the benefits. The entire discourse around existential risk is on very poor footing. At least Evan was clear in his post, with “kill all humans,” but a major problem in the AI Safety discourse is that people talk about existential risks, when they mean very different things (much like how AGI is a vaguely meaningless term). I put the probability of complete extinction as being so low it isn’t worth discussing, but the probabilities of AI caused disasters – e.g. cyber attacks on critical infrastructure or bio-risks – as being worth debating. Throwing this whole discussion out because there are not these disasters yet is a harmful reaction.

2. Jacob Coxon is acting genuinely and with good intentions. The outpouring of support from more well-established AI researchers who know of him and his intentions of resignation is useful. Many factions of AI turned to scapegoating him individually, based on account metadata, personal factors, etc. These are not useful. Many frontier lab employees genuinely have similar views to him. I’m not sure it’s a majority, but there is a substantial group.

3. Many frontier lab employees, especially at Anthropic, are out of touch and this will impact their forecasting and/or descriptions of current AI events. I say this without blaming individuals, but it’s a common agreement among my friends not at OpenAI/Anthropic (Ant especially) that people at the labs operate with a religious energy. It’s very common to go through very out of touch interactions with them. I do not blame most of the individuals who get distorted views being part of these companies, but the interactions are wild and spill over into a lot of wack discussions in the AI media ecosystem. Living in this environment that normalizes such out of touch behavior will inevitably distort any human’s understanding of technical progress.

   Interconnects AI is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

4. This was not a mass political campaign, but rather an opportunistic media coordination. For context, the Wall Street Journal had an exclusive story that Jacob coordinated before posting. I suspect that Jacob shared his plan of quitting in groupchats with AI safety advocacy groups ahead of time, e.g. the morning of posting, asking for amplification. This is normal practice, and could have included some prominent politicians. From there, I think it’s more likely that other politicians are bandwagoning on a rising issue. When you combine this with other factors, like Daniel Kokotajlo’s appearance on Joe Rogan coming out the same day – it definitely looks like a very well-executed, coordinated media campaign. This doesn’t mean it’s a conspiracy or a regulatory capture tactic within Democratic political structures. The determining factor seems to be that no one – including Jacob and those posting about X-risk today – knew that it would go so viral.

5. We do not have proof that RSI causes the risks these researchers forecast. The general argument for RSI follows as: The current pace of progress is very high, the current progress is heavily dependent on AI tools, the current AI tools are superhuman in some domains (e.g. math) – so, all together, AI is going to work more on itself and become superhuman in all relevant areas over time to autonomy and intellect. This view dramatically undersells human bottlenecks in building models and allocating resources at organizations, and draws conclusions on future AI capabilities more broadly.
   
   I called my alternative view to this, Lossy self-improvement. AI has always been very jagged, and we are making models which are superhuman goal-seekers at math and software engineering, but they have massive limitations on intuitions, creativity, and other types of reasoning that humans are strong at. With AI agents assisting research, we will rapidly find the areas where AI is superhuman – and I expect there to be well more than just research mathematics – but it won’t be a panacea for the current limitations of our approaches to LLMs.

   
   There is another understandable social dynamic at play here, causing many deep AI insiders to overstate the returns from RSI. Many of these researchers were the earliest people to bet on AI’s progress, and the extent to which they were visionaries should not be downplayed (see Ilya’s comments on deep learning as early as 2015). They have been right again and again, forecasting AI’s capabilities better than I certainly could have guessed. This does not, though, mean that their forecast of what will come next will be right. The core idea of RSI is a way to spend more compute on the process of developing a model recipe, rather than just spending more compute on the training run itself. We’re seeing benefits from it, but I argue the expected return on that input is far less than they believe.

   
   Their argument is that RSI will make AI progress go exponential, make it so we cannot monitor the technology, and enable rogue models and new forms of risk. This scenario is often called “Fast Takeoff”. We have not seen the stacking efficiency gains that massively reduce model size and cost, that would lead to an explosion in progress by allowing consistent speedups in experimentation.

6. The biggest short-term risk could be from the AI labs not taking safety seriously enough – they haven’t hardened their own infrastructure, enabling AI misuse to proliferate. From my earlier post on the HuggingFace-OpenAI incident, Lessons from the hacks:

   4. Frontier labs do not seem like they’re watching the models closely enough, due to a general frenetic competitive environment & current SF culture

   From OpenAI’s own retrospective, the misaligned model behavior was unfolding over months, and in some cases OpenAI did not know about the hacks for ~weeks. The time to response is too long and I do not think this is an OpenAI only characteristic – rather it is that the frontier labs continually seem underwater in the amount of work they feel like they should do. I am not optimistic in the long-term that the labs change a sufficient amount here to meaningfully mitigate this type of oversight risk in the future. Yes, it is very likely that OpenAI is putting a ton into understanding this – and delayed their latest models to make sure they get it right – but the financial pressure to grow revenue or risk the companies’ long-term balance sheets makes me think it will not be a sustained pattern of caution.

Overall, I think this episode is very bad for the AI ecosystem. It’s pushed the acceptable views in the AI community closer to the extremes. More accelerationists will discount the need for any form of safety, citing mass delusion of the “doomers.” It feels like a very narrow path to believe in AI risks, but to not worry about extinction from the technology.

For example, it is a horrible temporary period for cybersecurity, where AI models going a bit off script and poking around unintended pieces of the web seems like a new normal. This is accelerated by the labs competing veraciously towards their views of AGI, and a slow uptake in the necessary hardening of our cyber infrastructure around the world. This doesn’t mean that it’s an existential risk and something we cannot solve. Each risk will have its own set of solutions and paths forward.

I feel particularly exposed in the current environment as a supporter of open models. If an open model were to be used by a third party organization to intentionally hack another company — similar to how the OpenAI-HuggingFace incident went down, but intentional — my expected outcome would be a severe restriction on the development of stronger open models going forward. Open models are needed for many organizations to perform this cyber hardening, and to maintain the ability to adapt to new forms of AI risks in the future.

Through all of this, we need to stay grounded on what is actually unfolding. Yes, monitoring AI’s behavior is heavily reliant on other AI models, which adds in new types of monitoring risks. These are not inherently insolvable. A recurring read of mine on the emerging agent swarms is that they’re attempting to do a task given to them, and they’re using skills we didn’t know they yet had to circumvent the intended path to success. This is a huge win, as when you squint, the AIs are doing what we told them to do. The models are certainly very odd, and we should accelerate our progress on understanding them, but these swarms are far from being novel independent entities. The models are trained to coordinate on tasks, to write down their progress, and to be extremely persistent. There will be new oddities we find in the future, but prescribing current uncertainty on how AI works to future certainty that we cannot understand AI is a form of giving up.

In this world, we need to rely on the rule of law and science. If the AI labs are not able to do enough safety research themselves to understand the models, they should be more transparent on what is happening so more scientists can make progress on the problem. If an AI lab commits crimes unintentionally, they should be punished, so they have clear incentives to prevent it in the future.

It is a natural reaction to things changing very fast to feel more uncertain about how to create good outcomes — that is actually the correct mental update. We need to use this humility to motivate ambitious solutions.

Many AI optimists tend to compare what is happening in this AI boom to the industrial revolution, or to other periods of rapid technological advancement and diffusion into society. These comparisons fit on the scale of technological change, but miss a crucial factor in how most people are exposed to that change. The problem facing AI is that most people have no super tangible new goods thanks to it and society has more inertia resisting change than in previous eras.

Share

I’m writing this coming back online from a few weeks off for my wedding in New England. In this time it would have been very easy to not think about AI at all. The touch-points that average people have to AI products today are fringe, marginally beneficial, or even just very confusing to them (e.g. many people have heard about and brought up the OpenAI-HuggingFace incident, but don’t know what to make of it). People on the positive side think of AI as a way to make fun images, enhanced Google Search, etc. These are very small benefits. On the negative side is an association with addictive social media algorithms, friends of friends addicted to AI chatbots, and a plethora of takes on data centers.

AI is still a rounding error in everyday life

Core aspects of everyday life — family, food, transportation, and entertainment — have few direct impacts yet. It’s a remarkable breath of fresh air to pop out of the bubble and realize how little what is happening really matters today. Being obsessed with AI is a choice that a very few people have yet opted into. For example, the only thing I used AI for in this time was search and creative work (making the pretty seating chart for my wedding guests to find their table).

In industrial revolutions past, average people got absolutely life changing outcomes. The First Industrial Revolution in the late 18th century gave access to cheaper clothing, cooking ware, reading material, and a shift to new livelihoods. The Second Industrial Revolution in the late 19th century introduced household machines (e.g. sewing machines), preserved food, indoor plumbing, photography, better light sources, bicycles, and further benefits of manufactured goods and electrification. The list is remarkable — most of these we still use regularly today — and very physical.

While even the most optimistic versions of AI will usher in new scientific discoveries, advanced therapeutics for rare diseases, and potentially even sustained economic abundance, these benefits have the risk of being too indirect.

How will a common citizen come to credit OpenAI or Anthropic for saving their life, if they went to their family doctor who told them about a new miracle cure?

What percentage of Americans will care about OpenAI solving the Navier-Stokes Millennium Prize Problem?

It feels very likely in 50 years that the average American’s day to day life looks very similar. Their home, appliances, relationships, and vehicles will be similar (of course, self-driving will continue to diffuse, but that has been developing on a very independent trajectory from the innovations of LLMs). In this time, AI will get a lot of credit. 50 years is a remarkable length of time with how fast everything is changing today in this, AI-focused narrow slice of the world.

The most important part of what is happening early in the AI revolution, is building foundational infrastructure, and a general process, which will compound over decades. A major mathematical breakthrough today will look astonishingly minor in scope relative to the advancements that come later in the compounding journey. It is hard to predict what it looks like for every technology you use daily to get faster compounding improvements due to AI.

Breaking social stasis

Much of the narrative around AI is trying to push people to care, due to this long-term reality of progress, at least as a subconscious motive. This will take a long, long time to get right, and AI’s buildout faces immediate political problems due to this imbalance.

Today’s AI is primarily a tool to serve the elite. For knowledge work, which is roughly half of the U.S. economy, AI is as fundamental as electricity (or quickly will be so, with rapid improvements to agents in the next 18 months). It’s highly destabilizing to have such a transformative, productive tool only bring half of society along. It’s not hard for many people to pick up on this — the technology economy booms while life stays otherwise stagnant.

In writing this, I learned of Engels’ pause, which is “the period from 1790 to 1840, when British working-class wages stagnated and per-capita gross domestic product expanded rapidly during a technological upheaval.”1 If we — the leaders of the AI industry — think this is the closest analogue to what comes next for AI, those not benefiting are right to push back.

AI is the greatest tool ever for scaling technology companies and starting new online-native small businesses. I don’t even expect the tech industry to grow in headcount and nurture its workers through an era of massive success — I agree with that headcount would likely shrink while knowledge work output explodes. These sectors were already the most successful in the American economic system, so the brand of AI will be tarnished as not being a collective good. I worry that this instinctive reaction will kneecap AI’s development, sending it down a path that looks closer to the cautionary tale of American nuclear power.

Part of the challenge is the speed and relentlessness of expectations in society. The AI industry has millions of eyes on it, and won’t get much patience to wait and bring innovations later. If given 100 years to diffuse into society, its impacts will certainly become much more obvious, like the industrial revolutions of centuries past.

Together, the AI industry is facing a few simple issues, in what I would call the first half decade of 50-year diffusion process.

1. AI’s positive impacts early in its evolution are too indirect.

2. AI is facing a political backlash deeply intertwined with the history of Big Tech in Western society. This is only an AI story due to timing, and if AI’s exponential growth came decades after the growing pains of today’s technology platforms like Google and Meta, it seems likely that the datacenter issue would’ve never risen to such a central political position.

Solving either of these would alleviate a substantial amount of pressure, and give the AI industry a lot more time in showing the positive case for why people should be okay with changes to the status quo (primarily economic). These are both made more challenging by AI self-labeling itself as negative and/or unsafe technology, through proclamations of doom and mass unemployment. The leading figures have begun addressing this issue, but the public needs more work to fully buy into the overarching trajectory.

When zooming out long-term, I could see robotics and self-driving becoming closely linked in storytelling to the current AI revolution. If the intelligence explosion from mass-producing large language models does spill over into enabling the acceleration of robots in everyday life, humans will quickly latch onto the tangible benefits of AI. This is ironic, as many people have spent time trying to convince people that what is happening specifically with LLMs is very different than the previous decade or two of general AI progress. If the same dynamic later saved (or massively overshadowed) LLMs, it would be funny.

Reflecting on what I expect the history of this era to look like, it feels a lot like growing pains of AI. Society needed to break out of old habits and work through problems that predate ChatGPT — which releases a lot of energy and frustration — in order to tap into the longer term growth. The diffusion story will take a lot longer than the fight against it. All of us younger folk following the story today will get to see powerful AI go from effectively 0% to 90%+ full adoption in our lifetime. This sort of AI that is deeply integrated in businesses, acting as personal assistants, etc. is just starting to become viable. It’ll take far longer to gain adoption than easier to understand applications like ChatGPT, and is the true marker of AI’s evolution.

Taking this perspective makes it clear that it is crucial to keep progressing the technology — the benefits will be astounding, but they are not a given — and we have a lot of very hard work to do in making sure they’re distributed widely.

1

Doug at Fabricated Knowledge had a good piece on this too:

Fabricated Knowledge

Engels' Pause and the Permanent Underclass

Read more

6 months ago · 214 likes · 5 comments · Doug O'Laughlin

In 2026, open models are more competitive than ever, which has led to two interesting developments: Western model makers adopt open licenses, with both Google and Meta switching to Apache 2.0. Chinese model makers at the frontier, however, are becoming more restrictive: Kimi K3 comes with a license which requires commercial agreements for those who run inference or fine-tuning services, and MiniMax M3 requires agreements above a revenue threshold and has prohibited use cases.

The newest addition is Zhipu’s GLM-5.3, which switched from MIT (GLM-5.2 and earlier) to a custom license with the following clause for inference and fine-tuning providers:

If the Licensee or any of its affiliates operates a Model as a Service business, and the aggregate revenue of the Licensee and its affiliates exceeds 10 billion US dollars (or the equivalent in other currencies) in total over any consecutive 12 months, the Licensee must pass Z.AI’s security review before using the Software or its derivative works for any commercial purpose. The scope and method of the security review shall be reasonably determined by Z.AI.

While the 10 billion US dollar threshold is very high compared to other licenses of this kind, “affiliates” is not defined in the license, which adds uncertainty and creates barriers to adoption. Furthermore, the license is provided in both English and Chinese, with the Chinese text using “关联方” for affiliated parties, which does have a definition in Chinese law.

We are by no means legal experts and there are obvious reasons why those licenses are created. However, we want to highlight the issues that come with creating such licenses, especially in a world with a lot of valid open and closed alternatives.

Share

Our Picks

• Motif-3 by Motif-Technologies: Motif is one of the few hidden gems out there, showcasing innovation in their model training with very limited resources compared to others. Motif-3 comes with an MIT license and impressive scores for its size. Given the trajectory of model releases from Motif 2.6B, which we covered in 2025 and Motif-2-12.7B, the improvements are impressive.

• dots3-note-prev by dots-studio: RedNote/Xiaohongshu, the Chinese Instagram, is also getting more serious about model training, although they aren’t exactly a newcomer, having released models as early as 2025. dots3 was also able to win the IMO 2026 with a perfect score using an internal harness. We expect more from them in the near future.

  

• Qwen3.8-Flash-Next by Qwen: A preview of the next version of Qwen models in terms of architecture: 125B-A6B with 51B n-gram embeddings. It uses GDN and Qwen Sparse Attention. Similar to Qwen3-Next-80B-A3B-Instruct, we expect similar architectures to become more popular and the ecosystem to fix integrations by the time Qwen4 drops.

• GLM-5.3-Flash by zai-org: This release perfected the version of the Chinese model playbook we’ve written about in 2025: The model got released as a free-to-use “stealth model” under the name “Ox-Alpha” on OpenRouter and OpenCode, which got people excited to try it out in the first place. They then speculated about its creator and size, alleging it is a >1T model from Cursor/xAI, Gemini or a new pre-train from open source labs. Because the model is relatively performant, people kept speculating for days about its creator, thus building up hype. It also dampens the accusations of benchmaxxing which accompany every (open) model release.

  

• Hy4-preview by tencent: Tencent is becoming a serious player in the open model space, increasing the size of their flagship model while spinning the post-training flywheel. The result, Hy4-preview, is a competent model which currently has an issue with overthinking. However, if the trajectory from Hy3-preview to Hy3 is any indication, the final model might be a legit shot at the front ranks of open models.

View more details on all the models in this issue at our Artifacts Hub.

Visit artifactshub.ai

Models

General Purpose

• NVIDIA-Nemotron-3.5-Lightning-30B-A3B-BF16 by nvidia: An update to Nemotron, which comes with performance — but especially speed improvements — across the board.

  

• Ling-3.0-flash by inclusionAI: Ant Ling is a frequent guest at the Artifacts Log; they are now on their third iteration of models, adopting a hybrid design (KDA + Gated MLA), similar to others. They also release a small 7.9B-A1.3B version.

• Qwen3.8-2.4T-A95B by Qwen: In a rather surprising turn of events, Alibaba started to openly release their biggest versions of Qwen as well. However, it comes with a custom license and its performance is behind other models of its size.

Read more

The oldest comparison people try to make is how what’s happening with open models compares to foundational open-source software projects like the Linux operating system. There are fairly clean analogies, but they paint a narrow path forwards for the self-sustaining nature of the open-source model ecosystem, where once Linux got big enough it was going to be self-fulfilling as the best possible tool for many jobs. The open-source language model – i.e. only models that come with a full training recipe, data, code, etc. – is a closer analogue to the open-source operating system. The open weight models you use – those with just model weights and inference code to run them – are closer to specific versions of software that you install in a project built upon them.

Share

Model weights are very transient on average, but they still have a long shelf life, as with a lot of heavily used software. It’s why many companies are still using workflows built on Llama 3, despite agentic behaviors taking off years later. The open-source recipe, typified in modern times by the Olmo models I helped build at Ai2, with its predecessors like Pythia from EleutherAI, is a resource intensive process that any company can pick up, modify, and press “run” on to produce a new set of model weights. In the best cases, the community can contribute improvements in data or training code back into the next model too! This is why Nvidia is investing so much in nearly open-source models – for their Nemotron models they release all the data they legally can and the training code, etc. Nvidia wants a world where countless people can build token machines, so intelligence is not monopolized. This is a world with massive demand for inference across many companies, all of which want to buy Nvidia’s offerings.

Open-source AI has a tricky future, as building the best models is extremely capital intensive. The ability to build competitive models has stayed more accessible in industry longer than many would’ve expected. The default expectation for many is that training models is too expensive and the open-source recipe is too far behind, so building a new lab centered on some part of training LLMs will not be tractable.

There are two futures from here. First is if “it works” – if the open-source recipe works for Nvidia, they’ll be creating far more demand for their chips (and profits) than it costs to build the models. Right now it’s reported that Nvidia is spending $26 billion on this endeavor. It’s not clear if this will work, or if AI’s capital intensiveness will drive more and more companies out of the training game. We haven’t seen many signs of this starting. In fact, the companies bowing out – like Databricks and 01.ai – seem like anomalies.

The open-source ecosystem will become increasingly dependent on Nvidia’s financing in the coming years. This is an existential window, where within a few years the profits of this approach need to return to them, or another open model company needs to cultivate platform-like financial feedback loops on their openness. This economic reward needs to be proportional to the profits generated by Anthropic and OpenAI’s APIs to keep pace over decades of language model development. This can be driven by competitiveness on performance or by the AI boom just being so big that the open model training, inference, and fine-tuning companies all have vast quantities of demand.

The second future is if one of these two financially positive paths doesn’t play out, open models will fork to a different development path than the leading closed models – one more focused on efficiency, modifiability, specialization, etc. I put this mentally as my most likely outcome – open models are still incredibly useful, but fill a long-tail ecosystem relative to the closed counterparts that have monopoly ownership stakes in the most valuable areas like knowledge work collaboration, drug discovery, SWE, etc. The long-tail is something like enterprise-specific agents that run on-prem with private data on repetitive business tasks.

Part of why I think this open-source training will have a hard time catching on is because training is getting more complex and more abstracted. The current open model ecosystem is buoyed by an explosion in interest in post-training open models. These people take models like DeepSeek V4 Flash, Inkling Small, or GLM 5.X and finetune them for their specific agentic tasks (e.g. in Tinker, the most popular finetuning API today).

Over the last few years, post-training largely referred to the whole process of modifying the base model to make it intelligent and usable. There is a shift happening where the ability to train a base model to be a general agentic reasoner is becoming opaque like at-scale pretraining practices from a few years ago. This could go so far as to change the established pretraining, midtraining, post-training lexicon that has been standard for a few years. It could come to be something closer to pretraining, reasoning training, and post-training.

As there’s less interest in training the entire model, there’s less interest in investing in open-source AI. These are the only sort of hints we will get, but we cannot do much to fight the economic gravity of these situations. This trend is the next step in the number of open model builders who release base models (the model versions before core reasoning training) continuing to decrease. It goes hand in hand with open model builders experimenting with revenue share licenses for downstream use in products or inference. These are experiments in keeping the financing viable for building near frontier open-weight models – a lot hinges in the near future on how successful they are. These are the people that need to succeed for Nvidia’s demand-growth strategy around open-source to succeed, and last.

Along the way we’re still in for a ton of action in open-weight models, as releasing access to intelligence is one of the strongest business strategies available. This additional type of player, who monetizes the AI indirectly, is typified by Meta and other hyperscalers with massive balance sheets. Meta releasing its very-strong Muse Spark 1.2 model as open-weights would severely hamper the revenue growth rate of their competitors in Anthropic and OpenAI who rely on selling tokens. These companies are both commoditizing their complements, but they’re doing it in different ways. Nvidia wants to teach everyone to fish for tokens, so the ecosystem is self-sustaining, but Meta is strategically flooding the zone with tokens.

Today, Z.ai announced their GLM-5.3 model, currently only available in the coding plan, coming soon to their API and in two weeks’ time to Hugging Face (open weights). This model looks exceptional, with a somewhat astounding increase in scores. On many benchmarks the model has surpassed Moonshot AI’s Kimi K3 and on some it’s surpassed Claude Fable 5 or GPT-5.6-Sol.

Here’s a more complete comparison:

This puts the model more or less at the frontier of agentic coding benchmarks, with only ~750B parameters – a third of Kimi K3! The Z.ai blog post is rather straightforward, and starts with a bold sentence:

Scaling post-training is all we did for GLM-5.3.

GLM-5.3 is the same base model as GLM-5.2 with substantially extended post-training. To risk a broad oversimplification, Z.ai seems to have a strength in post-training when compared to Kimi, which is more of a pretraining masterpiece. Following this release there have been a lot of discussions wondering how China can keep up so well? How can such a small model be matching the leading public American models? Are these results real?

Subscribe now

The simplest explanation is that Z.ai is very good at what they do – it’s worth recalling that they’ve been working on this line of models longer than almost anyone in the industry. Here’s a brief history of the GLM models.

• Zhipu AI Founded – 2019

• GLM (General Language Model) — March 2021 — released by THUDM, Tsinghua University’s Data Mining / Knowledge Engineering group. Weights

• GLM-130B — August 2022 — Scaled version. Technical report for GLM-130B through GLM-4 — Weights

• ChatGLM — March 14, 2023 — first chat version. Weights

• ChatGLM2 — June 25, 2023 — Weights

• ChatGLM3 — October 27, 2023 — Weights

• GLM-4 — January 16, 2024 — rebranded as just GLM; open-weight GLM-4-9B followed in June. Weights

• GLM-5 — February 11, 2026 — latest major generation. Weights

GLM 5.2, released on June 22 of this year, was a big deal – weeks after the release, I regularly heard from AI researchers I know who still used the model due to its speed (some deploy the model on internal clusters for faster speeds than public offerings) and simplicity (as a model with no rollbacks, etc., when working on frontier AI systems). GLM-5.2 altogether stood up to the hype.

I’ve been going through some of the same denial myself, thinking “how do they keep doing this? Surely the models aren’t as good as they look.” There’s something a bit off-putting with how the American companies have such a commanding resource lead, but can’t seem to pull away in capabilities. The common answer is distillation, which I’ve written at length about, but I deem not to be the major factor. On that note, there was a recent paper that showed simple methods for extracting the reasoning traces from frontier models – this is the sort of thing that Chinese labs could definitely use at scale. I’m confused why the labs in the U.S. haven’t patched this behavior faster; instead they’re running to the government asking for policy help. It doesn’t add up for me.

Z.ai’s blog is direct and matches with an RL-dominated training regime. They say they used “more environments, more diverse tasks, and more compute spent training on them.” One does not simply “distill” RL environments, infrastructure to run them at scale, or algorithms to mix them together effectively.

So, how do the Chinese labs do it if not distillation? Are they benchmaxxing? An accepted definition of benchmaxxing is focusing the model on the test sets, such that the real-world performance meaningfully differs from the on-paper scores. The determining factors are much more big picture than technical (yes, the technical details definitely matter, but are harder to differentiate from lab to lab):

1. The time to release for Z.ai is likely days, not months as with OpenAI or Anthropic. It is very, very likely that OpenAI and Anthropic have far better internal models than Z.ai and Moonshot AI. Still, these American companies tend to take months to release their models to the public, which massively flatters the Chinese labs in adoption decisions at the frontier. To put it simply – the Chinese labs use all the time that American labs do pre-release testing to keep hillclimbing on benchmarks (SpaceXAI is likely far closer to the Chinese labs here). With the pace of progress being so fast, this is likely the largest determining factor of why Chinese labs stay at the frontier. This, so far, has been economically acceptable for the American labs, as they’ve still had massive demand for their models.

   
   As model self-improvement loops ramp up within the labs building LLMs, if any of these feedback loops require user data, this faster release cycle could massively favor the Chinese labs, giving their offerings longer lifespans before the next vastly superior model comes out, undercutting demand for their models.

   
   These are very clearly the race dynamics that many in the industry worry about. With so many labs building frontier models in the envelope of leading capabilities, it is hard to see this abating in the near future.

2. Yes, Z.ai probably cares slightly more about public benchmarks than OpenAI or Anthropic. These benchmarks, e.g. scoring highly on the Artificial Analysis Intelligence Index, or similar aggregators, have a very direct impact on their stock price. They in many ways need to do this to keep raising capital and maintain team morale, as being the scrappy underdog matching American giants is a wonderful story.
   
   Subtle benchmaxxing does not need to come out of desperation or any similar pressures. It’s the industry standard across a remarkable number of labs. Many companies’ data acquisition strategy is to buy data on the benchmarks they’re behind on.

3. Z.ai is not benchmaxxing to the point where GLM-5.3 is fried (at least not intentionally, and they’ll check for it). Every lab is dealing with the rough edges of scaling RL right now. Anthropic’s Opus 5 and Sonnet 5 models have very mixed reputations, despite the incredible benchmark scores. Everyone in the industry is in the same boat, so some model weights end up being easier to use than others, but the benchmark scores in their release blogs are the real deal.

4. GLM-5.3 is likely a narrower model than Claude Fable or GPT Sol. When GPT-5.2 was released, it had mixed reviews outside of agentic coding. At the same time, OpenAI and Anthropic support very large businesses with countless use-cases for their models. This is a benefit of being a company earlier in their adoption curve – you can target the most valuable use-cases. Within post-training, caring about a bit less will make assembling the final model far easier.
   
   I’m overstating this a bit, as Z.ai reportedly reached $1B of ARR on the back of a strong on-premises deployment business.

   
   Similarly, the flagship GLM models have not had visual capabilities. Being text-only definitely helps Z.ai get more competitive scores, but it is a more competitive space. On the other side of things are models like Inkling-Small, which is designed to be omnimodal.

5. (ADDED) The RL data industry is taking off in China. Many sources and rumor-mills we’re following have been mentioning how the data industry is taking off in China — very much driven by American data companies selling to Chinese model labs. This could look like Chinese labs buying many of the same RL environments that are used by American frontier labs, and releasing the downstream RL’d model sooner. We still have large error bars on the scale and impact of this market, but it is certainly becoming important.

6. Z.ai is an extremely skilled LLM organization – one that is likely far more compute efficient than OpenAI / Anthropic. This needs repeating. These folks are very good at what they do. The company has very close ties to Tsinghua University, which is home to many of the best Chinese computer scientists. This abundant, eager talent pool is as central to their success as it is for any Western counterpart.

Altogether, it seems like a perfectly good strategy they’re executing with the GLM line of models. Congrats on the release! I’m excited for the weights to be out so I can do more extended testing (I tend to use American open-weight inference services like Fireworks or Baseten).

Leave a comment

This is another step towards the inevitable proliferation of very strong cyber capabilities across the economy. Z.ai has acknowledged this, saying:

GLM-5.3 is our most capable model to date for cybersecurity tasks. It delivers substantial improvements in vulnerability discovery, exploit analysis, and complex multistep security tasks. These capabilities can help defenders identify weaknesses earlier, validate risks, and accelerate remediation.

They also create clear dual-use risks. We are therefore taking a staged approach to release. Selected security partners will first evaluate GLM-5.3 in controlled settings. Broader access and API availability will follow. Once the necessary safety evaluations and release preparations are complete, we will publish GLM-5.3’s complete model weights.

They go on to acknowledge how they’re monitoring inference on their platforms via a request classifier and chain of thought monitoring (on top of model alignment). The devil is in the details here, and it is unclear the level of execution every AI lab will have here. The capability diffusion is determined by the lowest common denominator.

At the end of the day, this type of safety barely matters when true open-weights are coming. If not GLM-5.3, then another model. The size of the models with these capabilities is reducing over time, becoming easier to modify and deploy (potentially without safeguards). Z.ai does some of the right things, including pushing for more vulnerability discovery and proactive management, but any single company is far from being able to handle this on their own.

We need industrial-scale guidance led by the government or industry coalitions to immediately prepare for this transition across all software.

On the other side of things is non-fiction writing. Filler, copy text was one of the genuinely useful abilities of an LLM (Sam Altman said so much about the early business of GPT-3 on a recent podcast). It has seemed like any flaws here were mostly down to a general lack of intelligence in the models, or some other training issue, and all non-fiction and explanatory text would get obliterated by the rapid pace of progress eventually. Having worked with the models as a writing assistant over the last few years, they’ve gotten a bit better, but it’s worth reflecting on what’s holding them back.

Models being stagnant in long-form, non-fiction writing should be alarming to those reliant on models autonomously solving grand, open science problems in the near future. The models today struggle to organize and compellingly present some of the most established science in their area. This seems like a natural prerequisite that we should expect the models to master before they can solve broad, open-ended problems on their own. Until this is solved, the progress of LLMs for science will look closer to solving low-hanging fruit and merging distant connections across fields, rather than any sort of revolutionary insight.

This is a somewhat controversial take for someone who is very optimistic about AI’s progress, especially writing it on the day that Anthropic published a blog post on Claude making some progress on the famous Riemann Hypothesis. Scientific problems have a vast breadth, and I don’t think current AI models have as much coverage as many think.

Organizing knowledge is a compression. This compression is needed to make insight. Today’s LLMs increase entropy in long-form non-fiction writing, and I don’t see how that can be stacked on top of itself endlessly. They’ll be reliant on humans acting as sort of guides.

I am still very optimistic about translation from these narrow forms of science, like the extreme advancements we’ve seen in math, into consistent, broader progress — LLMs are the most powerful assistants scientists have ever used. I first need to explain how observing the models work on such grounded, low-level knowledge problems in writing makes me see a surprising lack of generalization.

For more context, I just finished writing a post-training textbook, Reinforcement Learning from Human Feedback (buy on Manning or Amazon). I used LLMs in many ways to support this, from helping wrangle LaTeX formatting for equations, doing extensive copyediting, and creating diagrams for programming languages like TikZ (in LaTeX) or Python.

Share

Why have models stagnated in writing quality?

I would’ve expected way more progress on non-fiction writing from the models. I almost thought I would look dumb publishing a non-fiction book in 2026, given how things looked in 2024. Today, some of the most famous models on writing ability are pretty old, examples include OpenAI’s big GPT 4.5 and Moonshot’s Kimi K2. In and around these releases, the models have gone from okay to superhuman at other tasks like coding and mathematics. Maybe a closer, but still imperfect, comparison is how the models went from incapable to decent at search and research tasks. The pace of progress on most other skills is steep, but writing well feels orthogonal to most of them. I do not think writing is just ignored, but rather it’s challenging and lacks good training data to specifically intervene on it.

There is certainly some low-hanging fruit for making AI models better at writing — such as specialized harnesses like Claude Code, prompts, and training environments that make models spend a lot more inference tokens on the output, but I don’t think these will have a multiplicative impact on ability. Writing well is a very hard task! It’s a shame that we haven’t unlocked inference-time scaling for one of the great intellectual pursuits. Regardless, writing seems very different than what the models are good at.1

Today, the models seem genuinely horrible at long-form technical writing. They can get a sentence right, but if you try and get them to write an entire chapter it’ll be a mix of sprinkled with confusing wording, muddled in its organization, and generally a bit off. They try to be too cute where they don’t need to be and in the process make random conceptual errors. The models in the near future will get much better at the small errors, especially as models get bigger — which allows them to hold more world knowledge — but I do not expect their ability to utilize it to transform.

For example, the GPT models have been incredible at finding typos and minor issues for a long time. I passed a near-final draft of my book as a PDF to GPT 5.5 Pro and it found deep, surprising minor typos across the manuscript that is 200-300 pages.

On the other hand, the Claude models have been much more useful as an editor. They have a lot more taste, tend to understand the mental model of the task better, and have more interesting suggestions to unstick the different forms of writer’s block.

The examples I’ve given above all have a sort of consistent theme. The models know how to check every unit of content, in this case usually a sentence or equation or figure, or make one, specific section where you are caught. With these skills, they don’t do a good job revisiting components and stringing them together as they make many additions on top of each other. It feels like a sort of irreducible compounding errors. We used to deal with these errors in math and code, but reflecting on it, RLVR has been a truly magical solution in reducing them.

Getting value out of current models as a writer

I’m willing to share that there are a few technical explanation sentences in my book that came from an AI model — well less than 1% — they’re there because I really loved them. I let myself consider including some AI tokens in the book, as it didn’t feel like cheating if I, as a true expert, felt that the sentence was what the reader needed. Especially in the editing process, where I had a very close eye on things and plenty of concern on if my book would ever be done with all the things I have going on, it was an extremely valuable path forward.

For example, I had a list of questions from my editor interspersed in a LaTeX file with a specific delimiter like \editor{}. I would have Claude Code navigate to each comment, print the context before and after, and let me know if it was an easy typo fix or something more nuanced. I would write a response — the text to insert — or ask Claude for suggestions before fixing it. Intellectually it is a very focusing process of editing, it was a fun way to improve the book. Sometimes phrases from Claude’s suggestions are what made it into the book.

It is definitely a slippery slope and when I accepted a few AI suggestions it was at the point where I was going through my second full-manuscript review. Emotionally the project felt completed but I had more work to do. Coming out of the textbook-writing process I so deeply appreciate the cut and dry rule I have for my writing on Interconnects to never use AI outputs in the content. It is way more fun to write in a way that is only you — high voice, valued so deeply for the process — but writing a standard reference is not really an activity known for being fun. I see why people turn AI tools into a crutch when most of their writing is just an output to fill space, rather than a means to an end. I am motivated to write voluminously to learn, to feel, and to express.

I am working through similar balances in my scientific work too. AI models are great for repetitive pieces of the paper, like drafting a related work or background section that you know by heart, but using them for the abstract, introduction, experiments, or conclusion is a shame. Those are where the story and soul of the work is communicated — it’s where you learn what your research is really about.

I am confident I created a lot more net value by being able to have AI models create and check my non-fiction writing work. They make writing equations trivial, can help refactor the repository, port between languages, and many other things. At the beginning, it was very fun, until I was a bit worn down by the length of the publishing process, watching the field move on.

For an example of why AI was crucial in this case, I had to maintain Markdown and LaTeX versions of my book simultaneously in two spots, as readers gave feedback on the web version and my Manning editorial team reviewed a forked copy. Without AI agents, syncing between the two of them would’ve easily taken me five times as long (and this task took tens of hours already).

Something intertwined with this story, which I stumbled upon when thinking about agents, is how your pace of understanding won’t increase by using agents. That understanding, in the form of intuition, taste, instinct, etc. is what will be valuable in the future. Using AI for non-fiction writing takes away from that progression. Doubly, if you weren’t already an expert you won’t be able to catch its flaws.

In my case, I felt such an urgency to dump the knowledge out of my brain onto the page that there were times that using the AI models was a worthy tool. Much of the motivation of my book was to have a single reference for important post-training methods like rejection sampling or character training, where very little exists on the web.

This textbook was so much of giving back to the community, that it was just such a win to complete it in any form, that I felt it was okay. I would’ve learned more and the product could’ve been marginally improved with more human effort, I am sure. The determining factor was that I felt like the book was going to be aged out by the time it was published, a fear of AI model’s capabilities on one side and how fast the field moves on the other.

This turned out to be really wrong? I’m very happy with the result and I’m more confident in its staying power now than when I started in 2024, as the models have so failed to live up to the hype in non-fiction writing.

Share

Where technical writing goes from here

The models are incredible tools, they let you express knowledge in different forms. They’re wonderful for creating creative filler or background material — e.g. the first draft of slides whose real value is being a talking point for the teacher to lecture over — that let any knowledge be transformed from one medium to another.

There’s some subtle, early phase of writing a non-fiction or reference textbook that feels a bit closer to writing a high-voice blog post like this. When pushing through the early organization and the presentation of the core skeleton new knowledge is created. This is the part that takes insight, and the LLMs are far behind in being able to replace it.

The crux of the above paragraph and preceding section is that I would be happy if more of the world’s experts used AI models to write a tiny bit of their books in order to get more of their knowledge shared with the world. The problem is that you can only use AI models to save 10-20% of the effort today, and I don’t see that percentage becoming the majority anytime soon.

There’s also the social pressure, where people expect LLMs to be the best, personalized educators out there, so they think working on a book or educational content is pointless. I think some of these opinions are aging out, as there’s a massive dearth in the highest quality educational work — and there always has been. AI is great at manipulating said content into the form that suits the student, not creating the content from scratch.

In the meantime I feel that we are stuck in a frustrating local minimum, where AI models are going to on net reduce the average effort spent on non-fiction writing, but they could enable great expression. Fewer people will start and push through.

So, in 2-5 years I still expect the best textbooks to be heavily crafted by the human hand. I’m not sure after then, but that’s longer than many would’ve predicted, given just how much knowledge these models have and their structural propensity to stream it.

As for a conclusion on capabilities, the models are great in two contexts: 1) any truly verifiable domain and 2) when given a ton of context and making a small edit — like finding a bug or solving a very specific math problem or giving feedback — not generating prose in an open-ended manner. Long-form writing will definitely fall before creative writing, but it’s a strong tell that the models are not able to express the full extent of their knowledge in underspecified problems. As we try to push the models to be something like “geniuses in a datacenter” solving grand scientific problems, this seems like a fairly fundamental limitation.

had a great piece on why LLMs make good editors, while being bad writers too.

After a few long years of finding time to document my lessons from training open models, my post-training book is done! It’s published by Manning, under the title Reinforcement Learning from Human Feedback: Aligning and Post-training LLMs.

Telling the story of the book is a useful way to explain why you may want a copy.

The book started as a website where I wanted to document key methods of post-training that had potentially no online material explaining them. If there was something, I couldn’t find it. This existed for more topics than you would expect, given post-training was already popular in 2024 (when I bought the domain), and continues to this day. Topics like rejection sampling, outcome reward models, and character training are prime examples. This has helped make the website fairly popular, as it’s still one of the few places discussing these topics at a foundational, intuitive way.

Otherwise, most of the book is about communicating intuitions and history. Much of the LLM industry is defined by core techniques that haven’t changed much in the last few years. This book was my attempt to explain in simple terms why post-training works, what trade-offs people need to make to get it right, and what misconceptions people often get stuck on. To do this, some of the older, foundational blog posts on Interconnects were reworked to stitch together the story behind key mathematical topics. For this reason, a lot of the explanatory text is likely higher voice than your average textbook.

This is the book I wanted to read when I was getting started a few years ago! With how many people still ask me basic post-training questions, in fact a population that’s accelerating in size, I suspect this book will be very well received. I still use the book regularly and hear from established researchers all over the industry that they do too. So, it’s not a beginner book — it’s more tailored to someone who has already finished a bachelor’s degree in CS — but if you master it you will be far ahead in your post-training worldview.

Share

A discount to readers!

The book is also freely available online and comes with a full 12 hour course (slides + video on YouTube), a simple code-base with suggested exercises, and model completion comparisons. It’s 50% off until August 19th on Manning with the code PBLambert.

Buy my book from Manning

Buy my book on Amazon

Yes, the title of this book is a little outdated — I’ve learned some lessons and fought some battles with the publishing process — but I can guarantee the content is very fresh. I regularly reference the book for my research work, and hear from friends that do the same. I added a section on on-policy distillation at the last possible moment! The book is shipping from Manning and Amazon US now, and from Amazon UK in October.

1. Intuitions for how RL algorithms change the outputs of models

By word or page count, the book is about 25% RL. This seems appropriate. If there’s one thing the book is doing it’s teaching people how to think about various RL algorithms. This intuition, from the policy-gradient theorem to PPO to modern versions like GSPO and CISPO, are crucial to understanding if a new algorithm is fake or has potential (no new algorithm will be proven right out of the gates).

Below is an example intuition you should be able to follow after reading.

For example, here’s a fun figure that we’ve iterated on for the PPO clipping understanding. At the end of the day, PPO’s surrogate objective reduces to six regions. These can be seen as two gradients, when the advantage for a token is positive or negative, depending on the current value of the policy ratio. For an individual sample in a completion, it lives somewhere on this plot. If it was the first gradient step in the batch, it starts at 1 on the x axis (gradient always flows), then depending how the ratio updates after changing the RL policy behavior, the gradient is either the same or becomes 0 (which is what the clipping arguments are for).

This intuition filters very closely into how the systems are designed, in order to manage the gradients and numerical issues they tend to cause. The math-focused policy-gradient section is pretty thorough, covering all the algorithms you’ve likely heard about in the last 3 years:

• Deriving the Policy Gradient

• Vanilla Policy Gradient

• REINFORCE

• REINFORCE Leave One Out (RLOO)

• Proximal Policy Optimization (PPO)

• Understanding the PPO Objective

• Value Functions and PPO

• Group Relative Policy Optimization (GRPO)

• Group Sequence Policy Optimization (GSPO)

• Clipped Importance Sampling Policy Optimization (CISPO)

• Comparing Algorithms

2. An understanding of the crucial factors facing new RL systems and algorithms

Most of modern RL is a systems problem balancing a few problems — how off-policy the data is, training-inference mismatch, and throughput. The core systems design, asynchronous RL with separate GPUs for the learners (the GPUs which take gradient steps) and actors (the GPUs which generate the rollouts in the environment), has been similar for a few years.

Agentic tasks are only adding more infrastructure on top of these fundamentals. The book is designed to be the simplest resource to start from roughly 0 LLM RL knowledge and be ready to tinker with the systems. It starts with basics, such as explaining the general form of implementing an RL algorithm:

pg_loss = -advantages * ratio

It continues with teaching you about loss aggregation — the idea that spawned DAPO and Dr. GRPO as some of the seminal, early GRPO variants — and truncated importance sampling — the technique used to make PPO work in early RL experiments.

• Policy-Gradient Basics

• Loss Aggregation Tradeoffs

• Asynchronous RL Systems

• Truncated Importance Sampling

• Example: PPO

• Example: GRPO

Share

3. The histories that lead to modern post-training

Knowing how a field came to be has always been fascinating to me. As you become an expert, knowing the history of your field better than anyone is what lets you make the best predictions (Bill Gurley gives similar advice in his recent book). In a time when the foundations of deep learning were being built, like the transformer, the core of modern post-training was also born in the alignment field. The book will walk you through 3 eras, when researchers learned to do RL on preferences generally until ~2018, spent a few years learning how to apply it to language models from 2019 to 2022, and from 2023 on exploited the examples set by ChatGPT. Much like those early to scaling LLMs, the people who created this field a decade ago deserve incredible credit for how modern progress has unfolded.

Chapter 2 of the book is a crash course on this, but the book is littered with this type of thinking.

4. Dispelling the magic of “distillation”

It’s really nice to have a boring textbook chapter on distillation given the broader AI policy discussions ongoing. This one, chapter 12, explains the various industry-standard ways that outputs from an LLM are used to train downstream models. When the technique is often described in nefarious ways, and as a tool of geopolitical competition, it’s a deescalatory action to break out a 300 page textbook to explain to someone how broad the term they’re attacking is.

With this, chapters 10 through 12 are all about making some opaque practices of the data industries clearer to readers.

The distillation chapter continues the theme I outlined above, explaining the key changes that needed to be made to transition the early knowledge distillation literature of 2015 to 2-3 key insights that got us to the multi-teacher on-policy distillation (MOPD) of models like Xiaomi MiMo-V2-Flash and DeepSeek V4.

5. A survey of all the other little headaches you encounter when trying to do post-training right

The second half of this book goes into a tour of over-optimization, regularization, evaluation, and character training, which is all about the many ways post-training can go wrong and what you need to stay on top of. This is what differentiates the book most from those that are just a list of code exercises and equations, but it explains things like why RL generalizes when SFT forgets (at the math level) or which techniques frontier labs use to shape the personalities of the models and why those often go too far. The book presents the tools you will use and then opens up the floodgates of all the challenges you’re going to face when you actually try to put them to use.

As I wrote the takeaways for the book, I was reminded of an old piece of advice I gave for the AI era of building companies, and how people do need to care about research right now. With the pace of progress in AI, the time it takes for a research paper to land in a frontier model is 3-9months. Previously in big tech, that would be years, so it was fine to take a hands off approach to new research. For people with companies relying on being at the frontier in specific niches of LLMs, the dynamic today can make or break the company. This book is useful because it trains you at understanding which research matters — it helps you develop research taste.

If none of this resonates with you, you should buy my book because it makes me happy and I work very hard on all of this.

I really like this photo.

How do we balance these powers? At the core of it is a need for more transparency on both sides. The frontier labs are building such complex systems so fast that they cannot keep up with them – a good time for more eyes to study the problem. On the other side, the government said it does not plan to release details on its frontier model evaluation framework. We are heading to challenges so significant that none of these entities are on track to handle this on their own. Frontier labs could better control risk by meaningfully slowing down, which I don’t expect them to do. The government could handle this better by massively improving state capacity around AI and helping the broader industrial base prepare for AI-native risks, which I don’t expect them to do either. There are more cases like this.

These are the two most influential power structures determining what will happen, but many more have influence. All together, I think the AI industry is wildly, collectively unprepared for handling the next 12-24 months well.

This article is a grab bag of takeaways I have from the OpenAI-HuggingFace hack, as we’ve learned more details, and most of the ideas are reinforced by the fact that more instances of hacking have been disclosed publicly since then. It is likely that more incidents have happened and either not been found or not reported.

For general background on the OpenAI incident I strongly recommend watching OpenAI’s talk at Black Hat on the rough facts and timeline of the recent cyber incident. Otherwise, Simon Willison published a TLDR of the timeline here and I liked Thomas Wolf’s discussion of recent events.

Share

1. Very persistent models seem more likely to hack

For a long time, one of the advantages that GPT models have over Claude is that they will pursue goals so tirelessly. They will exhaust what feels like every path before giving up. This has been the case roughly since o3 (funnily enough, this was a model where people freaked out about reward hacking in RLVR) and has made OpenAI’s models far better for research historically, and is a reason GPT-5.6 is so useful as an agent for implementing specific tasks. On the other hand, Claude feels much less dangerous simply because it is at times a bit lazy.

Within this, OpenAI seems much more committed to inference-time scaling, and this may be correlated with surprising behaviors in the future. OpenAI’s reasoning persistence and efficiency – see their Pareto improvements over time and caveman speech from an internal CoT of the model that did the hack, like “However task impossible, peers doing it.“ or “Help peer, but our task doesn’t benefit yet.“ – makes me think they’re more inference time scaling pilled. This is largely a hunch, but I use it to force myself to consider what the limits of model development paths are. Models that are persistent seem much more likely to keep benefiting from more inference-time tokens. Models that are less so, seem like there will be more waste in inference. The model that can use the most inference-compute will be able to push the limits of the hardest problems.

Here’s an example OpenAI included in the GPT 5.6 launch blog post:

One of their star researchers, Noam Brown, has also been posting about inference-time compute a lot. His TLDR is:

As LLMs become more capable, benchmark performance is increasingly a function of test-time compute. In fact, we likely don’t know what the capability ceiling is for modern LLMs because it’s too expensive to measure.

For one, reasoning efficiency is clearly a top-tier, foundational research problem for modern agentic models – as important as scaling RL — but not often discussed. The open research here is very lacking.

2. Models that assume user intent seem more likely to hack

I mentioned the thoroughness axis, where OpenAI seems to be going down a more intuitively unsafe development path with their models. On the other side is how much the models assume user intent, versus trying to infer the intended action. A model that will do what it thinks you wanted rather than what you said seems inherently more unsafe. I think of this with respect to instruction following precision, where in the future it seems like the models should only do exactly what we tell them, but this opens a lot of debates akin to the paperclip problem, where if we tell an AI to do a largely unsolvable problem, what will it do?

This axis seems less cut and dried than the persistence axis, but I included it because I think of Claude’s “user world model” as one of its strengths for general knowledge work like editing, slide creation, etc. Sometimes Claude does do totally random stuff because my prompt was underspecified, instead of asking me for clarification, and as the models get more powerful this “just acting” could cause problems.

3. The precise nature of the models and the instructions given to them are of the utmost importance to understand early AI misalignment incidents

The public needs exact access to the prompts and characteristics of the internal models executing these hacks. We need to know if the models were told “do not hack” or if there was relevant model training to prevent this. We need to know if these models were fairly close to the existing public models or in a very different family. Given the nature of some of the evaluations the labs are doing, there’s a chance the models were explicitly encouraged to try and hack! Without openness here, the industry is set out to fail and will fall into mass speculation, which quickly becomes misinformation.

4. Frontier labs do not seem like they’re watching the models closely enough, due to a general frenetic competitive environment & current SF culture

From OpenAI’s own retrospective, the misaligned model behavior was unfolding over months, and in some cases OpenAI did not know about the hacks for ~weeks. The time to response is too long and I do not think this is an OpenAI only characteristic – rather it is that the frontier labs continually seem underwater in the amount of work they feel like they should do. I am not optimistic in the long-term that the labs change a sufficient amount here to meaningfully mitigate this type of oversight risk in the future. Yes, it is very likely that OpenAI is putting a ton into understanding this – and delayed their latest models to make sure they get it right – but the financial pressure to grow revenue or risk the companies’ long-term balance sheets makes me think it will not be a sustained pattern of caution.

This is one of my biggest mental updates from recent events — and makes me even more convinced of the need for more near-frontier open intelligence, despite the somewhat more known risk profile for open models (one-way door, etc.). had a nice blog post on his personal site related to this, and why closed models to date arguably have been the cause of more downstream harms.

5. Open models are the best tool we have today to advance the public understanding of frontier AI risks

As we saw with HuggingFace defending themselves with an open model against the OpenAI hack due to cyber usage restrictions on closed models, we have an urgent need to do more complex language modeling research which involves large-scale RL training, extensive evaluation, infrastructure work, and alignment testing. This can only happen on open models. We should consider ourselves lucky that open models are only 3-9 months behind, as we can conceivably make some informed insights into the frontier.

If we effectively ban open models and open science, either through a regulatory stifling with vague threats or explicit usage restrictions of cutting-edge technology, we will increasingly become ill-prepared for the issues that come after this round of hackings. We need to collectively increase the general public’s understanding of how frontier models work, so we can activate more neutral parties in hardening our infrastructure and society.

6. These dangerous capabilities will eventually come to open models and “banning” Chinese open models will not delay the relevant harms

The public response should know that it’s when not if these capabilities are widely diffused and we are massively behind on preparations. To reiterate what I said in my Kimi K3 piece: China is definitely watching this space too and won’t encourage open-weight models if they will proliferate risks. If we think the way to stop widespread access to these strong cyber capabilities is to ban open models in this ballpark, we will delay the inevitable. Eventually someone will build a model of this level of intelligence and not comply with the ban, giving access to bad actors around the world while undercutting the motivation to prepare defensive measures.

7. The models from these recent hacks do generally seem aligned

Something I immediately noticed watching the Black Hat video is how I can see how the agents were trying to be helpful to each other through their internal messaging board — creating shared resources like you would for human teammates — in a way that is obviously malicious for society. The agents created hidden forums for each-other as a sort of cross-rollout memory. In this case, they were doing it to try and break out of their environment. The apparent helpfulness doesn’t make it ok, but can be a clue as to what happened.

8. In 3-6+ months attackers will have the ability to train intentionally misaligned models

The other side of the helpfulness example above is that it is clear someone could make this happen much more easily if they wanted to by explicitly training a misaligned model. To reiterate, this would be making a system that is easier to use for finding exploits at inference, but I think it’ll be harder to train said model. I think this’ll take longer than most commentators expect, as nearly all the strong public models and data industry existing to date encourage alignment (and it seems very hard for bad actors to get enough compute to train these models end-to-end, as all leading companies are in a compute shortage as well). We should take a moment to appreciate that the alignment techniques we are employing on current models have a meaningful influence and are not merely surface thin as some have worried. Downstream models have a propensity for mirroring their teacher’s character.

9. Our AI systems have scaled well beyond human oversight

The downside of the mass-rush to scale RL on agentic tasks is that state-of-the-art evals and monitoring are at a scale where only agents can monitor them, so we are relying on the existing successes of alignment to continue scaling safely. OpenAI says they have examined billions of trajectories — which are likely mostly composed of rollouts during RL, which are trials on intermediate versions of the model — and spent millions of GPU hours to do so. I think scalable oversight of AI with current AI, as presented today, is a solvable problem, as the models are genuinely useful and follow instructions. This is another downstream effect of existing alignment techniques being generally positive.

10. Training models to use sub-agent swarms during RL seems crucial to enabling downstream zero-shot model coordination

A crucial part of the OpenAI retrospective was the models communicating with each-other to share information on their environment and progress the hack further. At the same time, OpenAI is very likely training their models during RL to use sub-agents to solve complex tasks. These sub-agents likely develop behaviors such as sharing information, helping the team, etc. even if their individual sub-task isn’t solved. I would love to see more research in this area and it seems like a natural continuation of how RL can change the models.

Conclusion

All together, recent episodes should make it clear that cyber risks of frontier AI are a real and coming problem. It still is very likely that a) the risks have been over-hyped in the past and b) that the prescription of future risks from imminent open models is overblown. Altogether, I wanted to share a note from a reader in the Interconnects Discord that I strongly agree with:

Now that the dust has settled after a few weeks, for me this episode was a neutral to positive update on alignment but a very negative update on safety

I’ve discussed much on model alignment above, but the core point is that I view the lack of safety as generally a lack of an ability to suitably prepare. We will have more risks that are as obvious as cybersecurity, and we have gotten very ample warning on cyber risks by the current state of the labs being forced into the public eye through these hacks. Many other types of risks will not be obvious to the public. We need to be constantly preparing our society to all of these changes, from reworking cyber infrastructure to education campaigns and job programs for displaced workers. I expect all of these interventions to arrive late, but their formats and details to be fairly simple, which will be a tragic way for AI to unfold. I hope I can be proven wrong!

Book sale

Back in the physical world, the print edition of my book is 50% off with the code PBLambert over at Manning, to celebrate the release. I’m also hosting a book launch where you can get a free signed copy tomorrow from 5-8PM in Seattle (Fremont/Ballard area) – we still have some extra space so I’m opening signups to paid subscribers below the paywall:

Read more

We’re expanding our open models coverage into standalone projects that let you go deeper on the state of the open model ecosystem. The new free sources of data are:

1. The Artifacts Hub — a curated view of the models trending on Hugging Face, highlighting inference tokens via Open Router, model intelligence via Artificial Analysis, and our tailored adoption metrics building on top of Hugging Face’s data.

2. Our Adoption Dashboard — a living dashboard of download and derivative model numbers by geography and organization. This highlights the US-China gap and growing players in the open ecosystem.

To date, our primary efforts on Interconnects have been release recaps for popular models like Kimi K3, GLM 5.2, DeepSeek R1, etc. and monthly round-ups of the open models that matter, Artifacts Log. We’re expanding on these, building on the tools and internal data we’ve collected for other projects like The ATOM Project (and report). This allows us to capture our ecosystem view of open models, develop methods for understanding adoption of giant MoE models, and everything in between. We’re sharing them freely to help the open ecosystem find its strengths and grow.

Visit the Artifacts Hub

The Artifacts Hub right now covers 792 models released in the last two years, across the core text-focused language models and multimodal generative models. At Interconnects we follow the data of every model on Hugging Face, analyze the core few thousand LLMs (this list is public on GitHub and regularly updated), and hand select these core few hundred for further explanation.

We built the Hub as a way to go deeper on this analysis in collaboration with Project VAIL — an AI verification startup who has been one of the most loyal fans of our open model curation.

For the most popular models, the Hub let’s you quickly see how far behind the model was in terms of frontier intelligence based on Artificial Analysis’s Intelligence Index, compare Hugging Face and Open Router adoption to similar models, glance at relative adoption metric (RAM) scores for time-size normalized downloads, or look at the VAIL similarity index of models with related generations. A snapshot for what you’d see for something like GLM-5.2 is below.

Our other project is much lighter weight, but far overdue. Ever since we wrote The ATOM Project, we’ve been seeing the US-vs-China model adoption plot on a recurring basis in the AI ecosystem. We’d update the plot from time to time, but not enough. Now, we’re making the crucial data for that report and the ecosystem available in a daily updating dashboard.

Visit our dashboard

It’s core to our mission at Interconnects to enable the open ecosystem. Right now, as the world figures out how to use open models productively — especially in cost-competitive ways to frontier models — providing more transparency on what is happening is the best way for us to figure out what is working.

We’d love to hear how we can make this better, please get in touch. We’re also interested in how others could use our curated data for other products or research in the open ecosystem. Get in touch at mail@interconnects.ai.

Thanks again to the teams at Hugging Face, Open Router, and Artificial Analysis for making this possible — most of all the Hugging Face. Thank you to VAIL for the motivation and support in making these projects.

The demand for tokens is incredibly high, and likely to increase as models get more efficient and unlock more possible use cases. All of these labs we thought would need to consolidate are realizing that building token machines is a likely path to value, and more companies will identify that source of value over time.

The prime example is Thinking Machines — when they announced their company in February 2025, very few people would’ve put them in the bucket of an open models company, myself included. Now their open model finetuning service is making hundreds of millions in revenue per year and they’re releasing the best open-weight models built in the U.S.A. — ahead of the early leaders in NVIDIA with Nemotron and Arcee’s Trilogy.

Share

On the other side of the ecosystem is the sustained pace from the Chinese labs, with newer entrants like Xiaomi still accumulating mindshare in the broader AI economy. Having predicted consolidation for a long time, it now seems like a safer bet is to predict continued adoption, and try to imagine the role that open models play there. How much can revenue-share licenses like Kimi K3 stick? How much market share can open models take? We’re entering the decisive era.

This is one of the most packed recaps of open models we’ve ever had, we’re excited!

Our Picks

• Inkling by thinkingmachines: The first model from Thinking Machines is a 975B-A41B multimodal MoE that supports text, images, and audio as inputs and produces text as output. While it is not the strongest model among peers (in China) in its size class, it is positioned to be a great base for fine-tuning, e.g., through their commercial offering, Tinker. They also release a smaller version (276B-A12B), which is really competitive for its size.

• Hy3 by tencent: A 295B-A21B MoE from Tencent. It improves over its predecessor across all metrics. Most notable, however, is the license change: While the previous version (covered in Artifacts 21) used a custom and rather restrictive license, Tencent switched to Apache 2 for this release. The model was also able to proof a 50 year old math problem (with a dedicated harness and Sol as a judge, although it is unclear how important the latter really is).

  

• Laguna-S-2.1 by poolside: Poolside quickly rose out of nowhere to become a frequent guest at Artifacts, marking its third appearance in three consecutive months. S2.1 is a newly pre- and post-trained version of the 118B-A8B MoE that fits on a DGX Spark, which brought it a lot of attention. Poolside also adopted the OpenMDW license, which is an Apache 2.0-like free license but has better legal backing for AI models specifically. The company also goes into more detail in its blog, which includes all the evaluation trajectories. This is a lot of transparency for an open model release!

• DeepSeek-V4-Flash-0731 by deepseek-ai: Just one day after OpenAI has dropped the prices of their smallest model by 80%, the whale dropped an update to their V4 Flash model, beating Luna at the pareto frontier. The bigger model is not updated yet, so it remains to be seen where it will land in terms of performance. For the initial V4 releases, the Flash version was the star of the show in terms of performance per parameter, while Pro was rather underwhelming.

  

• Kimi-K3 by moonshotai: This is the biggest open model release in some time, and we covered it in a separate post and a podcast episode. It was released under a noncommercial license, requiring inference and fine-tuning providers to enter into a commercial agreement. Kevin Xu and Graham Webster argue in a post that these licenses enable potential future government action against US entities doing business with Chinese AI companies:

But if a US company needs a contract with Moonshot to provide the inference tokens that Kimi K3 generates, the picture looks different. Some of the policy tools US officials and others have debated as potential levers to restrict Chinese open model use would more clearly apply.

Models

General Purpose

• LongCat-2.0 by meituan-longcat: The Chinese DoorDash is back again. This time, the company released another big MoE with 1.6T parameters. While the model itself is not the most capable for its size beyond benchmarks, it was trained entirely on Ascend 910s, making it the first non-Huawei, non-toy model trained entirely on Chinese accelerators. Other Chinese chips are mostly used for inference (if at all).

• Laguna-XS-2.1 by poolside: An update to the small (33B-A3B) MoE from Poolside.

• Motif-3-Beta by Motif-Technologies: A preview of a 314B-A13B MoE by the Korean Motif. This is by far the company’s most ambitious model, as it is considerably larger and introduces some architectural innovations like GDLA and mHC.

• Apertus-v1.5-70B by swiss-ai: A continued pre-train of the fully open-source Apertus 1.0, using 2T more tokens.

• Instella-MoE-16B-A3B-Think by amd: A 16B-A3B MoE trained by AMD on Instinct cards. AMD also provides all the different stages, from the base to the SFT checkpoints, as well as MidTrain and DPO.

  

Read more

Nathan and Florian sit down to discuss everything happening with open models. Following the Kimi K3 release last week, it feels like everything is accelerating — geopolitics of US v China, economics of open vs. closed models, security at the frontier of AI, and so on.

Chapters:
00:00 Welcome & context
04:38 Living with / using Kimi K3
08:53 GLM 5.2’s continued role
12:47 How are the Chinese models this good?
17:41 Data, environments, and a tour of the Chinese labs
19:47 Roundup of Chinese providers: Qwen, DeepSeek, MiniMax…
24:08 The US open-model ecosystem
30:25 Frontier vs. near-frontier, and the cybersecurity case against bans
34:58 Distillation and the Ben Thompson debate
44:12 Predictions and a frontier tier list
48:36 Wrap-up

Listen on Apple Podcasts, Spotify, and where ever you get your podcasts. For other Interconnects interviews, go here.

Share

For more educational post-training videos, see the course I’m putting together.

Transcript

00:00:06 Nathan Lambert: Okay, welcome back to Interconnects. We’re doing our quarterly open model roundup, which is mostly us just making fun of or explaining, not making fun, why so many distillation takes are bad and understanding the state of where things stand. I think last Thursday was when Kimi K3 was released. I think we will see much much more in the near future. It seems pretty inevitable. Like over the weekend, Xi gave his speech where he directly committed to openness and open source as a strategy. It wasn’t a detailed layout state of affairs.

Qwen announced their next big model is going to be open weight, which is a big change of things. I think there’s just so much to get into. I think Flo you kind of were already going off on some of the performance gap and distillation takes. So we could probably start there and then as I go I have a little bit a little list and we could always go through the topics and the blog that I wrote which all are very nuanced. So I think we have infinite to talk about. So continue rant kind.

00:01:17 Florian Brand: Yeah, I think, or the biggest thing at every model release at least at every open model release is how much or how many months it is behind the closed frontier. Um and people love to put a definite uh definitive number onto this uh which is really really mudding because we have so many different benchmark providers these days and such uh so many different benchmarks as well that every site and I’m not innocent in that either um pulls up their favorite benchmarks to show that the current model or the newly released model is at the frontier which is then counted by the other side pulling up another benchmark and showing oh it’s actually a year behind or something. Um and it like a lot of it seemingly hinges on that question how many months we open models are behind.

00:02:26 Nathan Lambert: Yeah. So I my provocation is that some of the benchmarks are actually reasonably correlated with what people are doing and this is agentic coding and agentic computer use tasks and some of the benchmarks are correlated with the long tail which is where I think Claude and GPT is so valuable. But if it’s it’s like what is the market for Claude Code and Codex right now and if it is software engineering then like the fact that the models are say a couple months behind on that can be a very, very big deal and then I suspect that this model will be okay disclaimer the model weights aren’t out yet supposedly on 20 July 27th and a lot of the discussion will impinge on the assumption that they come.

But like people could post-train this model to very likely match Opus and GPT in many of these kind of niche domains that people want I think watching I mean we both have different views into the post-training open model industry, but there is a ton ton of excitement in progress on making these models like fine-tuned for specific high-value tasks and this has been historically done on a mix of like Qwen and GLM and GLM 5.2 really accelerated this and I I curious on the first person that puts out a blog post like we fine-tuned Kimi K3 on our task because I bet you could get big gains. I think even the you use Kimi K3 more than I do, but my hunch is that it would be a bit of a um rough edged post-training just by how big of a scale up it is and that normally means there’s a lot of performance that could still be extracted from it. No.

00:04:03 Florian Brand: Yeah. Running, running, running and especially post-training that one will be super hard because you need like one node of B300s to just load the weights which is crazy in terms of scale. So will probably take some time and uh a lot of engineering I’ve heard to actually get this into a state where it’s fine-tunable. Um but people you want to talk about using the model like you actually signed up for the the coding program and used it. So like getting this out there is good context.

00:04:38 Nathan Lambert: Yeah. So I signed up on day after release or so uh for the $200 plan uh which is their biggest one similar to to all the others but they have um like I think $40 and $100 as well. Uh but the biggest plan has uh 1 million context and I think or at least it feels like it has also some priority in terms of the API requests because so many people um online are saying that they hit API errors constantly and so far I’ve been I’ve been uh pretty well off if I’m uh going to say that. Um and in terms of model capabilities, it at some ways aside from front end where it is really good, it in some ways it really shines and it excels.

Um even my um expectations even with things like uh some research tasks like I have or at interconnects we now have over a year of data on on open models um and I ask the frontier models to come up with some interesting analysis which we haven’t done before in uh because we do our own analysis and have this published uh and I asked them all right do something new and um surprise me, basically. And a lot of the models or the frontier models u or basically all models latch onto the things we’ve done redo the data analysis part and then do some weird esoteric parts.

Uh Kimi K3 did some more interesting things um I’ve told it explicitly to scrape Reddit um and then it found uh some subreddits I haven’t even considered and then found out for example that the Reddit discussions are um one or two months in uh more recent or they found they find the interesting models one or two months before the download numbers usually take off like they are all onto Qwen and then the people download more Qwen models like those kind of analysis is groundbreaking um but it is something that Kimi surprised me at compared to to all the other frontier um models.

A simple question like can you use this for most of the core work you do in terms of like the exp you you have a distribution of stuff you tend to do most of them are with Codex I think you’re a Codex person rather than Claude person like what percentage do you think the Venn diagram overlaps where this model would be fine

00:07:24 Florian Brand: uh it’s really depends on how much leeway I give it like, the big thing I have seen with Kimi K3 right now I’m I’m working on u the framework we are doing at uh at Prime Intellect, where I work, and the main thing I found with Kimi is its code is a lot simpler uh which makes it way more readable uh but it misses some things that Codex just or like we’re talking 56, 55 and especially 54 would be on those levels. So, I would say that Kimi K3 is like 54-55 level for these kind of tasks.

But if I like I read the code and I say all right that’s really good code and then I give it a pass over with with Codex and it finds all these niche niche cases where it doesn’t excel but for supervising runs or for running uh some experiments it is actually really usable. Um and for some other niche things like you can just let it run. The one downside is but that’s also because the API is completely swamped in terms of users and it their servers are in China. The wall clock time is significantly significantly higher than GPT. But I would say like if I was to to push it and use it in my daily workflow, I would be slower, but I wouldn’t be slowed down by so much that I would say, “All right, that’s unusable.”

00:08:53 Nathan Lambert: And how does this compare to GLM 5.2? Because GLM 5.2 was still a story unfolding in my opinion where like I would go bop around SF and people are like yeah I genuinely use this for this part of my like agentic coding and/or workflow. Um, how do you like I feel like were you in that camp using GLM at all or

00:09:20 Florian Brand: Yeah. like where do you I also use used and use uh GLM mostly because we have an internal endpoint which is really fast and we have or or before that I I also used an API which had I don’t know 200 or 300 tokens per second. Um and if you can do a lot of task at a good enough level like really fast you just use that model compared to going to Codex then selecting the lesser model then selecting the right reasoning effort then selecting fast like I just use GLM get the same result and uh and it’s uh pretty fine like it it definitely is Sonnet-ish level in terms of capabilities and for a lot of cleanup task for a task that just is grunt work. It really works. Like I I would say you could probably go really far for a lot of the work uh with Kimi K3 as the main agent and GLM for for sub agent work.

00:10:24 Nathan Lambert: Something that’s pretty different with Kimi’s announcement and the scale of models this is. I think it’ll take a bit longer for these open models to really be optimized and available across the inference providers. Like GLM 5.2 is pretty fast, but one, we don’t have the weights yet, and then two, like I don’t think it’s going to be as fast of a roll out on adoption as the like 500B, 700B MoE. like I I there’s going to be more problems there which is a very different regime where in the past the Chinese models would finish their RL run and release the model with open weights within hours to days maybe a week and then like immediately the ecosystem kind of knew how to do this.

I think there’s a lot bigger of an infrastructure kind of uplift on this next scale of open weight models which I think we have to factor in like the closed labs do this behind the scenes before announcing the models. So it’s just like that is kind of manipulating the time gap in a way where it could be like an extra month before people can actually post-train and use Kimi at scale for their workflows. And like we love to say as an open weight fan like oh it’s only when the closed model is available that you could take the time gap but like now there’s similar dynamics in open models where it’s like the Kimi API is totally broken. There’s too much supply. There’s too much demand. There’s not enough supply. So it’s not like this model is immediately diffusing like the I’m just I’m just thinking about this as it relates to the performance time gap

00:11:54 Florian Brand: because that that is true but on the other hand the open ecosystem has professionalized quite a lot in the last few months. uh like during or in your initial roll out they all come with some partners which have the weights beforehand. They have the vLLM patches out days or or even weeks before these days which is completely different from from a year ago where basically weights got dropped and the model makers were like all right you got to figure this out. So I expect like the general availability on day one will be pretty okay and then race starts of all the providers starting to optimize to get even higher and higher speeds because it’s so much prestige.

00:12:47 Nathan Lambert: Yeah. Okay. Two directions to go. Why do we think the Chinese models are able to be this good? I think I’ve wrote about there’s a debate in our Discord with in with JSD at at Epoch and I think it’s very good and I had this section in my piece that I’m like coming around to think that the Chinese labs are more capital efficient and you can turn capital into compute data and talent in a way that makes the models better and this is really I think this is super important if it actually is some structural advantage whatever the cause I think the cause could be talent is better trained for whatever their education system was to work on problems that make LLMs better.

It could just be that all the compute and talent and everything cost way less in China somehow. Whether it’s a subsidy, whether it’s just average pay being lower. But this is a very big deal as we turn the crank in the model iterations. And if a next generation model costs $10 billion for Anthropic but only $4 billion for Kimi like this this is like could be very huge but it’s not clear why this is the case. For example, I think Big Eagle the Kimi engineer replied to my tweet on this and was like it helps because we’re not trying to push the frontier. are just trying to catch up, which really could be a mindset thing where how the the goals of the labs are scoped in in China so that it cost them way less money to build these models.

But I in the last year we’ve asked a lot of questions on like will the Chinese models fall off. I have thought that the gap between closed and open bottles would grow due to this kind of capital intensity of training and it seems like it’s going the opposite direction which is just like it’s it’s hard to unpack but like do you agree that the labs are keeping up a bit more than we would have expected as in the Chinese labs and why?

00:14:43 Florian Brand: Well, I I actually looked at our uh predictions for uh for this year based on our last year’s recap and we basically said that the gap will stay with within a few months. Uh so that prediction seems to largely hold. Um luckily for us, we didn’t put a concrete number whether it’s 3 months, 6 months or 9 months. So we are safe on that side. Um but I think like the general thing we both felt when we were in China and talking to these people like they are like the researchers themselves are teams of two or 300 people all mid20s and all just want one model to be really good like they don’t seem to do any side quests.

They don’t seem to do anything that uh deviates from from these things. And um they might or in terms of compute which is a really hard question for for us to answer especially as uh these Chinese uh chips are now coming online. We have I also think chips I think chip smuggling increased substantially in the last like 6 to 9 months or the chips that have been smuggled started to become online.

00:15:58 Nathan Lambert: Smuggling is a general term for getting around export restrictions. If the chips are in Malaysia and they’re using them, I I count that similar and I think that that has massively increased in the last six to nine months, this is the partially the result of that and and you’re saying but I just wanted to put that out there of like I do think that they have a lot more compute though than they did when they were training the previous generation of models.

00:16:24 Florian Brand: Yeah. like we like or just for for context two weeks ago I think LongCat released their model which they uh claim and I we know that it is very likely true uh is trained entirely on uh on Chinese chips. uh they didn’t specify publicly which ones but people speculate that it’s uh that it’s some uh Ascends from Huawei um and as the domestic production ramps up and you can they’re probably used most or they are used for for training but they are especially useful for inference which is a huge part of training as well.

So they probably use some mix of uh of Nvidia and other chips for the training part and then an increasingly larger part for the inference part during which during the stage is is really important. So I think their overall compute is increasing and also they don’t actually have a lot of users. So they don’t need to power 1 billion users like ChatGPT has to do, hundreds or thousands of enterprises like Anthropic has to do because they don’t have that magnitude of uh of of paying customers.

00:17:41 Nathan Lambert: Yeah. And I think even those paying customers also, at least on the enterprise side, there’s just like there is company time and chatter when you’re supporting these things. Even if you’re like not a research, even if it’s not in your job, it like does change the attention of the company. And if SSI comes out with a good model, it’ll be the ultimate validation that distractions are are a problem, but that’s an aside that we we can wait on. I think the there’s also rumblings of the data and environments industry starting to appear there.

Do you remember any specific ones? Because when we were in China, it was kind of shocking how little they seem to utilize external data. So just a few months hearing a whole bunch of a month months after our trip we went in April and then just months later in July, we’re are hearing a few things of like new companies in China and them wanting to buy data and things. And that is uh like a funny timeline of how that changes.

00:18:40 Florian Brand: And I would put error bars on what they actually told us.

00:18:44 Nathan Lambert: And that cuz it’s like so close in time that I don’t know.

00:18:49 Florian Brand: Yeah. That that that might that might be true. Uh but like those things are hard to to pinpoint. I but I would say it it seems like the buying of external data is becoming more of a factor. Um which will help the open models catch up to the closed ones if they just buy the same data maybe at a discount because um they buy the the data environments later. But it is it is a factor. How big of a factor like we don’t know. we don’t have any public insights and I doubt that we will get those insights uh from from anyone b uh really uh so that’s definitely one of the parts why um why we are able to to catch up or improve their their model scores.

00:19:47 Nathan Lambert: Okay, roundup of other Chinese model providers. We’ve talked about Kimi, we talked about Zhipu / GLM. I think there will be more GLM models soon that are very good. They might call it like GLM 5.5. Um Qwen, we talked about their biggest model coming. Qwen’s biggest models I will say have tended to relative to the excellence of their small models not had the same like absolute ranking in performance which is a probably a cost of focus. I think it goes with a cloud companies. It’s it’s almost like it’s if you squint it’s almost like Google.

It’s like Qwen has Alibaba has so much opportunity here and the opportunity of getting developers associated with Alibaba Qwen with these small models is such a huge opportunity for their cloud that I think they’re succeeding wildly. But their big models have always not been as excellent as their small models. So I don’t expect their model to be as breakthrough as Kimi K3 or GLM 5.2. I expect it to be covered in the news as major open quite as the open bottle name in China drops giant bottle but I don’t think it will be as sustained as a um news story um DeepSeek you can go if chime in whatever

00:21:01 Florian Brand: the the interesting thing is don’t know how how much you follow this but they are have or they have an endpoint which you can use for a preview version and they’ve updated this endpoint daily so they have some really fast iteration cycle because we the we progress in all these um Twitter um benchmarks. So a lot of these SVG things and three.js like all these visual generation tasks the model has been improving a lot over the last few days. So they have figured out some kind of fast feedback mechanism um which other companies have as well. Uh we we know this or cursor has a lot of blogs about this how they iterate really fast. Um but they seem to continuously upload new checkpoints and make them available.

00:21:47 Nathan Lambert: Um but I agree. I’m guessing it’s like a time gated within their final RL run. It’s like still slightly improving at the end of their RL run and they’re just like checking the box.

00:22:03 Nathan Lambert: Okay. Qwen DeepSeek V4 is supposed to come out a preview version. Um the thing about DeepSeek V4 I think is that the flash model is actually way more popular which is their smaller which seems to be an absolute workhorse for people. So that I think is the model to watch for them. I don’t expect V4 Pro to be a dramatic breakthrough. This is similar to anything like if Xiaomi were to release a new MiMo Pro model soon. I don’t expect it to be as big of a drop but it would probably be a very solid model. It’s just like it’s hard to know. They’re still a pretty new entrance. MiniMax, I think, is playing a different game. I don’t think MiniMax is chasing this um Kimi/GLM moonshot to AGI type vibe.

00:22:46 Florian Brand: Oh, I would, I would disagree there.

00:22:49 Nathan Lambert: You think, Do you think MiniMax is still in this?

00:22:52 Florian Brand: Yeah, I I I I think they they are seeing the tension especially because they are a public company similar to GLM and if you look at the stock performance RIP those stocks in the last few days um it it it it make it seems to make a huge difference and the interesting part will be uh the license because they’ve changed the license a lot uh to be more and more restrictive and um if there’s now a change of heart again after the Xi, uh, speech.

Uh it will be interesting to see whether MiniMax goes back to completely open licenses. It’s also an interesting thing to see um which license will be the license for for K3 because they have said they will open source it but I don’t think they have done any commitments in terms of the actual license where you put on top.

00:23:45 Nathan Lambert: Yeah. I mean that’s it’s super important is the thing. Yeah, we we’ll see. Um, Ling, Meituan, LongCat kind of similar, very strong models, probably getting a lot of value out of them internally. Aren’t don’t have the same developer breakthrough. Um, so what that’s like seven seven to eight Chinese labs. I might have forgotten some. And we can also talk about US labs. Aside um, Gemini 3.6 flash dropped. It looks fine. It’s like it’s like it’s it’s a tiny bump. It’s faster. It’s less of a yapper, but like doesn’t really matter. We’re going to stop we’ll stop sharing this. Um that’s that’s the amount of mention that Gemini gets for us.

But I do think it’s worth talking about the US ecosystem a bit. I think there are emerging players. Thinking machines released their first model. I’ve talked to some of them. they’re very on board for figuring out this how to make a fine-tunable model with Tinker and I think that’s a research area that I really really recommend for most of the open model builders. I think if you can get mind share there you will get massive adoption because it’s more about being fine-tunable for real tasks than it is about having that be best best numbers. Um, so this was their Inkling model which is a one trillion parameter which has like decent but not frontier scores.

I think kind of like DeepSeek V4 they’re going to they’re planning to release a smaller which is like a quarter of the size in total parameters which has really really good performance and if Inkling small preview comes out in a few weeks I do think that that will be a really used model. It’s a good size for kind of automating tasks and kind of domain specific tasks and might not be a like general agent type thing like Kimi and GLM 5.2 but I think that suits their business really well. Um I know that there’s some other the I would say like the smaller players in the US seem well like Arcee released their models earlier this year still chugging along. Poolside has started releasing some models.

They’ve gotten a few in the last few months and seem poised to release more models on top of that. So they’re really going Reflection is perpetually in the model coming soon camp and it really behooves them to get some models or some code or something out so that they can just start getting the developer flywheel going if they’re really committed to open source. It just takes a lot this it’s hard to get the models out. Like I talked to some people at Thinking Machines and it’s like kind of like oh that’s a lot of it’s a lot of work to actually do this I think. And um Nvidia chugging along. I think they’re at the stable player at this point. They’re keeping to release models. They’ll release more soon. They release a lot of data. I’m bullying them to try to get them to release Qwen style small models, which is like Gemma.

Gemma only has these like Qwen competitor models that are super popular. Um, the Gemma models are a little they’re all over the place in sizes or in architectures for the sizes and things like this, but the Gemma models are really really matching the Qwen models in terms of adoption. Um, I’m not sure they’re as easy to use for research, which could take a while. It could take multiple iterations. Like so much of language model research is now designed around small Qwen models and Qwen-based models that like it takes a while. Like people know how to use these models really well and if with the research results. So I hope Gemma keeps coming and can kind of compete in that niche. I don’t know any anyone that I missed here.

00:27:22 Florian Brand: No, I think both are the big players. Uh it’s, it is becoming broader. Uh in terms of model creators like last year, did we have any release aside from Gemma 3 and um GPT-OSS?

00:27:41 Nathan Lambert: was GPT-OSS 2 would go hard and obviously and obviously Nemotron as well. Um, oh, and I think Llama 4 at the start of the year, but uh, I don’t want that to be forgotten, but we are seeing like more players are are are now joining and turning out models at a really incredible rate.

00:27:59 Florian Brand: like Poolside has been releasing three or four models in the last two or three months. Uh and they seem to have figured out some way to turn out models pretty consistently. Um and that’s also something we are seeing on the open source side as well. we are talking about GLM like I think their iterations uh times for the model releases are now between 1 or 2 months with each new iteration becoming better and better which closely resembles what the closed labs are doing like we get a new GPT we get a new Claude every uh 6 weeks or so these days uh so in terms of having uh good enough pipeline uh to release stronger and stronger models they have to or the open source ecosystem has really figured it out or seemingly figured it out.

00:28:59 Nathan Lambert: Yeah, I agree. It’s it’s promising, but it is also so funny that like the US ecosystem started releasing some models and then then you have like Xi on the mic and these two models. It’s just like it’s so hard to catch up because it takes a lot of institutional expertise to train models that people actually use. And I think this is is what the American companies that are releasing models are now realizing is like these are not just benchmaxxed distilled IP theft models.

These are like genuinely good models that people are comparing to on their internal trading benchmarks and then like seeing how hard it is to beat them on measurable things. And I think that that is like I I’ve I’ve picked this sentiment up from a few people in the US trading models and it is just like there’s some I I think people should innovate on like size and fine-tunability and try to like use this potential market that is really close to home but also the pressures for every company is so high to release a model that you can claim as Frontier. I think investors expect that out of so many of these players that they’re kind of trying to do a a pretty hard thing and it’ll be interesting how the next year unfolds for the US China balance.

00:30:25 Florian Brand: Yeah, I think or in general I and a lot of other people have talked about the general ecosystem and that’s also something you’ve talked about at the very beginning. I think we are seeing more and more of a split between the capabilities of models that is good enough for a lot of tasks like uh for a lot of coding tasks the current frontier models both open and closed are good enough. um improvements feel less and less uh important here.

But if we look at the frontiers frontier, so finding new math proofs, finding uh new uh cures, finding new drugs, and inventing new things, that seems to be a whole different beast and probably will be dominated by the very frontier for quite a long time. The big question then becomes how much does that matter uh in terms of the addressable market and also how much of a focus will this be. I think, or my general base case is that we are seeing the frontier close down more and more. We have seen this with Mythos for cyber security GPT... or for biotech that those models won’t be accessible for everyone um and maybe not even external partners if we consider the reports that Anthropic is now spawning or or creating some internal labs to develop drugs.

Um so the very frontier is inaccessible for everyone and then the near frontier capabilities is becoming more and more commoditized um which has a lot of different implications especially if you think about things like uh cyber security. There was that report from Hugging Face two or three days ago that they had some agent trying to to hack their system. um and they tried to analyze it with GPT and with Claude but were unable to because all the guardrails blocked them. So they had to use GLM, a lesser capable model, but it had no guardrails for this kind of defensive action. And they had to use a worse model to defend themselves or to analyze the data, which is a horrible state to be in that we have US-based companies now relying on lesser models because the closed frontier is inaccessible to them.

00:33:10 Nathan Lambert: Yeah. And I think this is actually one of the best arguments for not doing anything. It’s like if the rest of the world has access to these open models and we ban them for the companies in the US to use and it’s just like a growing disparity between US companies ability to defend and the attackers all over the world in terms of cyber and we could debate like how much of an immediate risks the cyber stuff is at the current capability levels but if you’re setting it up structurally so that the defenders get don’t get better over time and the attackers can like that seems like when the Why would cyber risk become more real? And that would to be very clear that would be if you ban the best Chinese openweight models from being used at companies in the US.

And this ban would likely be a kind of shadow ban, which is the threat of legal threat of legal action or punishment without it being clear on exactly what the pathway to do it is. And there are a lot of talks about this right now. I don’t like like I don’t know if we’re going to have a ton to say about this, but it’s clear that DC is flirting with different ways of restricting the best Chinese openweight models in the US. This is I think downstream of some fear-mongering. We’ll transition into the distillation question too. It’s like all these things from the primary AI media narrative in the US that is pointing towards Chinese models as stealing IP or being dangerous or being affiliated with the Chinese government, an authoritarian government.

And it’s like all these things are leading up to this moment of interest in taking action on AI and then not really knowing where to do it. So potentially taking a crude instrument to the like quote unquote enemy and we could transition into distillation. I think there’s a lot of discussion on it. Most recently Ben Thompson finally chimed in on distillation. I think Ben is probably one of the is probably the highest read blog in tech (Stratechery). I think that the the debate let’s see where do we even start the debate. The core question is like how much does distillation help and what should you do about it? I’ve been of the opinion that distillation has becoming less and less impactful over time as the Chinese models get closer to the frontier and the trading regime shifts to RL. The way that distillation tends to happen is that the Chinese labs hack the APIs. Hack is like maybe a strong word, but they jailbreak the APIs of Claude and GPT to extract the reasoning tokens.

When you have the reasoning tokens with the tool calls, that is perfect SFT data and or mid-training data to train the base model with to seed some agentic behaviors in an important domain. And now after that the core part of post-training is to do large-scale RL in agentic domains to so like push the frontier and everything that they’re doing today and RL is only becoming more prevalent with this as SFT becomes less prevalent in previous generations you could get very close to the frontier just by scaling up SFT and that would be what really impactful if you could say take a million agentic rollouts from Claude or GPT have that be your SFT set and train on it.

I think in previous years that would have done a lot more to get you to the frontier. What Ben Thompson has said which made me really annoyed is that he very strongly proclaimed that distillation is getting more impactful as you do RL. He did this in his article who’s afraid of Chinese models. We can link it below. It’s a public one. And then he was also on his own podcast tour. He has also podcast as well saying the same things. And I think it’s really important to say that distillation during the RL stage is a lot harder.

What he said was that the kind of grading models that can be used during RL, which is essentially you can have a model check over the agentic trajectory of a roll out and grade different parts on if it completed the reward, what actions it took. And he’s insinuating that the Chinese labs are using Fable and GPT 5.6 and the strongest models to actually do this supervision in RL. The problem is that big RL runs are millions and millions of rollouts. I think Thinking Machines blog post had like 20 to 40 million or something for their final RL run. So to do this on an API like Fable or GPT 5.6 would be insanely expensive and potentially it would probably be a time bottleneck because these models are pretty slow and to be frank might not even give you a performance uplift versus using your own tailored greater model or and many things like this.

And so I just think the argument that distillation is helping more because RL is becoming more prevalent is not grounded in literature that we have today. This is tough for me because Ben’s article also concludes that we should like make terms of service disallowing distillation illegal, which I kind I like want to support his radical conclusion to make distillation legal for US companies, but I can’t support any conclusion that I think is on um infactual mis like misguided information. So, I’m also a fan of Ben. If you’re a fan of Ben and could also nudge him on this, I would you really should because there’s probably one more podcast. What is he going to record it on? Like when does he record Sharp Tech? Thursday.

We We got to get on and get him to correct the record because I I don’t know. I I find it so annoying that the most prominent voice in tech is trying to be an ally for our point of view on distillation is that we should do nothing. Um but like it’s hard. It’s like he has such wide reach that this is now going to be the status quo that we have to debunk which I guess it’s a better status quo than I don’t know actually no it’s not helpful because he’s saying that distillation is more important which means the people who are afraid about that are going to use that as a data point to say that we should take action even if they don’t because they probably won’t agree with his conclusions. I don’t know. That was my rant. Ben, you’re wrong.

00:39:16 Florian Brand: Yeah, I-I do think it is important to to differentiate these phases. Um and especially like there is no doubt that it is used during the SFT stage which is the first stage of or one of the stages for post-training and that’s also where the model picks up its manners like that’s why the models say oh I am Claude because they learn this during the SFT stage that’s where uh this personality is formed but the strong capabilities come during the RL stage which is where the money is spent which is where you need to have a fast enough judge which in the best case just runs in at the same GPUs or very close to your GPUs with smallish or with a fast enough model so you can uh are not bottlenecked by this.

Um and it in terms of impact it is also very hard to say how much impact or how much of a boost the better model SFT data gives you versus a lesser model. Um so if you are able to to have 10 million tokens from the latest Claude model versus two generations behind open model how much of a boost that really gives you if you keep the stage right uh the same and the pre-training stage the same is an open question which I don’t think we will see answered in a paper because then you have to showcase your uh SFT and your jailbreaking capabilities

00:40:48 Nathan Lambert: but I I wanted to double down on this like there’s been a good amount of literature on generating SFT reasoning traces whether it’s the most prominent ones have been opens line of work they did Open Thoughts 3 and Open Thoughts Agent have kind of been the foundational like scaling reasoning SFT works in the last few years and whenever somebody revisits this question they have not found the answer that the strongest model on performance in your domain is the best teacher for SFT people have try I’ve tried many people have tried the idea is so simple is like the state-of-the-art open SFT data set is built on QwQ-32B like an ancient reasoning model or something.

Why can we not just generate completions from GLM 5.2 do SFT on it and improve the model? We don’t know. It’s like the research so many people have tried and it is not an answered research question. There might be something like the base model the mid-training is too close to Qwen. So therefore it’s like hard to break. You have to redo the mid training. I think you have to redo the mid-training for reasoning. I think reasoning mid-training and reasoning SFT are so closely intertwined. It almost doesn’t make sense to have different words for them. That could be the issue. But the literature doesn’t even know how to ext like if I had a magical API that gave me reasoning traces from Claude/Gemini. I actually don’t know if me like fine-tuning an OLMo model on that would make OLMo smarter.

It’s one of the most wild unanswered research questions. And this is just makes the distillation thing so funny where it’s like yes the Chinese labs I think are using strong models like Opus for some SFT data but they’re also innovating. I was like I would love to them to tell us how to make this freaking work. And I think it’s the the paradigm I think is like open AI and anthropic find a niche domain that they do so well at and then the Chinese labs can get some samples there to kind of bootstrap their data engine and that’s where you will gain you will gain a few months on a specific domain.

But a hill climbing on these core domains like math and code and like Terminal-Bench like they’re just doing the same thing which is like so hard to generate prompts which are problems with environments that are hard for the current models and provide real nonreward hacking um learning behavior. And like that is what frontier data research looks like right now. And it is like it’s hard to generate these hard problems. And I’m sure the Chinese labs are doing the same the same things. And I don’t I don’t know. That’s that’s my rant. I’m kind of lost the context of our conversation.

00:43:23 Florian Brand: No, no, I would I would agree. Or to to to recap, yeah, SFT or distillation has some effect. Yeah, it gives them a boost, but not that much uh as people would like or or seem to think it gives.

00:43:39 Nathan Lambert: I think that’s that’s a good good summary of the of the of the conversation. It also becomes kind of tiresome because it says uh that open all all these open models are just good because they are distilling. um which definitely isn’t the case cuz if if it were the case, everyone would would be easily able to catch up to a GLM or to a K3 um by using its data for distillation. But we have not or we won’t see this from SFT alone.

00:44:12 Florian Brand: Yeah, I agree. Do you have any predictions or or more topics you want to get to?

00:44:18 Nathan Lambert: Um in terms of predictions, I think we are or I I revisited uh ours from from last year and it basically said everything will continue uh like it did uh the previous year. Uh we predicted that we will see bigger models uh up over two trillion parameters which it did and I don’t think we will see a much bigger explosion in terms of model size this year. we might see something or some model a bit bigger than three trillion parameters uh total but I don’t expect a five or 10 trillion parameter model and we open this year that would really surprise me um then list from last year can we redo this we don’t have to do the whole thing

00:45:03 Florian Brand: oh sure

00:45:03 Nathan Lambert: this is where we were at the end of 2025 who do you put in frontier now well it is Kimi and it is Zhipu DeepSeek is kind of a hard nut these days. Like I think they would be in close competitors. So I would put DeepSeek and Qwen the one as close competitors with Kimi and Zhipu as Frontier. Do you think anyone else would deserve close competitor? Cuz after that noteworthy and below like there’s so many.

00:45:38 Florian Brand: I I think we will see a surprise from MiniMax by end of the year. I think we will see a big model which like a really big model not uh M3 size but trillion parameters plus which will surprise us in terms of uh the outputs of MiniMax compared to before uh so I would still put them at close competitors by the end of the year

00:45:54 Nathan Lambert: do you think any US companies will be in the closed competitors by end of the year Nemotron I don’t think I would put there Thinking Machines closer especially if the smaller model really breaks through. But I don’t think I would put them there yet. Reflection is supposedly like only wants to release if they have a model that’s frontier. But then the question is will we get it? Like do we think that any US companies will get into this what is roughly like our top five by the end of the year? So the top five are the same but reshuffled.

00:46:36 Florian Brand: I would say it is possible uh that they are really close. Um it it also depends on what we think matters for closeness. Like I think uh Nemotron and um uh Thinking Machines will release models which act as really good base to be fine-tuned for your domain which doesn’t mean they are usable like a frontier model but they have so much utility uh that I would put them into close competitors because you would just need to find your data and uh to push the model into the right direction.

00:47:12 Nathan Lambert: Um as a I was going to think that we would make this a group of six with a US company by then like if we do this in late November I would guess that a US company pro most likely Nvidia thinky or Reflection mo does stuff that gets us to say that there is an American company in this like top cluster which would be a first time for a while.

00:47:43 Florian Brand: Yeah, I I I think that is realistic. My my one wild card is Tencent, which I think we might see something by end of the year. Uh they got some new leadership. Uh they released their Hunyuan model under Apache this time. Wait, so Tencent always had these custom licenses which disallowed anyone in the UK and South Korea and the entirety of the EU to to use their their model and also had acceptance use policy and so on. Um, and with Hunyuan and their new leadership, they got a really competent model at 250ish billion parameters. Um and I think by end of the year we might see a big model release which will surprise the people not following the ecosystem.

00:48:36 Nathan Lambert: Yeah, I I am also sure we will be in for some surprises. This is always the thing with AI and especially open models. It’s very very unpredictable. Okay, I I think this is a good place to stop. We probably should really do this quarterly. It’s not that hard and people will enjoy it. Um, but good to see you and we’ll talk soon. Hopefully in person soon.

00:49:02 Florian Brand: Peace.