36 Comments
User's avatar
Krrish Agarwalla's avatar

Damm finally a fully open source model. I loved when labs released their models and initially called them open source rather than open weights.

Thank you for sharing the training pipeline as well

Rob Bru's avatar

This is great news I am psyched for many reasons. Look forward to exploring, tinkering, building with the Olmo 3 models. 🦅

Chef abee Crombie's avatar

wow. for 7B, the thinking model is pretty good. the foundational models are amazing for general tasks but if you can fine tune a 7b beast like this to your domain/problem i'll always err on the smaller open models for repeatable results.

Bill Bishop's avatar

CrowdStrike Research: Security Flaws in DeepSeek-Generated Code Linked to Political Triggers https://www.crowdstrike.com/en-us/blog/crowdstrike-researchers-identify-hidden-vulnerabilities-ai-coded-software/

Florian Brand's avatar

This is a rather weak analysis, imo. They do not compare the same prompts when used on the two western models (which likely are Llama 3 70B and GPT-OSS 120B).

A lot of these reports more or less test who much data the model was trained on and how "brittle" it is when used with prompts that throw the model off-guard. DeepSeek famously used little data and (relative) compute, which shows. GPT-OSS is the same (if you want to have fun, use the model with a non-English language).

As an example for both: CAISI has analyzed DeepSeek (and compared against GPT-OSS). Their reported numbers for SWE-Bench Verified (a software engineering benchmark) differs *greatly* from the official numbers cause their agent harness (i.e., the software the model runs in) is different from the ones used by the model makers. GPT-OSS and all DeepSeek models plummet, while the closed models (trained on more data, more robust, just better models overall) have ~ the same performance in the CAISI harness compared to the officially reported scores.

https://www.interconnects.ai/p/latest-open-models-15-its-qwens-world & https://www.nist.gov/news-events/news/2025/09/caisi-evaluation-deepseek-ai-models-finds-shortcomings-and-risks

Logan Thorneloe's avatar

Love to see this!

Ilia Karelin's avatar

Turns out, there’s someone who makes one of the best open-source models in US.

Amazing write up and amazing release!

Rob Neuhaus's avatar

Awesome. Great work! Was SuperBPE too risky to try for the tokenizer? It seems like a great effiency win for both training and inference.

Nathan Lambert's avatar

uhHHhHHh email Luca S or Alisa Liu

John Dvorak's avatar

Nathan,

My sincerest congratulations to you and the whole Olmo 3 team! I’m incredibly excited to dive in and kick the tires on these new models. And perhaps more eager to read your lessons learned and updated model training recipes. I’m certain all will provide great value to the global open source AI community.

This Fall has been a long-needed renaissance for American open source AI with the IBM Granite, Nvidia Nemotron, and now Olmo 3 model releases. And ICYMI, yesterday Deep Cogito released an absolute powerhouse V2.1 of their open source 671B MoE model. I’ve been running IMO Olympiad style math problems through and the results have been exceptional.

Congrats again on Olmo 3. The future is indeed bright for American open source!

Bill Bishop's avatar

Isn't Deep Cogito built on deepseek?

Nathan Lambert's avatar

It is. They asked me for feedback on their release before launching and I suggested they change their explanations. Alas.

Base models are a crucial part of the stack that cannot be ignored.

Bill Bishop's avatar

honestly their explanations seem a bit obfuscatory if not outright deceptive. Also, given the political environment in DC, good luck to any firm that uses a PRC model, no matter how forked, to write code or perform other services they want to sell to companies in regulated industries or ones that have US government contracts. Cheap now perhaps, not so cheap when they all have to be ripped out, and I will bet decent $$s that at some point bodies like the House Select Committee on China are going to make noise about this...maybe should start a polymarket...

Nathan Lambert's avatar

Agreed. I'm legitimately excited about people using our base models for real things. Mostly to date Olmo has been just a research artifact.

I'm long overdue for a DC trip...

John Dvorak's avatar

Yes, DC is a fork from DeepSeek. DC integrates its own training on top of the DeepSeek base using Iterated Distillation & Amplification and iterative policy improvement.

The idea is to internalize reasoning and iteratively shorten chains of thought. They put the high level details here:

https://huggingface.co/blog/deepcogito/cogito-v2-1

Bill Bishop's avatar

so just to be clear, very far from an American model

John Dvorak's avatar

Fair. A philosophical question (which I don’t know the answer to) is at what point does a fork evolve and mature to the point where the base ceases to matter in a meaningful way. As in DS was the bootstrap to an iterative process in which the final version retains the form of the original while having completely different weights, parameters, and reasoning chains.

Bern's avatar

deepseek doublespeek

Steven's avatar

Super excited to start setting it up. My one disappointment so far is that is not multimodal. Not sure if there are plans to support it in the future?

Nathan Lambert's avatar

We've had them as separate teams, will be working more closely in the future.

For now, some good news on that front soon.

Steven's avatar

Looking forward Nathan:)

JayNing's avatar

It seems that during the PyTorchCon presentation, it was mentioned that qk-norm might affect long context, yet it's still retained in Olmo 3. Was there any discussion about qk-norm?

Nathan Lambert's avatar

@JayNing, it's QK-Norm after the residual, which was more like the original transformer, that can cause problems.

Where some labs are QK norm pre residual, or somethin like this.

JayNing's avatar

I guess you mean Pre-LN and Post-LN? QK-Norm is usually strictly tied to the attention block, so I don't think it can really change positions relative to the residual.

Nathan Lambert's avatar

better response -- go look at the LC section of the paper (I will too)

JayNing's avatar

I did read that section, but I couldn't find any specific mention of qk-norm or its impact there. :(

Nathan Lambert's avatar

Omg lol now I need to know

Nathan Lambert's avatar

I’m too tired to think right now, but my instinct is qk norm made it harder, or something like that. Could also be the ordering of norms and my presentation wasn’t clear enough (I need to study modeling more tbh, maybe a reader will answer)