Interconnects

Home
Podcast
Navigation
Archive
About

RLHF

Tülu 3: The next era in open post-training
We give you open-source, frontier-model post-training.
 • 
Nathan Lambert
3
Reverse engineering OpenAI’s o1
What productionizing test-time compute shows us about the future of AI. Exploration has landed in language model training.
 • 
Nathan Lambert
2
Futures of the data foundry business model
Scale AI’s future versus further scaling of language model performance. How Nvidia may take all the margins from the data market, too.
 • 
Nathan Lambert
A post-training approach to AI regulation with Model Specs
And why the concept of mandating “model spec’s” could be a good start.
 • 
Nathan Lambert
A recipe for frontier model post-training
Apple, Meta, and Nvidia all agree — synthetic data, iterative training, human preference labels, and lots of filtering.
 • 
Nathan Lambert
RLHF roundup: Getting good at PPO, sketching RLHF’s impact, RewardBench retrospective, and a reward model competition
Things to be aware of if you work on language model fine-tuning.
 • 
Nathan Lambert
OpenAI’s Model (behavior) Spec, RLHF transparency, personalization questions
Now we will have some grounding for when weird ChatGPT behaviors are intended or side-effects — shrinking the Overton window of RLHF bugs.
 • 
Nathan Lambert
How RLHF works, part 2: A thin line between useful and lobotomized
Many, many signs of life for preference fine-tuning beyond spoofing chat evaluation tools.
 • 
Nathan Lambert
Stop "reinventing" everything to solve alignment
Integrating some non-computing science into reinforcement learning from human feedback (RLHF) can give us the models we want.
 • 
Nathan Lambert
Why reward models are key for alignment
In an era dominated by direct preference optimization and LLM-as-a-judge, why do we still need a model to output only a scalar reward?
 • 
Nathan Lambert
4
Alignment-as-a-service: Scale AI vs. the new guys
Scale’s making over $750 million per year selling data for RLHF, who’s coming to take it?
 • 
Nathan Lambert
RLHF learning resources in 2024
A list for beginners and wannabe experts and everyone in between.
 • 
Nathan Lambert
© 2024 Nathan Lambert
Privacy ∙ Terms ∙ Collection notice
Start WritingGet the app
Substack is the home for great culture