Chinese labs are just behind in time, and will benchmax less as they get PMF. I think DeepSeek probably benchmaxes the least bc they seem to not care what people think
I used to do it with AI, but doing it human has a very high ROI, it’s much better (it's richer, picks up on my emphasis), and is a very valuable proofreading pass for me. Lots of people I respect are very loyal listeners.
Timeliness and completeness are also drivers of mind-share for Artificial Analysis versus other benchmarkers. I checked every mentioned site, and Vals Index is the only other place that seems to have Kimi 3.1 Pro fully incorporated.
So much of the interest on a model surrounds launch, that having benchmark numbers up on launch day is going to drive attention to one site versus another.
Thanks for the piece! I’d be curious for any good critiques of ECI, which gets discussed less
{summoning florian, hopefully he doesn't hate my post, he's given a good reply here before but I don't remember where}
Thanks!
How do you reconcile the following:
1. OAI and Ant aren't benchmark-maxxing but doing post-training for optimizing other metrics.
2. Chinese labs are more into benchmark-maxxing.
3. Chinese labs want to buy the data and RL environments that OAI and Ant buy.
Chinese labs are just behind in time, and will benchmax less as they get PMF. I think DeepSeek probably benchmaxes the least bc they seem to not care what people think
> Housekeeping: I’m on the road so the voiceover will be late.
Oh! I had always assumed the voiceover was simply an AI model trained to sound like you! 😂😂😂
I used to do it with AI, but doing it human has a very high ROI, it’s much better (it's richer, picks up on my emphasis), and is a very valuable proofreading pass for me. Lots of people I respect are very loyal listeners.
Timeliness and completeness are also drivers of mind-share for Artificial Analysis versus other benchmarkers. I checked every mentioned site, and Vals Index is the only other place that seems to have Kimi 3.1 Pro fully incorporated.
So much of the interest on a model surrounds launch, that having benchmark numbers up on launch day is going to drive attention to one site versus another.
The lack of blue coloring for Kimi also initially tripped me up
potentially a psyop...