You’re framing this as a divide between open vs closed models, but isn’t it more a split between open and closed eval methodology/code, or is there something missing from API access to closed models? (in this GPT4 case, the eval isn’t released, and the model isn’t released in API form, but that feels like a different complaint)
I'm trying to get across a more polemic rather than analytic vibe here, because all the axes you mentioned exist and are hard to matter. So, in short, yes.
But in specific, API access doesn't mean raw weights, which is why I get sad.
You’re framing this as a divide between open vs closed models, but isn’t it more a split between open and closed eval methodology/code, or is there something missing from API access to closed models? (in this GPT4 case, the eval isn’t released, and the model isn’t released in API form, but that feels like a different complaint)
I'm trying to get across a more polemic rather than analytic vibe here, because all the axes you mentioned exist and are hard to matter. So, in short, yes.
But in specific, API access doesn't mean raw weights, which is why I get sad.