This reminds me of "revealed preferences" in economics, which are inferred from what people do. It sounds like you're saying that revealed preferences aren't the kind of preferences you're interested in?
Something I've been curious about: If you have a pretrained LLM and then some domain specific data, are you better off finetuning the LLM with SGD or use a reward model to refine it? For example, say that you wanted to generate news stories about sports and had some examples. Would you fine tune it with those stories, or train a reward model with the sports stories vs non-sports stories to refine the model. I'm not sure what the best practice is here.
The company operating the Fireside Chat form does a shady marketing tactic in it: After one enrolls for the lecture, auto-subscribes the applicant to their newsletter, asks to *mail them back* if one wants to unsubscribe, even though the mailing list wasn't mentioned anywhere in the event form:
"""
You've been added to Rora's newsletter list where you'll receive information on relevant events and industry updates. If you're not interested in being on this list, please email hi@teamrora.com.
"""
Also, your name is (mis?)spelled as Michael in the form questions.
This reminds me of "revealed preferences" in economics, which are inferred from what people do. It sounds like you're saying that revealed preferences aren't the kind of preferences you're interested in?
What's a better way to think about preferences?
I think I’m interested in two things:
1. Models are designed to optimize stated preferences now, seems risky.
2. Models could optimize revealed preferences in the future, seems risky.
Something I've been curious about: If you have a pretrained LLM and then some domain specific data, are you better off finetuning the LLM with SGD or use a reward model to refine it? For example, say that you wanted to generate news stories about sports and had some examples. Would you fine tune it with those stories, or train a reward model with the sports stories vs non-sports stories to refine the model. I'm not sure what the best practice is here.
Compare two baselines:
1. Fine tune the model on all the specific date,
2. Fine tune the model on part of the data, train a reward model on part, and run RLHF on the last part.
Best practices of RLHF is not so clear.
So basically for now all we know is treat using RLHF as a hyperparameter and crossvalidate.
It might make for an interesting paper to look at a few examples and see how they perform.
Agreed. We’re working on that.
The company operating the Fireside Chat form does a shady marketing tactic in it: After one enrolls for the lecture, auto-subscribes the applicant to their newsletter, asks to *mail them back* if one wants to unsubscribe, even though the mailing list wasn't mentioned anywhere in the event form:
"""
You've been added to Rora's newsletter list where you'll receive information on relevant events and industry updates. If you're not interested in being on this list, please email hi@teamrora.com.
"""
Also, your name is (mis?)spelled as Michael in the form questions.
I’ll let them know - it’s a small business, auto subscribe may be unintentional. Thanks for the feedback.