These are some one-liners that I want to expand into ~600-word focus posts to link to from other posts (or just find a good canonical reference that makes the point).
- eval data is what you eval on...
- Generally, there is no special token within a document, property within a JSON object, or column within a tabular dataset that makes a certain data record "eval data."
- Amongst the universe of all data objects, some are more useful for performing "evaluation operations".
- In traditional supervised ML, eval data is just holdout data.
- In the modern LLM world, eval data stands in stark contrast to pretraining data because it has lots of explicit structure. The contrast between eval and post-training data is less stark, making dual use easier.
- there's always a model in prod...
- all models are wrong, some models are useful, and the powers that be, by definition, always have a "model" running in prod (from a systems/control perspective -- there's always something making decisions, even if the decision is to go with the default, baseline, or dummy option)
- scaling laws for evals...
- pretraining detente and clearer public/private split
- centrist take on distillation
- we should freak out about Terms of Service from organizations trying to build "AGI"
- it's a worthy target for advocacy to just get key actors in the data debates to state earnestly the IP regime and data rules they'd like to see