These are some one-liners that I want to expand into ~600-word focus
posts to link to from other posts (or just find a good canonical
reference that makes the point).
eval data is what you eval on...
Generally, there is no special token within a document, property
within a JSON object, or column within a tabular dataset that makes a
certain data record "eval data."
Amongst the universe of all data objects, some are more useful for
performing "evaluation operations".
In traditional supervised ML, eval data is just holdout data.
In the modern LLM world, eval data stands in stark contrast to
pretraining data because it has lots of explicit structure. The contrast
between eval and post-training data is less stark, making dual use
easier.
there's always a model in prod...
all models are wrong, some models are useful, and the powers that
be, by definition, always have a "model" running in prod (from a
systems/control perspective -- there's always something making
decisions, even if the decision is to go with the default, baseline, or
dummy option)
scaling laws for evals...
pretraining detente and clearer public/private split
centrist take on distillation
we should freak out about Terms of Service from organizations trying
to build "AGI"
it's a worthy target for advocacy to just get key actors in the data
debates to state earnestly the IP regime and data rules they'd like to
see
Source details
Versions, revision history, mirror freshness, and source-level protocol data.
AT Protocol URI at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.document/3ms72ezdgnysp
Local AT Protocol-shaped preview used to inspect the record before an exact public cache is refreshed.
{
"note": "Local AT Protocol-shaped preview. Run `make garden-refresh-atproto` to cache exact public records where available.",
"sourcePath": "content/writing/short-posts/2026-08-03-stubs.md",
"uri": "at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.document/3ms72ezdgnysp",
"value": {
"$type": "site.standard.document",
"title": "August 2026 notes on stubs",
"description": "Longer focus posts I want to write or find",
"publishedAt": "2026-08-03",
"site": "at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.publication/3mmpcciuaj22a",
"content": {
"$type": "at.markpub.markdown",
"text": "These are some one-liners that I want to expand into ~600-word focus posts to link to from other posts (or just find a good canonical reference that makes the point).\n\n- eval data is what you eval on...\n\t- Generally, there is no special token within a document, property within a JSON object, or column within a tabular dataset that makes a certain data record \"eval data.\"\n\t- Amongst the universe of all data objects, some are more useful for performing \"evaluation operations\".\n\t- In traditional supervised ML, eval data is just holdout data.\n\t- In the modern LLM world, eval data stands in stark contrast to pretraining data because it has lots of explicit structure. The contrast between eval and post-training data is less stark, making dual use easier.\n- there's always a model in prod...\n\t- all models are wrong, some models are useful, and the powers that be, by definition, always have a \"model\" running in prod (from a systems/control perspective -- there's always something making decisions, even if the decision is to go with the default, baseline, or dummy option)\n- scaling laws for evals...\n- pretraining detente and clearer public/private split\n- centrist take on distillation\n- we should freak out about Terms of Service from organizations trying to build \"AGI\"\n- it's a worthy target for advocacy to just get key actors in the data debates to state earnestly the IP regime and data rules they'd like to see\n"
}
}
}