August 2026 notes on stubs

Longer focus posts I want to write or find

2026-08-03

These are some one-liners that I want to expand into ~600-word focus posts to link to from other posts (or just find a good canonical reference that makes the point).

eval data is what you eval on...
- Generally, there is no special token within a document, property within a JSON object, or column within a tabular dataset that makes a certain data record "eval data."
- Amongst the universe of all data objects, some are more useful for performing "evaluation operations".
- In traditional supervised ML, eval data is just holdout data.
- In the modern LLM world, eval data stands in stark contrast to pretraining data because it has lots of explicit structure. The contrast between eval and post-training data is less stark, making dual use easier.
there's always a model in prod...
- all models are wrong, some models are useful, and the powers that be, by definition, always have a "model" running in prod (from a systems/control perspective -- there's always something making decisions, even if the decision is to go with the default, baseline, or dummy option)
scaling laws for evals...
pretraining detente and clearer public/private split
centrist take on distillation
we should freak out about Terms of Service from organizations trying to build "AGI"

Source revision history

Selected Git commits that changed this source file.

4e8762cc69 2026-08-03 - Prepare public focus notes
d96a1e08ab 2026-08-03 - Revise drafts on evaluation debt and replacing actions

Source and AT Protocol record

Source path
content/writing/short-posts/2026-08-03-stubs.md

AT Protocol URI
at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.document/3ms72ezdgnysp

Local AT Protocol-shaped preview used to inspect the record before an exact public cache is refreshed.

{
  "note": "Local AT Protocol-shaped preview. Run `make garden-refresh-atproto` to cache exact public records where available.",
  "sourcePath": "content/writing/short-posts/2026-08-03-stubs.md",
  "uri": "at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.document/3ms72ezdgnysp",
  "value": {
    "$type": "site.standard.document",
    "title": "August 2026 notes on stubs",
    "description": "Longer focus posts I want to write or find",
    "publishedAt": "2026-08-03",
    "site": "at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.publication/3mmpcciuaj22a",
    "content": {
      "$type": "at.markpub.markdown",
      "text": "These are some one-liners that I want to expand into ~600-word focus posts to link to from other posts (or just find a good canonical reference that makes the point).\n\n- eval data is what you eval on...\n\t- Generally, there is no special token within a document, property within a JSON object, or column within a tabular dataset that makes a certain data record \"eval data.\"\n\t- Amongst the universe of all data objects, some are more useful for performing \"evaluation operations\".\n\t- In traditional supervised ML, eval data is just holdout data.\n\t- In the modern LLM world, eval data stands in stark contrast to pretraining data because it has lots of explicit structure. The contrast between eval and post-training data is less stark, making dual use easier.\n- there's always a model in prod...\n\t- all models are wrong, some models are useful, and the powers that be, by definition, always have a \"model\" running in prod (from a systems/control perspective -- there's always something making decisions, even if the decision is to go with the default, baseline, or dummy option)\n- scaling laws for evals...\n- pretraining detente and clearer public/private split\n- centrist take on distillation\n- we should freak out about Terms of Service from organizations trying to build \"AGI\"\n"
    }
  }
}