A short argument that "augment, do not replace" is about data control.
For some time now, figures in the AI industry have been saying things
along the lines of: “We want to build AI that augments, but doesn’t
replace, humans.” I think the idea is great. But so far, we haven’t seen
much evidence that expressing pro-augmentation views has led to firm
commitments about how to actually make this happen. Perhaps these
statements are shaping internal priorities (more compute for interface
experiments, more product work on copilots, more attention to human-AI
workflows, etc.), but we can't be sure.
Allocating more resources toward interface-focused research will
likely produce tools that are better at augmentation. However, I do
not think that making AI systems better at augmenting workers
will do much, by itself, to prevent substitution or replacement. In
fact, augmenting systems may accelerate replacement in domains that are
currently data scarce.
Anytime a worker uses an AI system to perform some task requiring
what we might now think of as "human stuff" (judgment, taste, domain
knowledge, private context, etc.) they produce rich workflow traces and
outcome data. If those traces are captured by the AI firm or employer,
they become training and evaluation data. The next model becomes better
at doing the task with less human input. The worker’s marginal
contribution and bargaining power fall, even though the original system
was “augmenting” at deployment time.
If we really believe in data scaling, we should expect scaling to
apply to any capability domain that can be captured in data records.
Many areas where models are currently bad are areas where data is harder
to get. But if we had the data, and no countervailing force, why
wouldn't models be able to learn the necessary patterns? (Of course, I
do not mean to argue that every social or relational dimension of work
will disappear or that demand for human labor will approach zero. People
may continue to value human presence, accountability, care, etc., but
even these aspects of labor will not be immune to data scaling.)
One tempting response is to say that we should simply impose
“augment-only” rules at the modelling level: build systems that assist
workers but are somehow prevented from replacing them. I am skeptical
that this is technically coherent. Once a model has learned a
capability, it is very hard to guarantee that the capability will only
be used in complementarity with human labor rather than in substitution
for it. The same ability that makes a system useful as a copilot often
makes it useful as a replacement, especially once the surrounding
workflow is redesigned around the model. So I do not think “augment,
don’t replace” can be secured primarily through model behaviour. It has
to be secured through constraints on data capture, deployment,
ownership, and use.
So I'd contend: an AI system will not be stably augmentative just
because it is deployed as a copilot. It can be stably augmentative only
if the institutions around it preserve meaningful control over use-time
information and rights over downstream training/evaluation data. Many
efforts to build augmenting systems -- with the best of intentions --
may directly support replacement unless they somehow restrict the flow
of data. This friction could come in the form of increased individual
data rights and/or an approach emphasizing data intermediaries and
collective bargaining.
ATProto local JSON preview
{
"note": "Local ATProto-shaped preview. Run `make garden-refresh-atproto` to cache exact public records where available.",
"sourcePath": "02_shortposts/2026-05-26-augmentation-is-a-data-flow-problem.md",
"uri": "at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.document/3mni4cwedk57p",
"value": {
"$type": "site.standard.document",
"title": "Augmentation is a data flow problem",
"description": "A short argument that \"augment, do not replace\" is about data control.",
"publishedAt": "2026-05-26",
"site": "at://did:plc:doxvahqvyhyqf32v7wz7p5xk/site.standard.publication/3mmpcciuaj22a",
"content": {
"$type": "at.markpub.markdown",
"text": "For some time now, figures in the AI industry have been saying things along the lines of: “We want to build AI that augments, but doesn’t replace, humans.” I think the idea is great. But so far, we haven’t seen much evidence that expressing pro-augmentation views has led to firm commitments about how to actually make this happen. Perhaps these statements are shaping internal priorities (more compute for interface experiments, more product work on copilots, more attention to human-AI workflows, etc.), but we can't be sure.\n\nAllocating more resources toward interface-focused research will likely produce tools that are better at augmentation. However, I do _not_ think that making AI systems better at augmenting workers will do much, by itself, to prevent substitution or replacement. In fact, augmenting systems may accelerate replacement in domains that are currently data scarce.\n\nAnytime a worker uses an AI system to perform some task requiring what we might now think of as \"human stuff\" (judgment, taste, domain knowledge, private context, etc.) they produce rich workflow traces and outcome data. If those traces are captured by the AI firm or employer, they become training and evaluation data. The next model becomes better at doing the task with less human input. The worker’s marginal contribution and bargaining power fall, even though the original system was “augmenting” at deployment time.\n\nIf we really believe in data scaling, we should expect scaling to apply to any capability domain that can be captured in data records. Many areas where models are currently bad are areas where data is harder to get. But if we had the data, and no countervailing force, why wouldn't models be able to learn the necessary patterns? (Of course, I do not mean to argue that every social or relational dimension of work will disappear or that demand for human labor will approach zero. People may continue to value human presence, accountability, care, etc., but even these aspects of labor will not be immune to data scaling.)\n\nOne tempting response is to say that we should simply impose “augment-only” rules at the modelling level: build systems that assist workers but are somehow prevented from replacing them. I am skeptical that this is technically coherent. Once a model has learned a capability, it is very hard to guarantee that the capability will only be used in complementarity with human labor rather than in substitution for it. The same ability that makes a system useful as a copilot often makes it useful as a replacement, especially once the surrounding workflow is redesigned around the model. So I do not think “augment, don’t replace” can be secured primarily through model behaviour. It has to be secured through constraints on data capture, deployment, ownership, and use.\n\nSo I'd contend: an AI system will not be stably augmentative just because it is deployed as a copilot. It can be stably augmentative only if the institutions around it preserve meaningful control over use-time information and rights over downstream training/evaluation data. Many efforts to build augmenting systems -- with the best of intentions -- may directly support replacement unless they somehow restrict the flow of data. This friction could come in the form of increased individual data rights and/or an approach emphasizing data intermediaries and collective bargaining."
}
}
}