An agent that regulates itself

Most safety work on agents is about the outside world: what the agent may touch, which tools it may call, whose approval it needs. That matters, and it misses the variable that actually predicts trouble — the agent’s own behaviour. An agent that notices its accuracy slipping and narrows its own scope is safer than one held in place by a longer list of rules, because the rules only cover failures somebody already imagined.

Four variables worth watching

Cost. Tokens, calls, wall-clock time. Not primarily a budget concern: a task that suddenly costs five times its usual amount is a task that has gone wrong, and cost is the cheapest early signal you will ever get.

Confidence, calibrated against outcomes. Not the model’s stated certainty, which is close to useless on its own, but the historical hit rate for this kind of decision. If proposals of this class have been accepted nineteen times out of twenty, that is a number you can regulate against.

Correction rate. How often you edit or reject what it produced, sliced by task type. A rising correction rate on one class of work is the clearest signal that something upstream changed — a new document format, a shifted process, a model update.

Autonomy actually exercised. How much the agent is doing without asking, compared to what it is permitted to do. A slow creep towards the ceiling deserves attention even when nothing has gone wrong yet.

The response has to be automatic

Measuring is the easy half. The property that matters is what happens when a variable leaves its range, and the answer has to be that the agent acts before you do.

Correction rate on invoice extraction doubles: that task drops back to proposal-only, and you get told. Cost per task jumps: the agent stops and asks rather than spending. Confidence in a decision class falls below its set point: the boundary for that class narrows automatically. Something looks anomalous in a way the agent cannot classify: it does the smallest reversible thing and escalates.

This is the homeostat pattern — a system that changes its own configuration to bring a variable back into range. Applied to an agent, the configuration being changed is how much freedom it is currently taking, which is exactly the right dial.

Why this beats a longer rulebook

The instinct after an incident is to write another rule. That covers the case you just saw and nothing else, and after two years the rulebook is a liability nobody fully understands.

Self-regulation inverts it. Instead of enumerating forbidden actions, you declare a small number of things that must remain true about the agent’s own performance. Inside that envelope it is free; at the edges it corrects itself. The rulebook stops growing because the boundary is defined by outcomes rather than by an ever-longer list of anticipated failures.

It also degrades in the right direction. A rule-based system that meets an unanticipated situation either blocks something harmless or permits something it should not. A self-regulating one gets quieter and more cautious, which is the failure mode you want from software acting on your behalf.

The uncomfortable parts

A homeostat only regulates what it can measure. Choosing the variables is the most important design decision in the system, and anything left uninstrumented can drift freely.

There is also a real tension with usefulness. An agent tuned too tightly retreats to proposal-only at the first wobble and becomes a slower way to do your own work. The set points are a product decision, not a safety checkbox, and they should be visible and adjustable rather than buried.

Which is the last argument for open source in this design: a regulator you cannot inspect is just another thing to trust. The whole point is not having to.