Skip to main content
A man in a long black coat faces a seated white-haired man in a pale suit, in a white room walled with small screens.

Justin Bartak · AI Org · · 5 min read

When Do I Get to Leave the Loop?

Originally published on Orbyt Labs

TL;DR

Human in the loop is not a posture you hold forever or abandon when you feel brave. It is a set of gates, each one opening on evidence you defined in advance. Some of mine have opened. Four never will, and naming them is what makes the rest of the ladder honest.

People ask how long the human stays in the loop as if the answer were a duration. It is not. Human in the loop is a set of gates, and each gate has its own answer.

Some of mine have opened. Four never will. Naming which is which is the only version of this question that has a real answer.

Time does not earn autonomy. Evidence does, and the evidence has to be defined before the run.

Why is "how long" the wrong question?

Because a period of good behavior is not proof of anything.

Six months without an incident could mean the system is safe. It could also mean nothing dangerous happened to be attempted. Human in the loop as a phrase invites this drift: the human gets tired, the approvals get faster, and one day the gate is open without anybody deciding to open it.

So the unit is not duration. It is a criterion, written down before the work starts, that a machine can check without anyone's opinion.

What does a real promotion criterion look like?

Mine live in a ladder. Each stage names the conditions a seat must satisfy to hold more authority, and every condition is deterministic: check results, committed artifacts, counted outcomes. Never self-assessment.

Two properties make it hold.

The criteria are never scored by the thing being promoted. An agent grading its own readiness is not a review, it is a request.

And the criteria file is pinned by hash. Change the bar and every streak earned under the old bar resets. Without that, a system can quietly lower the standard it is measured against and call the result progress, which is the most polite form of cheating a machine can commit.

Right now my ladder holds at stage 2 of 4, because two of its own criteria are unmet. It does not advance because I want it to. That is the property I was buying.

Which gates never open?

Four, and they are not slow lanes on the ladder. They are off the ladder entirely.

Money. Any spend, price, refund, or billing change. Reversible in accounting, not in trust.

Customer data. Anything that reads, moves, or exposes it. The blast radius does not shrink with model capability, and there is no undo for disclosure.

Public claims in my name. A published number is a promise. When it is wrong, it is wrong in a search result, in an answer engine, and in somebody's inbox.

The machinery itself. Charters, guards, gates, and the kill switch. A system that can widen its own authority has no authority worth measuring, which is why 0 of my 12 seats may merge to the branch that deploys.

Those four share one property: they are irreversible or reputation-bearing. Capability does not touch either. A model twice as good is not twice as safe to hand a refund button.

Everything reversible is negotiable. Everything permanent is not.

What does staying in the loop cost?

More than people admit, so here is my number.

30 of my last 50 Orbyt agent runs ended in a discard at a gate. An 84.3% completion rate, where roughly a third of the produced work is thrown away rather than waved through. That is real waste, and I publish it because it is the figure that argues against my own design.

The subtler cost is attention. A human asked to approve 200 items a week stops reading them by item 30. That is not a character flaw, it is arithmetic, and it produces the worst possible outcome: an approval record with no approval in it. An unread gate is more dangerous than no gate, because it manufactures evidence of oversight that nobody performed.

So volume is the real enemy of oversight, and the way out is fewer gates that mean more, not more gates that mean less.

What replaces you on a lane?

Never trust. Three specific things.

A criterion decided in advance, so the promotion is not a judgment call made while tired.

An outside checker that cannot write. Everything my agents can modify eventually reflects their own assumptions, so something outside the system has to grade it. Mine is a second model with read-only access and an inspector seat that reports and cannot act.

And reversibility. If a lane's worst outcome is a revert, it is a candidate for autonomy. If its worst outcome is a phone call to a customer, it is not, no matter how well it has behaved.

What to do Next

Write your ladder as three columns: what an agent may do alone today, what needs your signature, what is never delegated.

Most teams have never written the third column, and it turns out to be short and obvious once somebody says it out loud. Write it first. It makes the rest of the exercise honest.

Then take one item from column two and state the criterion that would move it to column one. Machine-checkable, defined now, scored by something other than the agent. If you cannot write that sentence, the gate is not ready to open, and now you know precisely what is missing instead of just feeling unsure.

The question was never how long I stay in the loop. It is what has to be true before a specific gate opens, and which gates I already decided never will.

Related reading:


Originally published on orbytlabs.ai on Oct 4, 2026.

Frequently asked questions

How long should a human stay in the loop with AI agents?

Time is the wrong unit. A gate should open when a pre-defined, machine-checkable criterion is met, not after a period of good behavior. Six months of nothing going wrong is not evidence, it is an absence of evidence. Write the criterion first, then let the system earn past it or fail to.

What makes an autonomy criterion trustworthy?

It has to be deterministic, checkable without a model's opinion, and never self-assessed by the agent seeking promotion. Mine are pinned by a hash of the criteria file, so changing the bar resets any streak earned under the old bar. Without that, a system can quietly lower the standard it is being measured against.

Which decisions should never leave human hands?

Four in my stack: anything touching money, anything touching customer data, any public claim made in my name, and any change to the governing machinery itself. They are not slow lanes on the ladder. They are off the ladder, because each one is either irreversible or reputation-bearing, and neither property improves with model capability.

Share this article

XLinkedIn
Justin Bartak, Chief AI Officer and AI-native product leader

Justin Bartak

4x founder and Chief AI Officer. $383M+ in enterprise value delivered across regulated fintech, tax, proptech, and CRM platforms. Recognized by Apple. Built Orbyt solo in 32 days with Claude Code. Founder of Purecraft.