ON THIS PAGE (10 SECTIONS)
- ✓A pilot is allowed to be small, to draft rather than write, and to be turned off without a customer notice.
- ✓The feature leaves that permission the moment it writes to a system of record or is treated as the official process.
- ✓Exit requires an owner, a cost ceiling, a measure tied to the decision, and a stop condition someone can execute.
- ✓The technical production checklist comes after this decision. A neat system can still be the wrong thing to operate.
The demo worked. A handful of people liked it. The next request is to roll it out. Nothing in that request says who is on call when the output is wrong, who pays the provider invoice, or what result would make the team turn the feature off.
A pilot is a learning setup with a small blast radius. Production is an obligation: a named owner, a budget, a way to stop, and a measure that is more specific than “people said it was helpful.” Calling the pilot live because the demo was impressive hands the risk to users who did not agree to be the test.
Leave the pilot, or keep it a pilot
Leave the pilot when you can name the decision the feature may influence, the person accountable for a wrong output, the cost ceiling, and the condition that stops it. If you cannot name those, keep the audience small or end the work. Widening access is not a substitute for the decision.
What a pilot is still allowed to do
A pilot may serve a named group. It may run in shadow, writing nothing. It may require a person to approve every action. It may change every week. It may be switched off without a customer notice, because the users know it is a trial.
Those permissions expire. When the feature writes to a system of record, sends a message a customer will read, or is left on for a team that treats it as the official process, it is no longer a pilot. It is an unowned production path.
Activity is a weak reason to continue
Pilots linger when success is defined as motion: prompts sent, a positive meeting, a screenshot. None of those say whether the feature should exist next quarter.
Martin Zinkevich’s Rules of Machine Learning, published by Google, start with a useful restraint. Rule 1 says not to be afraid to launch a product without machine learning. A model needs data, and a heuristic or a person can be the right system until that data exists. Rule 2 says to design and implement metrics before formalizing what the learned system will do. A pilot that cannot say what it is measuring is not gathering the evidence that would justify a wider rollout. The “100 percent versus 50 percent” line in that rule is Zinkevich’s illustration, not a measured result you should import into a business case.
A decision table for the exit meeting
| Question | Stay in the pilot | Call it production | Stop |
| What decision does this change? | The team is still finding the decision | The decision and the user are named | No decision is clearer with the model |
| Who answers for a wrong output? | The person running the trial | A role with time and authority | Nobody will own it |
| Does it write, send, or draft? | Draft or shadow only | Writes are validated or approved by a person | It writes with no check |
| What will you review for the next month? | Not defined | A measure tied to the decision, read by a person | The only number is usage |
| What stops it? | Not written | Written, and someone can execute it the same day | Stopping would be treated as failure of the sponsor |
| What is the monthly ceiling? | Unknown | Set, with an alert before it is hit | Spend can grow with retries and traffic |
A “yes” in the middle column is not the technical launch review. The Production Readiness Checklist for Enterprise AI Systems covers idempotency, quotas, injection containment, and evaluation. This table is the operating decision that comes first. A feature can pass a technical review and still be the wrong thing to leave on.
Illustrative scenario: draft replies for support
This scenario is illustrative. It reports no time saved, because that figure was never instrumented.
A support team tries a draft-reply tool with a small group of agents. Some drafts are sent after edits. Some are discarded. Leadership asks for every agent next Monday.
The exit conversation should stay specific. The feature may draft. It may not send. The sent text belongs to the agent who sent it. Handle time was not measured before the tool existed, so the team does not invent a savings number. It can measure the share of drafts discarded, and the support lead can review a sample each week for policy mistakes. The stop condition is written: repeated policy errors return the tool to the original group the same day. The provider spend stays inside a ceiling finance has already approved. Access widens only with those conditions in force, not because the calendar says Monday.
One page the on-call engineer can find
- Purpose: the decision this feature is allowed to influence.
- Users: who is in the audience right now.
- Prohibitions: what it must not write, send, or decide.
- Owner: the role that reviews bad output.
- Ceiling: the spend or rate that triggers an alert.
- Stop condition: the observation that returns it to the pilot group or turns it off, and who executes that.
- Review date: when the owner looks again, even if nothing is on fire.
Put that page where the person on call will actually open it. A trial without the page is a demo that forgot to end. The habits behind a stoppable release are part of how the engineering work is run, which the methodology describes without turning this decision into a tool purchase.
Questions that usually come next
Is a successful demo an exit criterion?
A demo shows that one path can work once. Exit requires an owner, a ceiling, a stop condition, and a measure you will review. Widen access after those exist.
What if the business wants it live next week anyway?
Keep the write path off. A draft a person sends is a smaller obligation than a feature that sends on its own. Put the owner and the stop condition in writing the same week. A date is not a reason to skip the stop condition.
Direct answers
Is a successful demo an exit criterion?
A demo shows that one path can work once. Exit requires an owner, a ceiling, a stop condition, and a measure you will review.
What if the business wants it live next week anyway?
Keep the write path off. A draft a person sends is a smaller obligation than a feature that sends on its own. Write the owner and the stop condition the same week.
References & Prior Art
Rules of Machine Learning
Martin Zinkevich • Google for Developers (2016)
Vishal
Founder & Engineering LeadVishal leads engineering at Vasiva, establishing merge gates, deterministic verification patterns, and human-in-the-loop review architecture across production AI systems.
