A customer-service AI agent handles a refund correctly during a demonstration. It finds the order, checks the policy and prepares the payment. The result looks convincing. But what happens if the customer submits the same request twice, the order system is unavailable or an email tells the agent to ignore the refund limit?
Those questions deserve as much attention as the successful demonstration. As businesses give AI tools greater freedom to act, they need evidence that the surrounding process can contain mistakes, detect exceptions and recover when something goes wrong.
MOYO has previously argued that autonomy changes the nature of business accountability. In published commentary, Dr Pierre Le Roux framed it as delegating authority to a digital actor. The practical next step is to test how that delegation behaves under pressure.
Test the business process, including the point where it should stop
A useful pilot should establish more than whether an agent completes a task. It should show whether the agent recognises when it lacks the information or authority to proceed.
Consider an illustrative procurement workflow. An agent may be permitted to prepare a purchase order from an approved supplier. Changing that supplier’s banking details or accepting a substitute product creates a different exposure. The workflow needs a clear point at which responsibility passes to an authorised employee.
That handover must work in practice. A request sent to an unattended inbox is not effective oversight. Someone needs to receive the exception, understand the issue and have time to intervene before the business makes a commitment.
Businesses should therefore test missing records, contradictory instructions, duplicate requests and unavailable systems. A safe refusal or timely escalation may be a better result than completing the task at any cost.
Put controls where actions happen
The security discussion is moving in the same direction. NIST’s work on AI-agent identity and authorisation highlights the importance of controlling access to tools and data, tracing actions and managing delegated authority.
For a business, this means a written spending rule should be backed by a control in the transaction process. The system that releases a payment should check the limit and approval, rather than relying entirely on the agent to remember an instruction.
It also means treating material the agent reads as information to assess, rather than a source of authority. A supplier attachment should not be able to grant permission to change an account or disclose another customer’s records.
Measure the cost of supervision
An agent that completes tasks quickly can still create an expensive operation if employees spend hours investigating its actions or repairing its mistakes.
Evaluation should include exception volumes, correction effort, unauthorised-action attempts and the time required to stop a faulty workflow. Commercial results matter too: did the process reduce cost, improve service or release useful staff capacity after supervision and rework were included?
Authority can then expand in stages. A business might start with recommendations, move to preparation of actions for approval, and permit limited execution once the evidence supports it. Changes to the model, connected tools or underlying process should trigger a review of that evidence.
Rehearse recovery before deployment
Someone must own the response when an agent behaves unexpectedly. That person needs a tested way to suspend its access, identify affected transactions and coordinate corrections.
Recovery cannot always mean undoing an action. A confidential message cannot be made unread, and a missed delivery deadline cannot simply be reversed. Those consequences should influence which actions remain subject to human approval.
The useful question for executives is whether the business can demonstrate control under realistic conditions. A successful demonstration shows what an agent can do. A failure test shows whether the organisation is ready to let it do it.
