Case study · Designed and built by me · AI

Overwatch

A control room for AI agents that can act on money and customer records.

I designed and built a working prototype that lets the right people approve, stop or recover risky agent actions.

Try the prototype

The action review for a duplicate €8,400 refund. Everything a reviewer needs sits on one screen.

Behind the work

My role: Product designer and builder, working alone
Type: Self-started 0-1 product, web app
Started: 1 September 2026
Status: Active
Built with: AI-assisted development with ChatGPT and Codex. I designed the product logic, journeys, permission model, review states and interface.
Users: Finance, operations, platform and audit teams
Data: Made-up billing scenarios. The audit trail is simulated and not stored permanently yet.

What I built

I built it, not just drew it.

Overwatch is a working product, not a mockup. I wrote the product brief, the roles, the policy rules and the failure cases. Then I built it with AI-assisted development and checked every result myself.

Product: The problem, the scope and what to leave out
Rules: Four roles, who may decide what, and the policies that route each action
Interface: Every screen, state and blocked state
Build: Developed the working prototype with AI-assisted coding in ChatGPT and Codex, then reviewed and tested every output
Testing: Role-play journeys, fixes, then a repeat pass
Live: Published on Cloudflare Pages, with source control on GitHub

Built with

Next.js and React
TypeScript
CSS
PDF.js for browser-based PDF text extraction
Lucide React for interface icons
Cloudflare Pages for public hosting
GitHub for source control
AI-assisted development using ChatGPT and Codex
Local synthetic data for product simulation

Document analysis is rule-based. No AI model is called while the prototype runs.

The problem

AI agents are starting to do things, not just answer questions.

They issue refunds, cancel contracts and update customer records. Most tools for agents are built around chat and background automation. That is fine while the stakes are low. It is not fine when one mistake costs real money.

Before anything runs, a team needs to know:

What is the agent proposing, and why?
What evidence supports it, and what is missing?
Which rule applies, and who is allowed to decide?
Which systems will change?
Could it run twice?

So the question I worked on: how can the right people control risky agent actions without watching everything the agent does? I designed the moment after an agent proposes an action. Tools for watching prompts and traces already exist. The gap is who decides, on what evidence, at what cost, and how to recover when something breaks.

How it works

Every action follows the same path.

AgentThe agent proposes an action.
PolicyA fixed policy decides who must approve it.
ReviewerA person with the right role reviews the evidence and the cost.
ExecutionApproved work runs once.
AuditThe result is recorded.

If a step fails, only the AI Platform Admin can recover it.

How evidence becomes a controlled AI action

Overwatch reads the available evidence, identifies decision signals, applies policy rules and creates a proposed action. Execution remains blocked until an authorised person reviews it.

Two ways in

Try it with a prepared case, or bring your own document.

A reviewer should be able to see the system work in two minutes, and a sceptic should be able to test it with their own material. So the prototype opens on a choice.

Guided scenarios

Three prepared cases: a €8,400 duplicate refund, a €3,900 cancellation with missing consent, and a €1,250 service credit. Each follows the full path to a human decision.

Your own document

Choose a PDF, TXT, CSV, MD or JSON file up to 8 MB. It is read in the browser, not uploaded or stored. Rule-based detection looks for signals such as refund, invoice, payment, consent, contract and incident, then builds a review item, keeps execution blocked and assigns the human role that must decide.

There is also a populated sample workspace to explore without starting a run.

Overwatch start screen offering three guided scenarios or analysing your own document locally — The start screen: three guided scenarios, or your own file.

Overwatch showing a review item ready from an uploaded document, with confidence, policy and the role that must decide — A review item built from a document. Execution stays blocked.

Four decisions

1. The AI advises. People decide.

The agent brings a proposal, its reasoning and a confidence level. A fixed, readable policy picks who must approve. In the prototype the interface enforces that; a real version would check it on the server. Rules a person can read are easier to explain to an auditor than hidden model logic.

Overwatch action review for an expired 2,100 euro credit, with agent reasoning at 89 percent confidence, the policy that applies and the three systems affected — The agent's reasoning, the policy that applies and the decision authority sit on one screen.

Decision 2

2. Limits show before the click.

If a role can't use a control, it is greyed out and says which role is needed. An error after the click comes too late.

Overwatch in read-only auditor mode, with Request evidence, Reject and Approve action switched off and a notice that decision controls are unavailable — The Read-only Auditor sees the same action with every decision control switched off.

Decision 3

3. Missing evidence stays missing.

Asking for evidence and receiving it are two separate states. Confidence stays at 74% until the source responds, then moves to 86%. The dashboard never suggests proof exists when it doesn't.

Overwatch action review for a 3,900 euro subscription cancellation with the notice Automatic execution blocked and 74 percent confidence — Automatic execution stays blocked until the source responds.

Decision 4

4. Cost first. Final means final.

Before an €8,400 approval, a dialog names the payment processor, the customer record and the internal ledger it will change. After approval, the buttons disappear. Each approved action gets one execution record and one idempotency key, so the same action can't run twice.

The Approve 8,400 euro dialog naming the payment processor, customer record and internal ledger, and stating the action can execute only once — The approval dialog names the money and the systems it will change.

Tested, broken, fixed

What broke, and what changed.

Five friends and colleagues tested the interactive prototype by taking on the roles of Finance Approver, Operations Reviewer, AI Platform Admin, Compliance Auditor and Finance Director. Their journeys were task-based and moderated. The findings informed the interface changes, followed by a repeat regression pass to verify the fixes.

This was informal role-play, not research with real customers. It does not prove that outside users would succeed.

Who	What broke	What changed
Finance approver	Expired approvals looked like normal pending work.	Expired items come first and stay blocked until reassigned.
Finance approver	Asking for evidence marked it as verified.	Requesting and receiving are separate. Confidence stays at 74% until the source responds.
Operations reviewer	Anyone could retry or roll back a failed run.	Only the AI Platform Admin can, and the reason is shown.
Finance director	The same €8,400 action could run twice.	Final-state controls and one idempotency key.

Show moreShow less

AI platform admin	Pausing an agent was instant and left no record.	An impact warning and an audit entry.
Compliance auditor	Decision buttons looked usable, then failed after the click.	They are disabled in read-only mode.
Finance director	The overview showed counts but no money.	It now shows value awaiting review and value protected.

The product

Two dashboards carry the story.

Overwatch action review queue with a new review item created from an uploaded document

Overwatch execution monitor showing completed updates and a partial failure with the failed system named

Overwatch overview with value awaiting review and value protected, labelled as synthetic data

Overwatch policy rules with each rule's name, version and owner

Overwatch agent controls with status, scope and pause or resume

Overwatch audit history listing each decision with actor, role and time

What is real

Real, simulated, not built yet.

Audit trail simulation: functional
Persistent backend storage: planned

Real

Role switching, decision guards, evidence states, confirmation dialogs, an audit trail for decisions, permission blocks and execution events, a responsive layout, and browser-based reading of your own file with rule-based signal detection and policy routing.

Simulated

Customers, amounts, agents, policies, connected systems, execution steps, confidence scores that start from a baseline and rise with recognised signals, and the audit history shown in the dashboard.

Not built yet

Sign-in and server-side role checks, persistent audit storage on the public deployment, real CRM or billing connections, live AI model calls, and separation of client data.

The prototype generates an audit trail for decisions, permission blocks and execution events. Persistent production storage is designed into the architecture but is not yet enabled on the public deployment.

What I learned

What I learned, and what's next.

The hardest design moments are the unclear, late, blocked or broken ones.
Showing a limit before someone reaches it works better than explaining it afterwards.
Saying clearly what the project can't prove makes the rest more believable.

What's next

Test with five people in relevant roles. Success means most can explain a stopped action quickly and recover a failed update without help.
Connect the audit trail to persistent storage, add sign-in and check roles on the server.
Connect one system read-only with sandbox data, then add one write connection with duplicate protection.
Link every AI claim to the exact passage in the document it came from.
Run a small pilot on one workflow, such as high-value refunds, and measure review time, mistakes blocked and recovery time.

No business results will be claimed until a pilot produces them.

Try the prototype