01
a chat message decided where a recovery link went
In March 2026 Meta launched an AI support assistant for Instagram. According to Meta's later disclosure, reported by Help Net Security, attackers used it to take over 20,225 accounts: they connected through a VPN near the victim's location, asked the assistant to link the target account to an email address they controlled, and the system sent a password reset link to that new address instead of rejecting the request. Accounts without two-factor authentication were lost. Meta says it found the flaw on 31 May, although a filing with the Maine attorney general lists 17 April as the incident date, and it disabled the tool and invalidated the reset links. Meta's explanation is that the assistant worked as intended and a bug in a separate code path failed to check that the email matched the one on the account.
Take Meta's account at face value and the lesson is the same. Somewhere in that system, a sentence typed into a chat window turned into a change to an account's recovery path, and the only thing that should have stood in the way was one check in one code path. If you're wiring an assistant into anything that writes (billing, permissions, orders, account settings), this is the failure to design against, and it doesn't need a clever prompt injection, only a request that sounds like normal support.
The fix is structural. A tool call from a model is a proposal written in JSON. It becomes an action only after your code has validated it, checked it against policy and current state, and, for the risky ones, had the right person confirm it on a screen the model doesn't control.
02
every call goes through a deterministic proxy
The model should never hold credentials for your billing API or your database. Put a thin proxy between the model's tool calls and the real systems, with one handler per tool and nothing clever in it. Each handler parses the arguments against a strict schema, checks that the user on whose behalf the assistant is acting may perform this operation on this specific resource, applies rate limits, and maps upstream errors onto a small internal vocabulary. Authorization uses the end user's identity and permissions, never a service account the assistant shares across all users, because otherwise every user can do whatever the assistant can.
The proxy is also where you get an audit trail that answers questions later. A log record per call with fields like these is enough to reconstruct what the model asked for, what your code allowed and what the system then did:
Keep the outcomes a short, finite list that the UI, support and analytics all share, for example done, needs_review, deferred, blocked_by_policy and upstream_unavailable. A user who sees "waiting for approval" understands what's going on, while a free-form agent trace where step seven was skipped doesn't help anyone. And when validation fails, the fallback is one of those states, not "ask the model to try harder". Retrying a transport error makes sense, but retrying because the model produced nonsense mostly burns tokens.
03
decide by risk which calls need a person
If every write needs approval, people approve without reading, so the policy has to sort tool calls by what they can break. Reads that the user could already see in the UI run immediately. Small, reversible writes in the user's own scope (a display name, a notification setting) get an inline confirm in the same session. A short list of operations gets the full treatment: anything touching identity or account recovery (email, password, MFA devices), roles and permissions, API keys, money above a threshold, and deletes. In most products that list is short and the operations on it are rare, so the friction lands where it belongs.
For that list, two rules come straight from the Meta case. The confirmation goes to the session that owns the resource, not to whoever is typing in the chat, so an attacker who talks the assistant into proposing a recovery email change produces a request that only the real owner can confirm. And the owner confirms with a fresh factor (a passkey or TOTP prompt), because a stolen session cookie shouldn't be enough to change recovery settings. Operations on other people's data, like a refund or a plan change in a B2B admin tool, go to an approver with a role for that specific tool, and the requester can't approve their own proposal.
These rules live in code next to the tool definitions. A line in the system prompt saying "always ask before changing billing" is a request to the model, and anything the model can be asked to do, it can be talked out of.
04
the queue is a postgres table
Proposals that need approval wait in a table, and Postgres is all you need for it. The tempting design with Redis Streams doesn't work as a source of truth: a stream is an append-only log, so you can't update an entry's status after XADD, and a consumer group hands out entries to consumers without any way to filter by field, so "give me approved entries" isn't something it can do. Mirroring state between Redis and Postgres in the same request handler is a dual write that will disagree with itself after the first crash. One table, with status as a column and row locks for the workers, gives you all of it in one transaction:
The statuses are deliberately more than approved and rejected. failed (approved, then the execution broke) is a different event from rejected (a person said no), and stale (the resource changed after the proposal was made) is different again. Every proposal also records resource_version, the version of the row it was made against, which is what lets the executor notice that someone edited the record in the meantime.
Approval is one guarded UPDATE, so the permission check, the state transition and the expiry check can't drift apart:
No row returned means the approval didn't happen, whether because someone else decided first, the proposal expired, or this user isn't allowed to approve this tool. This is the approver path for actions on other people's data. Self-service actions on the high-risk list use a separate confirm route that requires the resource owner's own session and a fresh factor, and that route sets the same approved status. Approving doesn't execute anything. A worker picks up approved rows with FOR UPDATE SKIP LOCKED, so several workers can run without blocking each other, and when the target lives in the same database, the change and the status update commit together:
The version = %s condition is the optimistic lock. If a colleague changed the subscription by hand between proposal and execution, the update touches zero rows, the proposal ends up stale, and nobody's manual edit gets overwritten by an approval that was based on old data. A periodic job moves anything past valid_until to expired, and if you want workers to react instantly instead of polling every second or two, NOTIFY after the approval commits and LISTEN in the worker.
When the action is an external API call, you can't put it in the same transaction. Commit the move to executing first, call the API with the proposal id as its idempotency key, then record executed or failed. A row stuck in executing after a crash is then something a reconciliation job can resolve by asking the upstream API what happened to that key, instead of a mystery.
05
approvers see a diff your server rendered
The approval card is a security boundary, so the model doesn't get to write any of it. It's tempting to ask the model for a friendly one-line summary of the action and show that to the approver, but then the approver confirms whatever the model says it's doing, and a manipulated model can describe "change the recovery email to the attacker's address" as "update contact preferences". The card has to be built by your server from the validated arguments and the current state of the resource:
Each tool declares which fields the approver sees, and each field shows its current value next to the proposed one, read from the database at render time. Everything is escaped, because customer names and free-text arguments are user input that may have passed through the model. If you show the conversation that led to the proposal, put it below the diff and label it as context, so nobody mistakes the chat for the thing being approved.
When the model returns several calls for one request, show them as one card with every step and approve the chain as a unit. Approving three dependent steps separately leaves you with half-executed sequences when someone approves the first and rejects the second.
06
the parts that bite later
Approval queues that nobody watches become slow execution queues, so notify the approver when a proposal arrives, remind them before valid_until, and tell the requester what happened either way. Keep valid_until short for anything where state moves quickly, measured in minutes for account changes and hours at most for back-office edits.
Rate-limit proposals per session for the high-risk tools. A real user doesn't try to change their recovery email nine times in two minutes, and a burst of proposals that the policy rejects is a strong sign someone is probing the assistant. Log rejected and expired proposals as carefully as executed ones, because that's the record you'll want when you look for the attempts before a successful one.
Finally, test the queue the way you test payments: propose against a row, edit the row, approve, and check that the result is stale; approve as the requester and as a user without the role, and check that nothing happens.
