All postsEngineering

Redesigning code review for teams where agents open the pull requests

Tier review by what a change touches, enforce it with rulesets and CODEOWNERS, record provenance in commit trailers, and let revert rates move the line.

Johnny Unar10 min read

01

review is where agent output piles up

Once coding agents start opening pull requests, the review queue becomes the constraint. Most teams still run the process they set up when every PR came from a person who decided the change was worth making: two required approvals, one PR template, whoever is free picks up the next review. That process assumes PRs arrive at human pace and that each one carries an author who can answer questions about it. Agent PRs break both assumptions, and if you're the person whose approval sits on dozens of them a day, you've probably noticed.

The survey numbers point the same way. In GitLab's AI Accountability Report from June 2026, a Harris Poll survey of 1,528 developers and technology buyers, 85% agreed that AI has shifted the bottleneck from writing code to reviewing and validating it, and 43% said they can't reliably tell AI-generated code from human-written code in their own codebase. The first number is an opinion, not a measurement, but the second one is a concrete gap you can close, and it comes up again below.

Stripe's account of its Minions agents is the most useful public reference for what high agent volume looks like when it works. In February 2026 Stripe wrote that it merges more than a thousand pull requests a week that are entirely produced by minions, with no human-written code, and that humans review every one of them. The surrounding setup does a lot of the work: minions run in isolated devboxes cut off from production and the internet, a local lint step runs on each push in under five seconds, CI selects tests from a suite of more than three million, and a run gets at most two CI rounds before it goes back to a human. Review didn't disappear at Stripe, it became cheaper because the work was constrained before the PR existed. If your codebase has thin tests and unclear ownership, agents will multiply the review burden instead.

02

tier by what a change touches

The change that pays off most is to stop reviewing every PR the same way, and to decide how much review a PR needs by what it touches rather than who opened it. A copy fix in a display component and a change to the refund handler need different scrutiny whether a senior engineer or an agent wrote them. Tiering by author fails in both directions: waving agent PRs through because they're small, or blocking all of them because they're agents.

Write the tiers down as path patterns in the repository, where changes to them are themselves reviewed:

text
1high services/payments/*
2high services/auth/*
3high migrations/*
4high .github/*
5medium services/*
6medium web/app/api/*
7low web/components/*
8low docs/*
9low *.md

The first matching pattern wins, so the high-risk paths go on top, and anything that matches nothing is treated as medium, so a new directory doesn't silently land in the cheapest tier. Note that .github/ is high, since a PR that edits the tier file or the workflows could otherwise lower its own bar. A small script reads changed paths on stdin and prints the highest tier among them:

python
1import sys
2from fnmatch import fnmatch
3
4ORDER = {"low": 0, "medium": 1, "high": 2}
5
6
7def load_rules(path):
8 with open(path) as f:
9 return [line.split(maxsplit=1) for line in f.read().splitlines() if line.strip()]
10
11
12def tier_for(path, rules):
13 for tier, pattern in rules:
14 if fnmatch(path, pattern.strip()):
15 return tier
16 return "medium"
17
18
19def main():
20 rules = load_rules(sys.argv[1])
21 paths = [p for p in sys.stdin.read().splitlines() if p]
22 tiers = {p: tier_for(p, rules) for p in paths}
23 for p, t in sorted(tiers.items()):
24 print(f"{t:6} {p}", file=sys.stderr)
25 print(max(tiers.values(), key=ORDER.get, default="low"))
26
27
28if __name__ == "__main__":
29 main()

fnmatch lets * match across /, so services/payments/* covers nested files too. Feeding it web/components/Button.tsx and README.md prints low; add services/payments/refund/partial.go to the same change and it prints high, because one risky file makes the whole PR risky.

03

enforcing tiers on github and gitlab

A common plan is to have a bot add a risk:low label and let branch protection act on it. GitHub doesn't work that way: neither classic branch protection nor rulesets can condition on labels. Rulesets target branches by name pattern and then apply rules, so labels are only useful for people scanning the queue. The enforcement has to come from rules that understand paths.

On GitHub, rulesets now have that. Under "Require a pull request before merging" there's a Required reviewers setting that takes up to 15 teams, each with its own number of required approvals (0 to 10) and its own list of file patterns in .gitignore syntax, with ! for exclusions. A payments team can be required for services/payments/** with two approvals while nothing extra is required for web/components/**. It isn't available on user-owned repositories, because those have no teams. The older route, CODEOWNERS plus "Require review from Code Owners", still works, but any single code owner's approval satisfies it, so it gives you "someone from this group" rather than a count. Turn on "Dismiss stale pull request approvals when new commits are pushed" and the most-recent-push approval rule as well, because agents keep pushing after a human approves. GitHub also has a public-preview rule, on by default, that requires one extra approval for Copilot PRs not attributed to a person; it does nothing if the ruleset requires zero approvals.

The tier itself runs as a workflow you mark as a required status check. It labels the PR for visibility and turns on auto-merge only for small, low-tier PRs from the agent's bot account:

yaml
1name: risk-tier
2
3on:
4 pull_request:
5 types: [opened, synchronize, reopened]
6
7concurrency:
8 group: risk-tier-${{ github.event.pull_request.number }}
9 cancel-in-progress: true
10
11permissions:
12 contents: write
13 pull-requests: write
14
15jobs:
16 tier:
17 runs-on: ubuntu-latest
18 timeout-minutes: 5
19 steps:
20 - uses: actions/checkout@v7
21 with:
22 fetch-depth: 0
23 - name: Classify changed paths
24 id: classify
25 env:
26 BASE_SHA: ${{ github.event.pull_request.base.sha }}
27 HEAD_SHA: ${{ github.event.pull_request.head.sha }}
28 run: |
29 tier=$(git diff --name-only "$BASE_SHA...$HEAD_SHA" \
30 | python3 .github/scripts/risk_tier.py .github/risk-tiers.txt)
31 lines=$(git diff --numstat "$BASE_SHA...$HEAD_SHA" | awk '{s += $1 + $2} END {print s + 0}')
32 echo "tier=$tier" >> "$GITHUB_OUTPUT"
33 echo "lines=$lines" >> "$GITHUB_OUTPUT"
34 - name: Label and maybe queue auto-merge
35 env:
36 GH_TOKEN: ${{ github.token }}
37 PR: ${{ github.event.pull_request.number }}
38 TIER: ${{ steps.classify.outputs.tier }}
39 LINES: ${{ steps.classify.outputs.lines }}
40 AUTHOR: ${{ github.event.pull_request.user.login }}
41 run: |
42 gh pr edit "$PR" --remove-label risk:low,risk:medium,risk:high || true
43 gh pr edit "$PR" --add-label "risk:$TIER"
44 if [ "$TIER" = "low" ] && [ "$LINES" -le 200 ] && [ "$AUTHOR" = "our-agent[bot]" ]; then
45 gh pr merge "$PR" --auto --squash
46 else
47 gh pr merge "$PR" --disable-auto || true
48 fi

Auto-merge only fires once every requirement in the ruleset is met, so a PR that touches a path with required reviewers still waits for them; the workflow can't widen access, only skip waiting where the rules already allow it. Two details matter in practice. Auto-merge has to be enabled in the repository settings first. And GitHub's docs say events triggered by GITHUB_TOKEN don't create new workflow runs, so a merge performed with that token won't start the push workflow that deploys main. If you deploy on push, run the merge step with a GitHub App token instead. When merges start colliding, a merge queue tests PRs against the latest base before merging, and your CI workflows need to listen for the merge_group event or queued PRs will never get their checks.

On GitLab, Code Owners sections carry approval counts directly: a section written as [Payments][2] with its paths and owners requires two approvals from those owners, and ^[Docs] marks a section as optional. Code Owners approvals need the Premium or Ultimate tier.

One caveat before you enable any merge without a human approval. Many companies have change-management controls for SOC 2 or ISO 27001 audits that require a second person to approve production changes. An unreviewed auto-merge tier can conflict with those, so talk to whoever owns the controls first. It's often easier to keep one quick human approval on the low tier and spend the savings on the high tier.

04

record provenance in commit trailers

Six months from now, someone looking at an incident will want to know which commits came from an agent, which model, and which task queued the work. If you don't capture that when the commit is made, you won't reconstruct it later, which is the 43% problem from the GitLab survey. Git already has a convention for this: trailers, the Key: value lines at the end of a commit message used for Signed-off-by and Co-authored-by. The agent wrapper adds them with git interpret-trailers rather than string concatenation, so formatting stays valid:

bash
1printf 'Fix button focus ring\n' | git interpret-trailers \
2 --trailer "Provenance: agent" \
3 --trailer "Agent-Model: example-model-1" \
4 --trailer "Agent-Task: TASK-812" > msg
5git commit -F msg

Humans get Provenance: human from a commit template or a prepare-commit-msg hook, and anything without the trailer reads as unknown, which is itself worth tracking. Git can read the values back without any parsing on your side:

bash
1git log --format='%h|%(trailers:key=Provenance,valueonly,separator=)|%(trailers:key=Agent-Model,valueonly,separator=)|%s'

If you squash-merge, check that the trailers survive. GitHub's default squash message keeps the commit message only for single-commit PRs; for multi-commit PRs, set the repository's default squash message to "Pull request title and commit details", or have the wrapper write the trailers into the PR description and use "Pull request title and description".

05

let revert rates move the line

With provenance and tiers both recorded, you can decide where auto-merge applies using evidence instead of arguments. This script walks the first-parent history of the main branch, assigns each merged commit its provenance and its tier (computed from the files it touched, with the same rules file), and counts how many were reverted within 14 days:

python
1import re
2import subprocess
3import sys
4from collections import Counter
5
6from risk_tier import ORDER, load_rules, tier_for
7
8
9def git(*args):
10 return subprocess.run(["git", *args], capture_output=True, text=True, check=True).stdout
11
12
13rules = load_rules(sys.argv[1])
14fmt = "%H%x1f%ct%x1f%(trailers:key=Provenance,valueonly,separator=)%x1f%B%x1e"
15commits, reverted_at = {}, {}
16for entry in filter(str.strip, git("log", "--first-parent", f"--since={sys.argv[2]}", f"--format={fmt}").split("\x1e")):
17 sha, ct, provenance, body = entry.strip().split("\x1f", 3)
18 for target in re.findall(r"This reverts commit ([0-9a-f]{40})", body):
19 reverted_at[target] = int(ct)
20 if not body.startswith("Revert "):
21 files = git("show", "--name-only", "--format=", sha).split()
22 tier = max((tier_for(f, rules) for f in files), key=ORDER.get, default="low")
23 commits[sha] = (int(ct), provenance.strip() or "unknown", tier)
24
25total, reverted = Counter(), Counter()
26for sha, (ct, provenance, tier) in commits.items():
27 total[provenance, tier] += 1
28 if sha in reverted_at and reverted_at[sha] - ct <= 14 * 86400:
29 reverted[provenance, tier] += 1
30
31for key in sorted(total):
32 print(f"{key[0]:8} {key[1]:6} merged={total[key]:4} reverted={reverted[key]:3} rate={reverted[key] / total[key]:.1%}")

Run it from the repository root as PYTHONPATH=.github/scripts python3 .github/scripts/revert_rate.py .github/risk-tiers.txt 2026-01-01. The output is one line per combination of provenance and tier, with merged count, reverted count and rate.

The rule we'd use to widen auto-merge is simple. Turn it on for one low-tier surface with good test coverage, such as a component library. Compare the agent revert rate on that surface with the human revert rate on the same surface, never with a different one, because a component library and a billing service have different baseline rates. Widen to the next surface only when the agent rate is at or below the human rate, and only after enough merges that a couple of reverts don't swing the percentage; with a few dozen merges per cell, one revert moves the rate by several points, so look at the raw counts too. If the agent rate is higher, you've learned where your tests aren't protecting you, which is useful even if it isn't the answer you wanted. Reverts undercount problems that were fixed forward, so treat the rate as a floor.

06

ci capacity is part of the review system

Review tiers fix half the bottleneck. The other half is that agents will happily keep runners busy with full test suites on branches that never merge, and human PRs wait behind them. Order your pipeline so the cheapest checks fail first: lint and type checks in a minute or two, then unit tests, then the expensive integration suite only after those pass. A concurrency group keyed on the PR number with cancel-in-progress: true, as in the workflow above, stops a branch that gets pushed five times from running five pipelines. On a merge queue, the build concurrency setting caps how many merge_group runs are dispatched at once.

Stale agent PRs are the other leak. An agent PR nobody touched for a few days is rarely coming back, and it keeps getting rebased, conflicting and rerunning CI. A scheduled job closes them:

yaml
1name: reap-agent-prs
2
3on:
4 schedule:
5 - cron: "0 6 * * *"
6 workflow_dispatch:
7
8permissions:
9 pull-requests: write
10
11jobs:
12 reap:
13 runs-on: ubuntu-latest
14 steps:
15 - env:
16 GH_TOKEN: ${{ github.token }}
17 GH_REPO: ${{ github.repository }}
18 run: |
19 cutoff=$(date -u -d '3 days ago' +%Y-%m-%d)
20 gh pr list --state open --author "app/our-agent" \
21 --search "updated:<$cutoff" --json number --jq '.[].number' \
22 | while read -r pr; do
23 gh pr close "$pr" --delete-branch \
24 --comment "Closed after 3 days without activity. Reopen or re-queue the task if it's still needed."
25 done

app/our-agent is the author filter for a GitHub App's bot account; replace it with your agent's app slug. Three days is a judgment call, and the closing comment matters, because whoever queued the task should be able to see why the PR vanished and requeue it if the work is still wanted. If a person regularly reopens these, the threshold is too short or the agent is being given tasks nobody was going to review.

Thanks for reading.
Johnny Unar

Written by

Johnny Unar

Keep reading

Stop patching.Start building.

Tell us what's slowing you down. We'll bring coffee and opinions.