01
green ci means something different now
A CI pipeline was built around a quiet assumption: the person who wrote the code understood the system around it, and wrote tests aimed at the parts they weren't sure about. When a model writes both the implementation and the tests in one session, that assumption is gone. The tests come from the same picture of the system as the code, so they agree with it by construction, and a green build mostly tells you the two halves are consistent with each other. If your team merges a lot of assistant or agent output into Go or Python backends, this is the gap you're exposed to.
The surveys are consistent with that, though they're all vendor-run and should be read that way. CloudBees' 2026 State of Code Abundance report, a survey of 213 enterprise technology leaders run by TrendCandy with a stated margin of error of plus or minus 8 points, found that 92% expressed confidence in the production readiness of AI-generated code while 81% reported an increase in production issues tied to it. Lightrun's 2026 State of AI-Powered Engineering report, a survey of 200 SRE and DevOps leaders at director level and above, says 43% of AI-generated code still needs manual debugging in production after passing QA and staging. That's a survey answer, not telemetry. CodeRabbit's December 2025 analysis of 470 open source pull requests (320 it classified as AI co-authored, 150 as human-only, with authorship inferred from signals rather than confirmed) found 10.83 issues per AI PR against 6.45 for human PRs, roughly 1.4 to 1.7 times more critical and major findings, logic and correctness issues 75% more common, and error-handling gaps nearly twice as common.
That last category tells you where to look. The failures cluster where the code meets something it didn't write: an upstream API that sometimes returns null, inputs nobody thought of, and behaviour that slowly diverges between two generated patches that each looked fine on its own. Adding more of the same checks doesn't reach those places. What follows are four checks that do, ordered roughly by how cheap they are to adopt.
02
ask for the assumptions before the diff
The cheapest check happens before any code exists. For a non-trivial change, ask the model to write down every assumption the implementation will depend on, and have a person read that list before reading the diff. It's far easier to spot "assumes the provider never returns a payment without settled_at" in a ten-line list than to find the same assumption buried in three hundred lines of plausible code.
A prompt along these lines works with any assistant, and the output belongs in the PR description, where a reviewer sees it first:
The "verified or guessing" column does most of the work. Guesses about upstream behaviour are exactly where the contract tests in the next section come from, and a guess about rounding or time zones tells you which function needs a property test. Even when the list holds no surprises, reading it takes a minute and reviewing the code afterwards gets faster, because you know what the author believed.
03
contract tests at the seams
The integration boundary is where generated code fails most silently, because the model has only ever seen the happy example in the docs. A contract test pins the real shapes down in the repository: a fixture file for each response the upstream actually sends, including the ugly ones, and a test that the boundary model accepts or rejects each of them on purpose. Once those fixtures exist, every future generated change has to pass through them, whether or not the model remembers the constraint.
In Python, a strict pydantic model at the boundary does the parsing, and anything inside your code only ever sees the validated object:
The fixtures are plain JSON files captured from the provider's sandbox or from logged production responses with personal data removed: a settled payment, a pending one where settled_at and failure_code are explicitly null, a failed one carrying a field you don't use yet, and one where the amount arrives as a string. The test says which shapes you accept and which you refuse to guess about:
strict=True is what makes the string amount fail instead of being quietly coerced to an integer, and extra="ignore" keeps a new provider field from breaking you. Those are decisions a person should make once, deliberately, and then keep in the repo. For seams between your own services owned by different teams, Pact formalises the same idea: the consumer publishes the interactions it relies on and the provider's build verifies them. Within one team, fixture files and a test like this get you most of the value without a broker.
04
properties, fuzzing and mutation testing
Example tests check the cases the author imagined, which for generated code means the cases the model imagined. Property-based tests flip that around: a person states an invariant that must hold for every valid input, and the framework generates inputs trying to break it. Here's a function that splits an amount in cents into equal parts, written the way an assistant plausibly would, along with the two example tests that came with it and passed:
The property test states what fairness means for money: as many shares as requested, they add up to the total, none is negative, and no two differ by more than a cent.
Hypothesis found two counterexamples in under a second and shrank each to the smallest input that fails. split_cents(3, 5) returns [1, 1, 1, 1, -1], a negative share, and split_cents(2, 4) returns [0, 0, 0, 2], because Python's round rounds halves to even and round(0.5) is 0. Neither case is exotic, and neither would come up in a test someone wrote by hand while believing the function was right. The fix is integer arithmetic:
Go has the same capability built in since Go 1.18. A fuzz target is a test function that takes *testing.F, the seed inputs run on every go test, and go test -fuzz=FuzzTruncate keeps mutating them. This one checks a function that shortens a string to a byte limit:
Against the naive return s[:maxBytes], both seeds pass, including the Czech one, because cutting "Příliš" at 3 bytes happens to land on a character boundary. The fuzzer needed well under a second to find "řÚ" with a limit of 3, which cuts the two-byte Ú in half and returns invalid UTF-8. It wrote the input to testdata/fuzz/FuzzTruncate/, which you commit, so the case runs as a regular test from then on. The fixed version backs up to the start of a rune:
Mutation testing answers a different question: whether your tests would notice if the code were wrong. A tool such as mutmut for Python, or Stryker for JavaScript and TypeScript, makes small changes to the code (flipping <= to <, // to /, nudging constants) and reruns the tests; a mutant that survives is a behaviour change no test cares about. That makes it a direct check on tests that merely mirror the implementation. Take a late fee function:
With two example tests, one 10 days late and one 5 days early, mutmut 3.8 reported three survivors: days_late < 0, days_late <= 1, and amount_cents * 2 / 100, which would quietly return a float. Each survivor maps to a test somebody should write: what happens on day zero and day one, whether the result is still whole cents, and whether the cap holds. After adding three such tests all eleven mutants were killed. Mutation runs are slow on a large codebase, so in CI we'd restrict them to files changed in the PR.
05
shadow traffic without side effects
Tests can't show you the distribution of real requests, so the last layer runs the new version against live traffic without serving its answers. Envoy does the mirroring at the proxy: a route sends each request to the primary cluster and a copy to the candidate, and in Envoy's words it's "fire and forget", so it never waits for the shadow's response before answering the client. This config mirrors 10% of GET requests on one path and nothing else:
Running this on Envoy 1.40 in Docker with 100 GET requests, 7 reached the candidate (it's a random fraction), a POST to the same prefix went only to the primary, and the candidate saw the host header as localhost-shadow:10099, because Envoy appends -shadow unless you turn that off. Both sides received the same x-request-id, which is what lets you pair the responses afterwards.
The part that people skip is side effects, and with payment code it's the part that matters most. A mirrored request is a real request. If the candidate writes to the production database, charges a card, sends an email or publishes to a shared queue, you've just done it twice. Only mirror read-only endpoints, or give the candidate its own database copy or replica and stub every outbound integration that changes state. The simpler alternative of copying requests to a goroutine inside the same service is worse on both counts, because it shares the process, the connection pool and the downstream credentials, so the shadow can add latency to real requests or crash them along with itself.
Comparing responses only works after normalisation, otherwise every pair differs by its timestamp. Each side logs its response body with the request ID, and a small comparator drops volatile fields and sorts lists before diffing:
On a pair where the line items come back in a different order and the timestamp differs, it reports only the real change, $.total_minor: 1000 -> 1001, plus the version field you'd add to your ignore list. Only sort lists whose order isn't part of the contract. A one-cent drift between two versions of an invoice endpoint is exactly the kind of behavioural change no unit test catches, because each version is correct by its own tests.
06
where to start and what this won't catch
If you add one thing this week, make it the assumptions list, because it costs nothing and tells you where the other checks are needed. Then write contract fixtures for your most expensive integration, and a property test for the function that does the most money or parsing work. Shadow traffic is the biggest lift and pays off on high-traffic read paths, while for write paths you'll need record and replay against an isolated environment instead.
None of this verifies that the code does what the business wanted. Every check here depends on a person stating something true about the world, whether that's a fixture, an invariant or the list of fields you're allowed to ignore. That's the point, since it's the human input the generated tests were missing, but it means the work moves to whoever owns those statements rather than disappearing.
