Test Design

Mutation Tests Are Negative. TDD Tests Are Positive.

Both get called tests, but they make opposite kinds of claim. A TDD test asserts what the system must do. A mutation run can only ever report what your suite failed to notice. Knowing which polarity you are holding decides what to do with the result — and it decides whether you can trust a test an agent wrote.

Review Our Test Strategy View QA Automation Services

Two Kinds of Claim

A test-driven test is written before the behaviour exists. It fails, then it passes, and from then on it is a standing statement: this input must produce that outcome. The suite accumulates into a specification you can read, and the act of writing it first puts pressure on the design, because code that is hard to call is hard to test. A mutation run does something structurally different. It takes code that already works, changes it on purpose, and reruns the suite. Every result is phrased in the negative: this alteration went unnoticed. Nothing in that output says what the software is for.

A green TDD suite is a set of statements about intent. A high mutation score is the absence of a particular kind of evidence against your suite. Only one of those two things is a specification.

The Positive Test

+ Written before the code, so the requirement exists in words before it exists in logic
+ Fails first, which is the only cheap proof that the test can fail at all
+ Names a rule, so it survives refactoring: the internals can move without the assertion moving
+ Applies design pressure while the design is still cheap to change
+ Readable as documentation by the next person, including the next agent

The Negative Check

− Runs after the fact, on code and tests that already exist
− A killed mutant conveys no new information; only survivors carry signal
− A survivor is evidence of blindness, never evidence of a defect in the product
− Says nothing about missing requirements, only about insensitivity to the changes it tried
− Produces a report and a worklist, not an artifact anyone keeps

The Same Word, Two Different Instruments

	The Positive Test	The Negative Check
What it claims	The system must behave like this.	Your suite did not notice this change.
What green proves	Every behaviour specified so far is implemented.	The suite is sensitive to that operator set. Nothing about correctness.
What it leaves behind	Tests, and a design shaped by having to write them.	A report. The value has to be extracted before it expires.
When it runs	Continuously, minutes at a time, while the code is being written.	In CI: incrementally on changed files, or as a nightly sweep.
Cost model	Paid steadily and amortised into development.	Mutants multiplied by relevant test runtime. Grows with both.
Effect on design	Testability pressure. An awkward design hurts immediately.	None. It grades whatever already exists.
How failure arrives	Red, on purpose, authored by you, one step at a time.	A survivor nobody wrote, which now has to be interpreted.

How to Actually Run It

Mutation testing has a reputation for being unaffordable, which is almost always a scoping problem rather than a tooling problem. This order of operations keeps it cheap enough to survive contact with a delivery schedule.

Get coverage in place first

- Mutation testing on uncovered code just reports the obvious at great expense
- Run coverage as the cheap continuous gate and keep mutation for code that is already covered
- Pick one module that matters — payments, permissions, pricing — rather than the whole repository

Scope every run

- Per-test coverage analysis is the single biggest lever: it runs only the tests that reach each mutant
- On pull requests, mutate changed files only, in incremental mode; that is a run measured in minutes
- Keep the exhaustive sweep on a schedule, off the critical path, where a long run costs nobody anything

Triage survivors into three buckets

- A missing requirement, which becomes a new positive test named after the rule
- An equivalent mutant that changes nothing observable, which gets documented and excluded once
- Code no requirement asks for, which gets deleted — the cheapest possible kill

Gate on regression, not on a number

- Set a break threshold that stops the score sliding, and stop there
- Report survivors as review items rather than build failures, so nobody learns to ignore a red build
- Track wall-clock per run as a first-class metric; a gate that gets disabled for being slow protects nothing

What This Changes When an LLM Writes the Code

Agents did not create this problem, but they industrialised it. The polarity distinction stops being philosophy the moment your tests and your implementation have the same author and that author is a model.

Ask an agent for a feature with tests and you get both, generated from one reading of the requirement, in one context. If that reading was wrong, the tests encode the same misunderstanding and pass — and coverage looks excellent, because every line the model wrote is exercised by a test the model wrote to exercise it. This is the failure that ordinary review is worst at catching, because the tests look reasonable in the diff. Mutation testing is the cheapest automated check that catches it, for one reason: it does not ask whether tests exist, it asks whether they react. A test that mirrors the implementation will not notice the implementation changing.

Put it in the loop, not in a report

A surviving mutant is unusually good feedback for an agent: it is executable, specific, and impossible to rationalise away. A paragraph of review advice gets acknowledged and ignored; a named change that went undetected gets fixed. This is exactly the shape of Uncle Bob's SwarmForge hardener role, which runs the mutation tool file by file and is not allowed to advance until survivors are dealt with.

Move the criterion into the writing instruction

The best line in that project is in its coder role prompt: write tests that would fail for a plausible wrong implementation. That is a mutation criterion embedded in the generation step, which is far cheaper than discovering the same thing in an audit an hour later. Put that sentence in your own agent instructions.

Separate the author from the hardener

The same model, in the same context, will explain why a survivor is fine, because the reasoning that produced the gap is still in front of it. Run the mutation pass as a separate step with a separate context and no access to the original rationale — only the diff and the tool output.

Expect score-chasing at machine speed

Told to raise the mutation score, an agent will write mutant-killing change detectors faster than anyone can review them. Goodhart's law is worse with agents because volume is free. Let agents report survivors and propose requirement-level tests; keep a human, or a specification role, on approving what the requirement actually is.

Evaluating Code an Agent Just Produced

Coverage and mutation score together are a usable rubric for AI-written work. Read them as a pair — the interesting information is in the disagreement.

Signal	What it usually means	What to do
High coverage, low mutation score	Tests were written to execute the code, not to check it. The classic shape of model-generated tests.	Keep the implementation under review, discard or rewrite the tests at requirement level.
Both high, on a small diff	Genuinely good work, or well-disguised change detectors.	Spot-check for spies, snapshots and call-count assertions. If the assertions name rules, ship it.
Survivors clustered on error paths	The model implemented the happy path properly and narrated the rest.	Specify the failure behaviour explicitly, then ask for tests on that specification.
Survivors in code no requirement names	Speculative generality — options, flags and defensive branches nobody asked for.	Delete the code. It is untested complexity with no owner.
Score rose, tests now touch internals	The agent optimised the metric by welding the suite to today's implementation.	Reject. The next refactor will fail these tests while behaviour is unchanged.
Equivalent-mutant exclusion list growing fast	The agent is arguing with the tool rather than improving the suite.	Read the diff yourself. Exclusions are a human decision, made once, with a reason recorded.

The Same Survivor, Two Responses

This is where the polarity stops being philosophy. A mutation report hands you a location; what you write next decides whether the suite gets stronger or just stiffer.

The code, and the mutant that survives

Removing .trim() is a standard method-call mutation. It survives whenever no fixture happens to carry surrounding whitespace, which is most fixtures — and fixtures a model invented for its own code are no exception.

// importer.js
const normalise = (value) => value.trim().toLowerCase();

export function importRows(rows) {
	return rows.map((row) => ({ email: normalise(row.email) }));
}

// Surviving mutant: normalise() with .trim() removed.
// The suite passes either way, so the report flags it.

Killing the mutant, and coupling the suite

This test kills the survivor. It also freezes the current implementation: normalise has to stay reachable and stay called exactly this many times. Rename it, inline it, or move it behind a boundary and the test breaks while behaviour has not changed at all. An agent optimising for score produces this shape by default.

it('calls normalise once per row', () => {
	const spy = vi.spyOn(internals, 'normalise');
	importRows([{ email: ' Ada@Example.COM ' }]);
	expect(spy).toHaveBeenCalledTimes(1);
});

// Green. Mutant dead. Score up.
// Nothing here states what an imported email address should look like.

Answering the question the survivor asked

Same mutant, same kill, but the assertion is a rule a product owner could read. It goes through the public surface, so the internals stay free to move, and it explains itself in the failure message six months from now.

it('trims and lower-cases every imported email address', () => {
	expect(importRows([{ email: ' Ada@Example.COM ' }]))
		.toEqual([{ email: 'ada@example.com' }]);
});

// Same mutant dead, but the suite gained a specification
// instead of a snapshot of today's call graph.

Keeping the audit cheap enough to keep running

json

Per-test coverage analysis is the single largest performance lever: it runs only the tests that actually reach each mutant. Incremental mode keeps a pull request run in minutes. The break threshold is a floor against regression, not a target to chase.

{
	"testRunner": "vitest",
	"coverageAnalysis": "perTest",
	"incremental": true,
	"mutate": ["src/**/*.js", "!src/**/*.test.js"],
	"thresholds": { "high": 80, "low": 60, "break": 60 }
}

The gate for an agent-authored branch

Two independent questions, asked by a step that did not write the code: is this code shaped like something you can change, and do the tests protecting it actually react when it changes? Both answers go on the pull request, and neither is negotiable by the author.

CHANGED=$(git diff --name-only origin/main... -- 'src/**/*.js')

# structural: complexity against coverage on touched methods
crap-report --changed-only --threshold 30 $CHANGED || exit 1

# behavioural: do the new tests notice anything?
stryker run --incremental --mutate "$CHANGED"

# survivors are review items with a named requirement attached,
# never a licence to write a test that pins the call graph.

Where the Negative Turns Toxic

Chasing the score

Once the mutation score is a target, tests get written to kill mutants rather than to state requirements. They pass review because they are green, and they are the ones that break on the next refactor even though nothing about the behaviour changed.

Fighting equivalent mutants

Some mutations change the code without changing observable behaviour. No honest test can kill them. Document them, exclude them, and move on. Time spent here buys a number, not confidence.

Asserting the mutation, not the rule

If you cannot name the requirement a new test protects, you have written a change detector. It will report every future edit as a failure and teach the team to stop reading test output.

Letting the author harden its own work

A model that wrote the code will find a reason each survivor is acceptable, because its own reasoning still looks sound to it. Hardening has to happen in a step that only sees the diff and the tool output.

Turn Every Survivor Into a Positive Statement

Read the survivor as a question

- What behaviour would have to be true for this change to break something?
- Which requirement, if anyone had written it down, would fail right now?
- Who downstream would notice if this mutation shipped?

Write the answer, not the kill

- Name the test after the rule, never after the mutant or the line number
- Assert through the public surface so the internals stay free to change
- If the rule can only be reached by exposing internals, that is a design finding, not a testing one

Or delete the code

- If no requirement can be named, nothing actually depends on the behaviour
- A survivor in unspecified code is often a dead requirement rather than a missing test
- Deleting it is the cheapest possible kill, and it makes the next audit faster

One Writes the Spec, the Other Audits It

Mutation testing is not better TDD, and it is not a replacement for it. It produces no tests, no design, and no statement of intent. What it produces is a list of places where your specification is less sensitive than you assumed, which is genuinely valuable and genuinely different.

That division of labour matters more now than it did when both jobs belonged to the same engineer. Tests should still come into existence through TDD, as positive claims about behaviour — including when an agent writes them, which is why the instruction to write tests that would fail for a plausible wrong implementation belongs in your prompts. Mutation runs then audit those claims from a separate context that cannot be argued with. And no survivor goes straight into a test: translate it into a requirement first, or delete the code it lives in. A negative finding only becomes durable value when someone turns it into a positive statement about what the software is for.

FAQ

Is mutation testing better than code coverage?

It answers a stronger question at a much higher price. Coverage counts whether a line executed; mutation asks whether anything actually asserted on it. Use coverage as the cheap continuous gate and mutation testing as a periodic audit on code that matters — and as the standing check on anything an agent wrote.

How does mutation testing help with AI-generated code?

It catches the specific failure that review misses: tests generated from the same misunderstanding as the implementation, which pass and produce excellent coverage while checking nothing. Mutation testing does not care whether tests exist, only whether they react to the code changing, so a suite that mirrors the implementation is exposed immediately.

Should agents run mutation testing themselves?

Yes, as a separate step from the one that wrote the code, and with the survivors treated as questions rather than a score to maximise. Let the agent report survivors and propose requirement-level tests; keep the decision about what the requirement is with a human or a specification role, or you will get change detectors at machine speed.

What mutation score should we aim for?

Scores above roughly 80 percent are commonly treated as strong and 60 to 80 percent as workable with real gaps, but the number only means something per module. A more useful rule is no regression on changed files, with a break threshold that stops the score sliding.

Which tools do teams use?

PIT is the established choice on the JVM, Stryker Mutator covers JavaScript, TypeScript, C# and Scala, and mutmut and cosmic-ray are the usual options in Python. All of them support scoping a run to changed files, which is what makes the practice affordable on a pull request.

Where should mutation testing sit in CI?

Incrementally on pull requests, scoped to changed files with per-test coverage analysis, plus a full sweep on a schedule. Report survivors as review items rather than build failures, and gate only on the score regressing.