teeth
Do your tests catch bugs? teeth puts small bugs in your code and counts.
teeth is a command-line tool and a Pi extension. It changes your TypeScript code in small ways, for example < to <= or && to ||. It plants 8 kinds of bug. For each bug, it runs your real test command. The share of bugs that your tests catch is the bite rate.
Exit codes: 0 the tests have teeth. 1 the tests are toothless. 2 the tests fail on the unchanged code.
Verdict disproven
The idea that we tested: put teeth in the agent loop, so that an agent cannot say "done" until its tests catch bugs. The result does not support this idea.
We used gleam-agents with Claude Haiku 4.5. Each arm had the same 34 seats. Hidden tests graded each seat. (recomputed from the per-seat receipts)
| Arm | Hidden tests pass | False "done" | Cost |
|---|---|---|---|
| off | 25 | 9 | $17.80 |
| placebo: one line of text, no bugs planted | 29 | 5 | $15.60 |
| on: teeth runs and blocks "done" | 31 | 3 | $21.09 |
The free placebo gets 4 of the 6 extra passes, at a lower cost. The real check adds 2 more passes and costs $5.49 more than the placebo. A "false done" is a seat that said "done" and then failed the hidden tests.
Use teeth as a scanner for tests that you already have. Do not put it in an agent loop.
What a scan finds
teeth scanned the real test suites of 58 TypeScript repos. It cost $0 and used no model.
- 901 bugs planted. The tests missed 481 (53%).
- 41 of 82 files have tests that catch fewer than half of the bugs.
- 26 files have tests that catch none.
We checked 2 misses by hand. In a phone relay, any POST reaches the publish handler, and all 9 tests pass. In a game world, the third creature appears in the wrong corner, and all 56 tests pass.
The scan does not remove equivalent mutants. These are changes that no test can see. In an earlier set that we labelled by hand, 29% were equivalent. Thus the real share of toothless tests is lower than the numbers above.
A real agent run
In this Pi run, the tests pass and teeth says toothless. The agent writes new tests. teeth says the tests have teeth. Then the agent says "done".
Use it
git clone https://github.com/acoyfellow/mutant && cd mutant && cargo build --release export PATH="$PWD/target/release:$PATH" git clone https://github.com/acoyfellow/teeth && cd teeth node bin/teeth.mjs check <repo> --files src/a.ts --test "npm test --silent"
teeth uses mutant to plant the bugs. You need Node 22.6 or later.
Limits
- Only 8 kinds of bug: boundaries,
&&/||,===/!==and returned booleans. - Equivalent mutants are not removed. Some hints ask for a test that cannot exist.
- It is slow: about one full test run for each planted bug.
- TypeScript only.