biome nursery/useConsistentTestIt: fix changes program behavior
Hypothesis Nemotron 3 Super predicted to fire fired 2×
order of evaluation when expressions are merged, duplicated or deleted (side effects run twice or never)
Program
Output
(no stdout) ReferenceError: test is not defined
(no stdout) ReferenceError: it is not defined
- fired
- 2 before · 0 after
- runs
- each version twice, same file name, no network
Judge Nemotron 3 Ultra advisory real · high
The rule assumes `it` and `test` are interchangeable globals provided by the test framework, but the program uses `test` as a local variable (likely an imported or defined function). The fix rewrites the local `test` calls to `it`, which is undefined in this scope, changing a working program into one that throws a ReferenceError.
Explanation Nemotron 3 Super unverified
The original program uses `test` as a function; rewriting replaces each call with `it`, which remains undefined in scope. Since both identifiers refer to non-existent functions, the behavior changes from one reference error (`test`) to another (`it`), altering the specific source of failure while preserving only the general class of error (ReferenceError). The substitution does not maintain semantic equivalence because it changes which identifier is accessed, affecting diagnostic output despite identical runtime outcome categories. The fix incorrectly assumes `test` and `it` are interchangeable, but they denote distinct bindings whose absence produces different error messages, violating semantic preservation. This constitutes a behavioral change observable via stderr content, even if both throw ReferenceError. No further assumptions about tooling or context justify the rewrite as safe under language semantics.
Reproduce
- file
prog.js(the before program)- minimized program
- r2-p008.js
- evidence sheet
- r2-p008.md
- regression fixture
- r2-p008.js the minimized program with a header stating the observed change, for the tool's test suite
node prog.js bunx @biomejs/biome lint --only=nursery/useConsistentTestIt --write prog.js node prog.js
Upstream search GitHub issues Tavily Nemotron 3 Super + Ultra
useConsistentTestIt fix · nursery/useConsistentTestIt array destructuring · useConsistentTestIt function call mapping · rule nursery/useConsistentTestIt test to it
- related biomejs/biome/issues/11453 Both involve useConsistentTestIt rewriting test to it, but the existing report concerns an imported binding while the new finding uses a bare global call.
Details
- program
- Order of Evaluation Side Effects
- hazard
- order of evaluation when expressions are merged, duplicated or deleted (side effects run twice or never)
- oracle
stderr before: ReferenceError: test is not defined stderr after: ReferenceError: it is not defined- run
- 2026-10-17T20-49-46-686Z-qbws-biome-nursery_useConsistentTestIt
generated program (4 lines) and the fix's diff
--- orig.js 2026-10-17 16:50:09.291304797 -0400 +++ fixed.js 2026-10-17 16:50:09.566310203 -0400 @@ -1,4 +1,4 @@ let counter = 0; const inc = () => ++counter; -[test('a', inc), test('b', inc)].map(([_, fn]) => fn()); +[it('a', inc), it('b', inc)].map(([_, fn]) => fn()); console.log('final counter:', counter);
rule documentation given to the models
Summary
- Name: useConsistentTestIt
- Fix: safe
- Default severity: warn
- Available from version: 2.4.11
- Diagnostic category: lint/nursery/useConsistentTestIt
Domains
- Name: test
- The rule is enabled when one of these dependencies are detected:
- jest@>=26.0.0
- mocha@>=8.0.0
- ava@>=2.0.0
- vitest@>=1.0.0
- The rule adds the following globals:
- after
- afterAll
- afterEach
- before
- beforeEach
- beforeAll
- context
- describe
- it
- expect
- run
- setup
- specify
- suite
- suiteSetup
- suiteTeardown
- teardown
- test
- xcontext
- xdescribe
- xit
- xspecify
Description
Enforce consistent use of `it` or `test` for test functions.
`it` and `test` are aliases for the same function in most test frameworks.
This rule enforces using one over the other for consistency.
Imported functions keep their original export through an import alias.
The fix is unavailable when the preferred name conflicts with another binding or global reference.
Examples
Invalid
```js,expect_diagnostic
test("foo", () => {});
```
Valid
```js
it("foo", () => {});
```
Options
`function`
The function to use for top-level tests (outside `describe` blocks).
Accepted values are:
- `"it"` (default): Enforce using `it()` for top-level tests
- `"test"`: Enforce using `test()` for top-level tests
```json,options
{
"options": {
"function": "test"
}
}
```
Invalid
```js,use_options,expect_diagnostic
it("foo", () => {});
```
Valid
```js,use_options
test("foo", () => {});
```
`withinDescribe`
The function to use for tests inside `describe` blocks.
Accepted values are:
- `"it"` (default): Enforce using `it()` inside describe blocks
- `"test"`: Enforce using `test()` inside describe blocks
```json,options
{
"options": {
"withinDescribe": "test"
}
}
```
Invalid
```js,use_options,expect_diagnostic
describe("suite", () => {
it("foo", () => {});
});
```
Valid
```js,use_options
describe("suite", () => {
test("foo", () => {});
});
```