arrow_backRetour aux issues
harshitagrawal2O/tensorkit
#15
Débutant
Ouvrirarrow_forward
Débutant
Ouvrirarrow_forward
Débutant
Ouvrirarrow_forward
Mutation testing on `tensorkit/scalar.py`: 385 mutants, 88 survivors, and four real gaps
ecoDébutant
help wanted
no-implementation-needed
mutation-testing
descriptionDescription
Baseline run of the new `cosmic-ray.toml` against milestone 1. Posting the method and the
numbers so the next run has something to compare against, and so anyone auditing the suite
does not repeat the triage.
## Result
| | Count |
|---|---|
| Mutants generated | 385 |
| Killed | 268 |
| Survived | 88 |
| Incompetent (mutant could not run) | 29 |
| **Mutation score** | **268 / 356 = 75.3%** |
A surviving mutant is behaviour the test suite does not pin down. On a repository whose central
claim is that the tests *are* the specification, that number is the claim being audited.
## Triage of the 88 survivors
**55 are equivalent mutants in type annotations.** `Value | float` appears in six signatures,
and `from __future__ import annotations` means annotations are never evaluated at runtime — so
mutating `|` to `+`, `/`, `>>` and so on changes nothing observable. Not findings. Worth
recording so nobody triages them twice.
**4 more are equivalent or cosmetic:** `__neg__`'s `self * -1.0` where `*` becomes `/` (equal
for -1.0), a `__repr__` string-formatting mutation, and removing `@staticmethod` from a helper
that is only ever called on the class.
**The remaining 29 come down to four genuine gaps.**
### 1. No test exercised a backward pass with an incoming gradient other than 1.0
The largest one — it accounts for roughly half the real survivors and touches `__mul__`, `exp`,
`tanh` and `relu`.
When `out.grad == 1.0`, these are all the same number:
```
self.data * out.grad
self.data / out.grad
self.data ** out.grad
```
So mutating the multiply in *any* `_backward` survived. Every product in the suite was either
the root of the graph or fed an addition, and both hand exactly 1.0 to the multiply. The chain
rule *through* a product was never verified.
Closed by nesting a product inside a non-linear op, where the incoming gradient is not 1:
```python
a, b = Value(2.0), Value(3.0)
out = (a * b).exp() # d(out)/d(ab) = e^6, not 1
out.backward()
# a.grad == 3 * e^6, b.grad == 2 * e^6
```
### 2. `__radd__` was never reached
Every binary-operator mutation of its body survived, which can only happen if no test calls it.
`test_division_and_reflected_operators` covers `__rsub__` and `__rtruediv__` but never puts a
float on the left of a `+`. Addition being commutative, a wrong `__radd__` stays invisible in a
symmetric expression — it has to be asserted directly.
### 3. `__rsub__`'s forward value was never asserted
Only its gradient contribution was. `__rsub__` is `(-self) + other`; mutate the `+` to `-` and
the gradient is still -1, so the mutant lived. The forward value is what separates them:
`10 - 4` is 6, the mutant gives -14.
### 4. The relu subgradient at exactly 0 was untested
This is the one I would call an actual documentation defect. `Value.relu`'s docstring says:
> Convention here: the subgradient at 0 is **0** [...]
> Tests: tests/test_scalar_autograd.py::test_relu_backward
That test was parametrised over 2.5 and -2.5. At exactly 0 the forward value is 0.0 under both
`>` and `>=`, so only the gradient separates them — `>` gives 0.0, `>=` gives 1.0. The
docstring named a test for a convention that test did not cover.
## Fixed, in two passes — the first one was not enough
Seven tests added in `tests/test_scalar_autograd.py`. **All seven pass against the current
implementation.** This was a gap in the specification, not a bug in the code, which is the
distinction mutation testing exists to draw.
The first pass added four and I assumed that closed it. Re-running and comparing
**mutant-by-mutant rather than by aggregate score** showed otherwise: 6 flipped from survived to
killed, 0 regressed, and the `exp` and `tanh` survivors were completely untouched.
The mistake is instructive. `(a * b).exp()` gives the *product* a non-unit incoming gradient —
which is what killed the `__mul__` mutants — but it leaves `exp` at the r
Issues similaires
calkit/calkit
star53
Poids du dépôt moyen
VS Code extension should be robust to YAML parser errors
Seeing this error: ``` Failed to read calkit.yaml: YAMLParseError: A block sequence may not be used as an implicit map…
Python
bug
good first issue
fu351/Doberman-Core
star211
Poids du dépôt léger
dash: a manual Refresh control
The dashboard polls: `refreshStats()` (`src/doberman/dash/app.py:408`) every 5 s and `refreshPending()` (`:546`) every …
Python
enhancement
good first issue
fu351/Doberman-Core
star211
Poids du dépôt léger
dash: "Copy details" button on each pending-approval card
Each pending-approval card in the dashboard (`renderPending`, `src/doberman/dash/app.py:448-544`) shows the risk badge,…
Python
enhancement
good first issue