Aller au contenu
login
arrow_backRetour aux issues
harshitagrawal2O/tensorkit #15

Mutation testing on `tensorkit/scalar.py`: 385 mutants, 88 survivors, and four real gaps

ecoDébutant help wanted no-implementation-needed mutation-testing

descriptionDescription

Baseline run of the new `cosmic-ray.toml` against milestone 1. Posting the method and the numbers so the next run has something to compare against, and so anyone auditing the suite does not repeat the triage. ## Result | | Count | |---|---| | Mutants generated | 385 | | Killed | 268 | | Survived | 88 | | Incompetent (mutant could not run) | 29 | | **Mutation score** | **268 / 356 = 75.3%** | A surviving mutant is behaviour the test suite does not pin down. On a repository whose central claim is that the tests *are* the specification, that number is the claim being audited. ## Triage of the 88 survivors **55 are equivalent mutants in type annotations.** `Value | float` appears in six signatures, and `from __future__ import annotations` means annotations are never evaluated at runtime — so mutating `|` to `+`, `/`, `>>` and so on changes nothing observable. Not findings. Worth recording so nobody triages them twice. **4 more are equivalent or cosmetic:** `__neg__`'s `self * -1.0` where `*` becomes `/` (equal for -1.0), a `__repr__` string-formatting mutation, and removing `@staticmethod` from a helper that is only ever called on the class. **The remaining 29 come down to four genuine gaps.** ### 1. No test exercised a backward pass with an incoming gradient other than 1.0 The largest one — it accounts for roughly half the real survivors and touches `__mul__`, `exp`, `tanh` and `relu`. When `out.grad == 1.0`, these are all the same number: ``` self.data * out.grad self.data / out.grad self.data ** out.grad ``` So mutating the multiply in *any* `_backward` survived. Every product in the suite was either the root of the graph or fed an addition, and both hand exactly 1.0 to the multiply. The chain rule *through* a product was never verified. Closed by nesting a product inside a non-linear op, where the incoming gradient is not 1: ```python a, b = Value(2.0), Value(3.0) out = (a * b).exp() # d(out)/d(ab) = e^6, not 1 out.backward() # a.grad == 3 * e^6, b.grad == 2 * e^6 ``` ### 2. `__radd__` was never reached Every binary-operator mutation of its body survived, which can only happen if no test calls it. `test_division_and_reflected_operators` covers `__rsub__` and `__rtruediv__` but never puts a float on the left of a `+`. Addition being commutative, a wrong `__radd__` stays invisible in a symmetric expression — it has to be asserted directly. ### 3. `__rsub__`'s forward value was never asserted Only its gradient contribution was. `__rsub__` is `(-self) + other`; mutate the `+` to `-` and the gradient is still -1, so the mutant lived. The forward value is what separates them: `10 - 4` is 6, the mutant gives -14. ### 4. The relu subgradient at exactly 0 was untested This is the one I would call an actual documentation defect. `Value.relu`'s docstring says: > Convention here: the subgradient at 0 is **0** [...] > Tests: tests/test_scalar_autograd.py::test_relu_backward That test was parametrised over 2.5 and -2.5. At exactly 0 the forward value is 0.0 under both `>` and `>=`, so only the gradient separates them — `>` gives 0.0, `>=` gives 1.0. The docstring named a test for a convention that test did not cover. ## Fixed, in two passes — the first one was not enough Seven tests added in `tests/test_scalar_autograd.py`. **All seven pass against the current implementation.** This was a gap in the specification, not a bug in the code, which is the distinction mutation testing exists to draw. The first pass added four and I assumed that closed it. Re-running and comparing **mutant-by-mutant rather than by aggregate score** showed otherwise: 6 flipped from survived to killed, 0 regressed, and the `exp` and `tanh` survivors were completely untouched. The mistake is instructive. `(a * b).exp()` gives the *product* a non-unit incoming gradient — which is what killed the `__mul__` mutants — but it leaves `exp` at the r
codeOuvre sur GitHub