Aller au contenu
login
arrow_backRetour aux issues
vshulcz/deja-vu #1269

Exact search ranks by date and breaks ties by topicality; relevance ignores age entirely

ecoDébutant help wanted

descriptionDescription

Two ranking paths treat a session's age completely differently, and only one of them is covered by any benchmark. **Exact search** multiplies every score by `1/(1 + age in days)` (`internal/search/search.go:530`, `608`). `deja blame` does the same (`blame.go:112`). The curve is hyperbolic, so it does not taper — it collapses: ``` age multiplier today 1.00 1 week 0.13 1 month 0.03 1 year 0.003 ``` A perfect match from six months ago keeps under one per cent of its score. BM25 scores do not span anything like that range, so on the exact tier the answer is ordered by date and topicality breaks the ties, which is the opposite of the intent the code states. **Relevance ranking** does not weigh age at all. `Updated` appears only as a tie-break, after scores compare exactly equal (`internal/index/retrieval.go:940-941`). So the same question, answered by two tiers, is ranked on two different theories of what matters. ## What was measured Replacing the decay with a bounded `0.85 + 0.15/(1+age)` — a floor of 0.85 instead of 0.003: ``` day0bench hit@1 21/40 hit@5 31/40 found@50 40/40 mrr .627 unchanged longmemeval 77.5 / 93.0 / 95.5 / 97.0, MRR .843 unchanged decisionbench green ``` Nothing moved, and that is the finding rather than a result: those questions are answered by the relevance path, which never consulted the decay. **No benchmark in this repo exercises the exact tier's freshness weighting**, so a 300-fold multiplier can be replaced by a 1.18-fold one without a single number changing. ## Why it was not changed The research pass that surfaced this asked for validation by LongMemEval question type and by LoCoMo temporal category, and said explicitly not to ship on aggregate MRR. `scripts/locomo` exists; `locomo10.json` is not on this machine, only `longmemeval_s_cleaned.json`. The regression this would cause — "what did we do most recently" landing on something from last year — is exactly what the missing dataset covers. ## What would settle it - LoCoMo, run by category, before and after. - A fixture on the exact tier where a strong old match and a weak recent one compete, since nothing tests that today. - A decision on whether the two tiers should agree at all. If recency is worth 300x on one path and nothing on the other, at least one of them is wrong.
codeOuvre sur GitHub