Aller au contenu
login
arrow_backRetour aux issues
haoxiang-xu/PuPu #226

web_fetch returns noise-dominated, silently truncated output that drives model fabrication

ecoDébutant bug Unchain good first issue

descriptionDescription

> [DRAFT — intent only. The agent responsible for implementation may refine this ticket directly.] **What & Why** — `web_fetch` returns raw HTML dominated by boilerplate and silently truncated, and models answer confidently from it instead of reading further or declining. Asked to read a GitHub repo page, qwen3:32b fetched successfully and then fabricated all three answers — inventing an entirely different project — while citing a URL it never fetched. The same model given the same repo's clean README answered accurately and volunteered which parts it was inferring rather than reading. The tool, not the model, is what fails here, and the failure is invisible: the user receives a fluent, well-structured, sourced-looking answer that is false. **Acceptance (outline)** - A model fetching a mainstream, boilerplate-heavy page receives content whose usable signal is reachable, not buried past the returned-character budget. - Truncation is a signal the model actually acts on — it continues reading or says it could not read enough, instead of answering from a partial fetch. - qwen3:32b answers the repo-page task correctly, or declines; it does not fabricate, and it does not cite pages it never fetched. --- ## Reproduction evidence (qwen3:32b via Ollama, 2026-08-22) **Failing run.** Task: read `https://github.com/haoxiang-xu/PuPu` and report purpose, primary language, latest release. | Question | qwen3 answered | Actual | |---|---|---| | What it does | "a lightweight Python package for single-cell RNA-seq analysis" | cross-platform desktop AI client | | Primary language | Python | JavaScript | | Latest release | v0.1.0 | v0.1.9 | It made exactly **one** `web_fetch` call, then claimed the version came "根据页面内容中的 `.../releases/latest` 链接" — a page it never fetched. **The tool did return the right answer for Q1.** Inspecting the raw return: HTTP 200, `returned_chars: 20000`, and the correct description sits verbatim at character **7600**: > `PuPu is a lightweight, cross-platform desktop AI client that works with both local and cloud-hosted models...` The model had it and still said single-cell RNA-seq. **Control run — same model, clean input, passes.** Given `raw.githubusercontent.com/.../README.md` (7198 chars of markdown, untruncated), qwen3 correctly reported the desktop AI client purpose, local/cloud model management, workspace-aware chat, and the MCP tool store — all verified against the file. Asked for the tech stack, it correctly said the README does not state one (`Electron` and `React` each appear 0 times) and explicitly labelled its Electron guess 「合理推测而非文档明确说明」. So the model **can** distinguish read-from-source and inferred, and **will** decline when the text does not support an answer. Noisy truncated input is what destroys that behavior. ## Contributing factors 1. **Signal-to-noise.** Of the 20000 returned characters, the first ~7000 are GitHub's `featureFlags` JSON blob (hundreds of entries like `billing_cost_center_list_assigned_resources`). 2. **Truncation is metadata-only.** The return carried `truncated: True`, `content_length: 66386`, `next_offset: 20000` — the model received under a third of the page and ignored all three fields. Both facts it got wrong (JavaScript, v0.1.9) were in the unfetched remainder. 3. **`mode` default.** `web_fetch(url, mode="raw")` defaults to raw HTML (`unchain/src/unchain/toolkits/builtin/core/core.py:200`). An `extract` mode exists; nothing steers the model toward it. ## Possible directions (not prescriptive) - Reconsider the default `mode`, or strip boilerplate before returning raw content. - Surface truncation as an in-band notice at the end of the returned text, where the model must read it, rather than as a sibling metadata field. - Prompt guidance that a truncated fetch is not a basis for a final answer. ## Notes - The fix is in the `unchain` repository (`toolkits/builtin/core/core.py` and its web backend); PuPu consumes it. - `.py` changes require a sidecar restart bef
codeOuvre sur GitHub