arrow_backRetour aux issues
MakazhanAlpamys/Soup
#302
Débutant
Ouvrirarrow_forward
Débutant
Ouvrirarrow_forward
Débutant
Ouvrirarrow_forward
Idefics3 / SmolVLM vision-SFT support in the LLaVA path
ecoDébutant
enhancement
help wanted
descriptionDescription
### Context
v0.71.32 shipped the `smolvlm-256m-sft` recipe (a tiny VLM). During the live vision smoke on the dev box (RTX 3050), `soup train` on `HuggingFaceTB/SmolVLM-256M-Instruct` surfaced a **pre-existing** incompatibility in the shared LLaVA vision path — SmolVLM uses an `Idefics3Processor`, which the LLaVA-era code assumes is a LLaVA-style processor.
### What happens
1. `training.lora.target_modules: auto` cannot be inferred by peft for Idefics3 → `ValueError: Please specify target_modules ...` (worked around in the recipe by pinning `target_modules: [q_proj, v_proj]`).
2. With explicit target modules, LoRA applies (755K trainable), then the SFT setup fails:
```
AttributeError: 'Idefics3Processor' object has no attribute 'pad_token'
```
The vision path sets `self.tokenizer = ` and accesses `self.tokenizer.pad_token`; on Idefics3 the pad token lives on `processor.tokenizer.pad_token`.
### Proposed fix path
- In `trainer/sft.py`, when the vision processor is an Idefics3/SmolVLM processor, route tokenizer-level attribute access (`pad_token`, etc.) through `processor.tokenizer`.
- Audit the rest of the vision collation for other Idefics3-vs-LLaVA processor API differences (image-token expansion, chat template) so a full SFT step runs end-to-end.
- Guard against regressing the existing LLaVA / ShareGPT4V path (Qwen2-VL, Pixtral, LLaMA-3.2-Vision).
### Acceptance criteria
- [ ] `soup train` on `HuggingFaceTB/SmolVLM-256M-Instruct` (llava format, `modality: vision`) completes at least one training step on CPU/tiny data without the `pad_token` / target-modules errors.
- [ ] The existing LLaVA vision recipes still train (no regression).
- [ ] `smolvlm-256m-sft` recipe upgraded from parse-only to live, with the recipe note updated.
Ships alongside v0.71.32 as a known limitation; the recipe currently ships parse-tested.
Issues similaires
calkit/calkit
star53
Poids du dépôt moyen
VS Code extension should be robust to YAML parser errors
Seeing this error: ``` Failed to read calkit.yaml: YAMLParseError: A block sequence may not be used as an implicit map…
Python
bug
good first issue
fu351/Doberman-Core
star211
Poids du dépôt léger
dash: a manual Refresh control
The dashboard polls: `refreshStats()` (`src/doberman/dash/app.py:408`) every 5 s and `refreshPending()` (`:546`) every …
Python
enhancement
good first issue
fu351/Doberman-Core
star211
Poids du dépôt léger
dash: "Copy details" button on each pending-approval card
Each pending-approval card in the dashboard (`renderPending`, `src/doberman/dash/app.py:448-544`) shows the risk badge,…
Python
enhancement
good first issue