Aller au contenu
login
arrow_backRetour aux issues
MakazhanAlpamys/Soup #302

Idefics3 / SmolVLM vision-SFT support in the LLaVA path

ecoDébutant enhancement help wanted

descriptionDescription

### Context v0.71.32 shipped the `smolvlm-256m-sft` recipe (a tiny VLM). During the live vision smoke on the dev box (RTX 3050), `soup train` on `HuggingFaceTB/SmolVLM-256M-Instruct` surfaced a **pre-existing** incompatibility in the shared LLaVA vision path — SmolVLM uses an `Idefics3Processor`, which the LLaVA-era code assumes is a LLaVA-style processor. ### What happens 1. `training.lora.target_modules: auto` cannot be inferred by peft for Idefics3 → `ValueError: Please specify target_modules ...` (worked around in the recipe by pinning `target_modules: [q_proj, v_proj]`). 2. With explicit target modules, LoRA applies (755K trainable), then the SFT setup fails: ``` AttributeError: 'Idefics3Processor' object has no attribute 'pad_token' ``` The vision path sets `self.tokenizer = ` and accesses `self.tokenizer.pad_token`; on Idefics3 the pad token lives on `processor.tokenizer.pad_token`. ### Proposed fix path - In `trainer/sft.py`, when the vision processor is an Idefics3/SmolVLM processor, route tokenizer-level attribute access (`pad_token`, etc.) through `processor.tokenizer`. - Audit the rest of the vision collation for other Idefics3-vs-LLaVA processor API differences (image-token expansion, chat template) so a full SFT step runs end-to-end. - Guard against regressing the existing LLaVA / ShareGPT4V path (Qwen2-VL, Pixtral, LLaMA-3.2-Vision). ### Acceptance criteria - [ ] `soup train` on `HuggingFaceTB/SmolVLM-256M-Instruct` (llava format, `modality: vision`) completes at least one training step on CPU/tiny data without the `pad_token` / target-modules errors. - [ ] The existing LLaVA vision recipes still train (no regression). - [ ] `smolvlm-256m-sft` recipe upgraded from parse-only to live, with the recipe note updated. Ships alongside v0.71.32 as a known limitation; the recipe currently ships parse-tested.
codeOuvre sur GitHub