Aller au contenu
login
arrow_backRetour aux issues
poojithdevan4D/pooji-vllm #3

Swap-to-CPU preemption as an alternative to recompute

ecoDébutant help wanted

descriptionDescription

When blocks run out, `LLMEngine._preempt()` evicts the newest request and re-prefills it later. vLLM also supports copying its KV blocks to host memory and back, which trades PCIe bandwidth for recomputation. **Where:** `pooji_vllm/llm_engine.py` — `_preempt`. **Interesting because** which policy wins depends on prompt length: long prompts are expensive to recompute, so swapping should win past some crossover. Finding and documenting that crossover is most of the value here. **Done when:** both policies are selectable and BENCHMARKS.md records where each one wins.
codeOuvre sur GitHub