Aller au contenu
login
arrow_backRetour aux issues
poojithdevan4D/pooji-vllm #5

Chunked prefill (stop TTFT growing with queue position)

ecoDébutant help wanted performance

descriptionDescription

Prefill currently runs one request at a time in its own tick. With 8 concurrent clients, measured TTFT climbed 47ms → 328ms purely by queue position. **Where:** `pooji_vllm/llm_engine.py` — `step()`, the `while self.waiting:` block. **Approach:** split a long prompt into chunks of N tokens and mix those chunks into the same batch as decode tokens, instead of running a dedicated prefill pass. This requires `_forward` to accept a batch with different T per request (ragged), which is the main design work. **Done when:** TTFT under 8 concurrent clients is roughly flat, with no throughput regression on `benchmarks/bench.py`.
codeOuvre sur GitHub