Skip to main content
The LLM slot is the third of the model server’s three slots, on internal port 8003, answering POST /v1/chat/completions. It is the only slot that ships turned off — LLM_MODEL is empty in .env.example — and it is the one part of the model server that has never been run on hardware.
The LLM slot has not been run on hardware at all. model-server/README.md states it plainly: llm/qwen3.5-4b/ is written but has never been built or started, so the vLLM flags in it are unverified against a live model. models.yaml repeats it against the catalogue entry. Everything on this page describes intended behaviour that no live model has confirmed.

​
qwen3.5-4b

Two other LLM ids are catalogued as planned: gemma and llama, both vLLM. Every flag in the Dockerfile is there for a reason: Runtime limits come from .env and apply to whichever model fills the slot: VLLM_MAX_MODEL_LEN (8192), VLLM_MAX_NUM_SEQS (20), VLLM_GPU_MEMORY_UTILIZATION (0.10), VLLM_QUANTIZATION (empty means bf16). Which model and which model-specific flags live in llm/<model>/Dockerfile, not in .env. GPU_MEMORY_UTILIZATION defaults to 0.10, and note what that means: it is a fraction of the card’s total memory, not of what is free. On a 143 GB H200 that is a hard ~14 GB reservation taken at startup. MPS does not partition memory, so oversizing it takes memory away from the production workers on the same GPU rather than failing cleanly. Never use vLLM’s own 0.9 default here. There is no fetch.sh: vLLM downloads its own weights from HuggingFace into the hf_cache volume on first start, and that volume survives restarts. First start therefore takes several minutes with no output on /health — watch docker compose logs -f llm rather than assuming it has hung.

​
Two vLLM bugs that affect this

Both produce a call with dead air rather than an error, so they are worth knowing before debugging a silent bot.
  • vllm#35574 — chat_template_kwargs: {enable_thinking: false} did not always disable thinking on Qwen3.5. Closed February 2026, so fixed well before the v0.27.1 pinned here, but it is why the runtime also appends /no_think to the system prompt as a second signal.
  • vllm#38894 — with the qwen3 reasoning parser, generated text can arrive in delta.reasoning while delta.content stays empty. Pipecat only forwards content to TTS, so the caller hears nothing. The runtime’s VllmQwenVoiceLLMService._normalize_qwen_chunk copies one into the other, but only while thinking is disabled — with thinking on, that mapping would speak the chain of thought aloud.
tests/test_llm_wiring.py pins both behaviours.

​
Why the folder is small

llm/qwen3.5-4b/ contains a Dockerfile and a README.md. There is no code, because there is nothing to write: vLLM already serves /v1/chat/completions, /v1/models and /health in the shape the gateway forwards. That is the clearest demonstration of what the container contract actually asks for. Some model folders are a full server — tts/indic-parler/ carries a paged-KV-cache engine. This one is about 30 lines of Dockerfile. Both satisfy the same interface. Adding another vLLM model is copying the folder, changing MODEL_ID, SERVED_NAME and EXTRA_ARGS, adding the id to models.yaml, and setting LLM_MODEL to the new folder name. It appears in scripts/start-model-server.sh’s menu on its own. Nothing in compose.model-server.yml or gateway/ changes.

​
The model-id agreement

vLLM rejects any request whose model field is not the name it was started with, and it does so at call time with a 400 — during a live phone call, not at deploy. Four files have to agree on the same string: Nothing else checks that, so tests/test_llm_wiring.py does. It reads the real files — by AST where they are Python — so the test fails when one of them drifts rather than passing against a copy. It skips when the runtime is not present in the checkout. Note that the LLM slot does not select by model name the way STT and TTS do. tests/test_client_selection.py enforces the <catalogue id>-<slot> convention for STT and TTS only; the LLM is asked for its served name instead, which is what test_llm_wiring.py covers.

​
Status

The slot mechanics are tested; the model is not. tests/test_llm_slot.py covers the slot itself, over a real socket rather than an ASGI test client — an ASGI client would not show whether tokens arrive as they are produced, which for a voice agent is the entire point. It checks that an empty slot answers 503 rather than a 404 or a hang, is not advertised at /v1/models, and does not mark /health degraded; and that a filled slot routes and streams token by token. The upstream in that test is a stand-in for vLLM, not vLLM. What that leaves unverified is everything about the model itself: whether the pinned vllm/vllm-openai:v0.27.1 image builds and starts here, whether --language-model-only behaves as expected against this checkpoint, whether the qwen3 reasoning parser splits <think> blocks the way the flag table assumes, and what the real memory footprint is at GPU_MEMORY_UTILIZATION=0.10. Treat the deployment as untried. Start it on its own, watch docker compose logs -f llm through the first weight download, and confirm /health and a plain POST /v1/chat/completions before pointing an agent at it.