vLLM Cheat Sheet

1Installpip / GPU

pip install vllm★
Prebuilt CUDA wheels. On uv: uv pip install vllm --torch-backend=auto auto-detects your GPU/driver.
# needs a supported GPU (NVIDIA/AMD/Intel) on Linux; Python 3.10-3.14note
Linux-first. CPU, TPU, Gaudi, and Apple-silicon builds exist but use separate install paths (see docs).
export HF_TOKEN="hf_..." # for gated Hugging Face models
Models are pulled from the Hugging Face Hub by id. Set VLLM_USE_MODELSCOPE=1 to pull from ModelScope instead.
from vllm import LLM, SamplingParams★
The two objects you use most: the engine and its generation config.

2Offline generate()batched local

llm = LLM(model="meta-llama/Llama-3.1-8B-Instruct")★
Loads weights onto the GPU once. Any HF model id (or a local path). Add tensor_parallel_size=N to shard across N GPUs.
outs = llm.generate(["Hello, ", "The capital of France is"], sp)★
Pass a list of prompts — vLLM batches them together. Continuous batching keeps the GPU saturated automatically.
for o in outs: print(o.prompt, o.outputs[0].text)★
Each RequestOutput has the prompt + one or more CompletionOutputs (.text, .token_ids, .cumulative_logprob).
LLM(model=..., max_model_len=8192, dtype="bfloat16", seed=0)
Common constructor args: cap context length, set precision, fix the seed. These mirror the vllm serve flags.

3SamplingParamsdecoding knobs

sp = SamplingParams(temperature=0.8, top_p=0.95, max_tokens=256)★
The core knobs. temperature=0 = greedy/deterministic. Also top_k, min_p, repetition_penalty, presence_penalty.
SamplingParams(n=3, best_of=3)
n = how many completions to return per prompt; best_of generates then keeps the best.
SamplingParams(stop=["\n\n"], stop_token_ids=[128009])
Stop strings/token ids. ignore_eos=True to keep generating past EOS.
SamplingParams(logprobs=5, prompt_logprobs=1)
Return per-token logprobs for the output (and optionally the prompt).

4Chat & multimodaltemplated

outs = llm.chat([ {"role":"system","content":"Be brief."}, {"role":"user","content":"Hi"}], sp)★
llm.chat() applies the model's chat template for you (vs. generate() which takes raw prompt text).
{"role":"user","content":[ {"type":"text","text":"What is this?"}, {"type":"image_url","image_url":{"url":"https://..."}}]}
Vision-language models take OpenAI-style multimodal content parts (image, audio, video for capable models).
LLM(model="Qwen/Qwen2.5-VL-7B-Instruct", limit_mm_per_prompt={"image": 2})
Cap multimodal items per prompt. For offline, images can also be passed via multi_modal_data in the prompt dict.
tok = llm.get_tokenizer() # the HF tokenizer
Access the underlying tokenizer for manual templating or token counting.

5Embeddings & poolingencoder tasks

llm = LLM(model="intfloat/e5-mistral-7b-instruct", runner="pooling")★
Load a model in pooling mode for embeddings/classification/reranking (the successor to the old task= arg).
out = llm.embed(["text a", "text b"]) out[0].outputs.embedding # vector★
Batched embeddings. Also llm.classify(...), llm.score(...) (reranking), and llm.reward(...).
# served: POST /v1/embeddings (OpenAI-compatible)
The online server exposes the same via the OpenAI embeddings endpoint — drop-in for the OpenAI client.

6vllm serve (OpenAI server)the gateway

vllm serve "meta-llama/Llama-3.1-8B-Instruct"★
Starts an HTTP server on :8000 exposing /v1/chat/completions, /v1/completions, /v1/models, /v1/embeddings.
vllm serve MODEL --host 0.0.0.0 --port 8000 --api-key secret★
Bind address/port and require a bearer token. Clients then send Authorization: Bearer secret.
vllm serve MODEL --max-model-len 16384 --dtype bfloat16
Same knobs as the LLM constructor, as CLI flags. --served-model-name sets the id clients request.
vllm serve MODEL --tensor-parallel-size 4 --gpu-memory-utilization 0.9★
Shard across 4 GPUs and use 90% of VRAM for weights + KV cache. See cards 8–9.
curl localhost:8000/v1/models · curl localhost:8000/health
Sanity endpoints: list loaded models and check liveness. Metrics at /metrics (Prometheus).

7Query the serveropenai client

from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")★
Point the official OpenAI SDK at vLLM — no other change. api_key="EMPTY" when the server has no key.
r = client.chat.completions.create(model="...", messages=[{"role":"user","content":"Hi"}])★
model must match the served id (default = the HF path, or --served-model-name). Read r.choices[0].message.content.
client.chat.completions.create(..., stream=True)
SSE streaming works exactly like OpenAI. Also client.completions and client.embeddings.
curl localhost:8000/v1/chat/completions -H "Content-Type: application/json" \ -d '{"model":"...","messages":[{"role":"user","content":"Hi"}]}'
Any HTTP client works — it's just the OpenAI wire format.

8Parallelism & scalingmulti-GPU

--tensor-parallel-size 4 # split each layer across 4 GPUs★
TP: use when the model doesn't fit on one GPU. Set to the number of GPUs on the node.
--pipeline-parallel-size 2 # split layers across nodes/stages
PP: stack across stages, typically for multi-node. Combine TP × PP for very large models.
--data-parallel-size 2 # replicas for throughput
DP runs independent replicas to raise throughput. --enable-expert-parallel for MoE models.
--max-num-seqs 256 · --max-num-batched-tokens 8192
Batching limits: concurrent sequences and tokens per iteration — the main throughput/latency dials.

9Quantization & memoryfit more

vllm serve MODEL --quantization awq # or gptq, fp8, ...★
Serve pre-quantized checkpoints (AWQ, GPTQ, FP8, bitsandbytes, compressed-tensors) to cut VRAM & boost speed.
--gpu-memory-utilization 0.90★
Fraction of VRAM vLLM may use. Lower it if other processes share the GPU; raise it to grow the KV cache.
--kv-cache-dtype fp8 · --max-model-len 32768
FP8 KV cache doubles cache capacity (longer context / more concurrency). Context length is capped by --max-model-len.
# CUDA out of memory on startup?gotcha
Lower --gpu-memory-utilization or --max-model-len, add TP, or use a quantized model. vLLM pre-allocates the KV cache at load time.

10Structured outputguided decoding

from vllm.sampling_params import GuidedDecodingParams gd = GuidedDecodingParams(json=MySchema.model_json_schema()) SamplingParams(guided_decoding=gd)★
Constrain generation offline to a JSON schema, regex, grammar, or choice list — the model cannot emit invalid output.
GuidedDecodingParams(regex=r"\d{3}-\d{4}") · choice=["yes","no"]
Also regex, EBNF grammar=, and a fixed choice= list.
# served: response_format={"type":"json_schema", ...} # or extra_body={"guided_json": schema}
Over HTTP, use OpenAI's response_format or vLLM's guided_json/guided_regex in extra_body.

11Performance & engine argsgo faster

--enable-prefix-caching★
Cache & reuse shared prompt prefixes (system prompts, few-shot) across requests — big win for repeated context. On by default in recent versions.
--enable-chunked-prefill
Interleave prefill with decode for smoother latency under load. Also default on for many configs.
--speculative-config '{"model":"...","num_speculative_tokens":5}'
Speculative decoding with a draft model (or n-gram/EAGLE) to accelerate token generation.
vllm serve MODEL --enable-lora --lora-modules sql=/path
Serve multiple LoRA adapters on one base model; clients pick the adapter by model name.
vllm serve MODEL --enable-auto-tool-choice --tool-call-parser hermes
Enable OpenAI-style function/tool calling (parser depends on the model family).

12Deploy & ecosystemship it

docker run --gpus all -p 8000:8000 \ vllm/vllm-openai:latest --model MODEL★
Official image bundles the OpenAI server. Mount ~/.cache/huggingface to avoid re-downloading weights.
# Kubernetes: vLLM Production Stack / KServe / Ray Serve
Autoscaling & routing via the vLLM production stack, KServe, or Ray Serve; llm-d for large distributed deployments.
# drop-in backend for LangChain, LlamaIndex, OpenAI SDK
Because it speaks the OpenAI API, vLLM slots under any framework that targets OpenAI — just change the base URL.
vllm bench serve · vllm bench latency · vllm bench throughput
Built-in benchmarking subcommands to measure your config before shipping.

vLLM cheat sheet

Outline

Offline inference

Chat / multimodal / embed

Online serving

Scaling & tuning

Deploy & ecosystem

Offline Inference

Chat, Multimodal & Embeddings

Online Serving

Scaling & Tuning

Deploy & Ecosystem

Worth memorizing