Three engines serve most self-hosted large language models, and one of them just told its users to leave. Hugging Face's Text Generation Inference — the engine that defined production LLM serving in 2023 — is now in maintenance mode, and its own documentation recommends vLLM and SGLang going forward. That single notice redraws the decision for every team running TGI, and it makes the vLLM vs TensorRT-LLM comparison the real choice for new deployments.
This benchmark compares the three engines on the numbers their own projects publish, with the arithmetic exposed so you can re-run it on your own hardware. Every throughput figure below carries its source, its hardware, and its dataset, and every cost figure states the GPU rate it assumes.
The three engines at a glance
| Attribute | vLLM | TensorRT-LLM | TGI |
|---|---|---|---|
| Origin | UC Berkeley Sky Computing Lab | NVIDIA | Hugging Face |
| Status | Actively developed, 2,000+ contributors | Actively developed, fully open-source since Mar 2025 | Maintenance mode — bug fixes and docs only |
| Hardware | NVIDIA, AMD, CPU, TPU, Gaudi, Ascend, Apple Silicon | NVIDIA GPUs only | NVIDIA GPUs (transformers-based) |
| Model support | 200+ Hugging Face architectures | NVIDIA-optimized model set, Day-0 new-model support | Most popular open-source LLMs |
| API | OpenAI-compatible, Anthropic Messages, gRPC | OpenAI-compatible via serving layer, Triton backend | OpenAI-compatible, Server-Sent Events streaming |
| Quantization | FP8, MXFP8, NVFP4, INT8, INT4, GPTQ, AWQ | FP8, NVFP4, INT4, FP4, KV cache compression | bitsandbytes, GPT-Q |
| Speculative decoding | EAGLE, n-gram, suffix, DFlash | N-gram, multi-token prediction (MTP) | Medusa, n-gram (via Guidance) |
| Authority | vLLM docs + launch blog | NVIDIA TensorRT-LLM README | Hugging Face TGI docs |
The table already contains the decision for most teams. If you run on anything other than NVIDIA, TensorRT-LLM is not an option. If you run on NVIDIA and want the maximum published throughput, TensorRT-LLM is the ceiling. If you want the engine with the broadest deployment footprint and the simplest operations, vLLM is the default. TGI appears in the table only so existing operators can see exactly what they are migrating from.
PagedAttention: the memory math that decides throughput
LLM serving is bottlenecked by memory, not by compute. In autoregressive decoding, every input token produces attention key and value tensors — the KV cache — that stay in GPU memory while the next token is generated. For LLaMA-13B, the vLLM team measured up to 1.7GB of KV cache for a single sequence. The cache is large, and its size depends on sequence length, which is variable and unpredictable. Traditional systems pre-allocate contiguous memory per sequence and waste 60–80% of it on fragmentation and over-reservation.
PagedAttention, vLLM's core algorithm, partitions the KV cache into fixed-size blocks — pages — that do not need to be contiguous in memory, exactly like virtual memory paging in an operating system. The waste collapses to the last block of each sequence: under 4% in practice. The arithmetic is reproducible from the model architecture alone:
For LLaMA-13B — 40 layers, 5,120 hidden size, FP16 at 2 bytes, times 2 for keys and values — that is 2 × 40 × 5,120 × 2 = 819,200 bytes per token, or 0.82 MB. At a 2,048-token context, one sequence needs 1.68 GB, which is the 1.7 GB the vLLM blog reports. The same formula on your own model tells you the KV budget before you buy the GPU: a 70B model at 80 layers and 8,192 hidden size needs 2 × 80 × 8,192 × 2 = 2.62 MB per token, or 5.37 GB per 2,048-token sequence — and that is before the weights, which at FP16 are 140 GB for 70B parameters and already exceed a single 80 GB card.
That memory arithmetic is the whole throughput story. More sequences batched together means higher GPU utilization; higher utilization means higher throughput. In the vLLM team's June 2023 benchmark — LLaMA-7B on an NVIDIA A10G and LLaMA-13B on an NVIDIA A100 40GB, request lengths sampled from the ShareGPT dataset — vLLM delivered up to 24x higher throughput than HuggingFace Transformers and up to 3.5x higher than TGI. Parallel sampling and beam search, which share prompt blocks through PagedAttention's copy-on-write block table, cut memory usage by up to 55% and improved throughput by up to 2.2x. In production at LMSYS, the Chatbot Arena deployment reached up to 30x higher throughput than its initial HuggingFace backend, cut the GPUs serving the same traffic by 50%, and handled an average of 30,000 requests a day with a 60,000 peak.
TensorRT-LLM: the NVIDIA ceiling
TensorRT-LLM takes the opposite approach to vLLM. Where vLLM optimizes for breadth — every accelerator, every architecture, one pip install — TensorRT-LLM optimizes for depth on one vendor's hardware: hand-written CUDA kernels, TensorRT engine compilation, and model-specific optimization passes that only make sense when the GPU is fixed. The README's headline numbers are all on Blackwell: Llama 4 at over 40,000 tokens per second on B200 GPUs, and a 1,000 tokens-per-second-per-user barrier with Meta's Llama 4 Maverick. Older but still current optimizations include 3x Llama 3.3 70B inference throughput with speculative decoding and more than 3x long-sequence throughput from multiblock attention on HGX H200. The project went fully open-source in March 2025 and ships Day-0 support for new open-weight models — GPT-OSS-120B and GPT-OSS-20B on launch day.
The cost of that ceiling is operational. TensorRT-LLM engines are compiled per model and per GPU: changing the model, the quantization, or the GPU means rebuilding the engine. vLLM loads a Hugging Face checkpoint and serves it. For a team that serves one model on a fixed fleet, the compile step is a one-time cost paid back in throughput. For a team that serves ten models across mixed hardware, it is a rebuild treadmill. The decision is not which engine is faster in the abstract — it is whether your deployment shape amortizes the build.
TGI: the maintenance-mode verdict
The TGI documentation page now opens with a notice that settles the question: text-generation-inference is in maintenance mode, accepting pull requests only for minor bug fixes, documentation improvements, and lightweight maintenance. The notice continues: TGI initiated the movement for optimized inference engines to rely on transformers model architectures, an approach now adopted by downstream engines, which the project contributes to and recommends using going forward — vLLM and SGLang — as well as local engines with inter-compatibility such as llama.cpp or MLX.
That is not a hostile fork or a community dispute. It is the upstream project telling its users the future is elsewhere, in writing, on its own documentation front page. TGI's feature list — continuous batching, Flash Attention, Paged Attention, tensor parallelism, Server-Sent Events streaming, Prometheus metrics, bitsandbytes and GPT-Q quantization, Safetensors loading — was the state of the art when it was written, and vLLM and TensorRT-LLM have since absorbed every item on it. Running TGI today means running an engine whose own maintainers have stopped adding features, while its two successors add them weekly.
Cost per 1M tokens: the reproducible formula
Throughput claims become a purchasing decision only when converted to cost. The formula is simple, and the only input you must supply is your own GPU hourly rate — the example below uses an illustrative $3.00 per GPU-hour for a Blackwell-class card and $1.50 for an A100-class card, and you should replace both with your invoice before deciding anything:
ext{Cost per 1M Tokens} = rac{ ext{GPU Hourly Rate} imes 1{,}000{,}000}{ ext{Tokens per Second} imes 3{,}600}
At 40,000 tokens per second — TensorRT-LLM's published Llama 4 figure on B200 — and $3.00 per GPU-hour, the cost is $3.00 × 1,000,000 / (40,000 × 3,600) = $0.0208 per 1M tokens. At 2,000 tokens per second on a $1.50-per-hour card, the same formula gives $0.2083 per 1M tokens — 10x more, for the same million tokens, because the throughput denominator is 20x smaller. The ratio is the point: a 24x throughput improvement is a 24x cost improvement at a fixed GPU price, which is why LMSYS cut its serving GPUs by 50% when it moved to vLLM. Run the formula with your own rate and your own measured tokens per second — the vendor's number is a starting point, not a quote — and check the result against our ROI Calculator with the GPU spend as the investment and the API cost avoided as the return.
The decision matrix
| Deployment shape | Engine | Why |
|---|---|---|
| New deployment, mixed or unknown hardware | vLLM | Widest accelerator and model support, one pip install, OpenAI-compatible API, PagedAttention memory system |
| NVIDIA-only fleet, one or two models, throughput is the business | TensorRT-LLM | Highest published throughput on Blackwell, speculative decoding and kernel-level optimization, Day-0 model support |
| Existing TGI deployment | Migrate to vLLM | TGI is in maintenance mode; vLLM is the migration its own docs recommend, and the API surface is compatible |
| Long-context RAG or agentic workloads | vLLM | Prefix caching and disaggregated prefill/decode; pair with the vector database comparison for the retrieval half of the stack |
| CPU-only or Apple Silicon edge | vLLM | The only engine of the three with CPU and Apple Silicon plugins; TensorRT-LLM cannot run there at all |
| Regulated environment needing audit trails | vLLM | OpenTelemetry tracing and Prometheus metrics are first-class in the docs; pair with the self-hosting LLM cost guide for the full bare-metal budget |
For the infrastructure half of the decision — the network path your tokens travel over — the Cloudflare Workers vs Vercel edge comparison covers the latency budget that sits in front of any inference server.
Founder & CEO Insight — Enow A. Jovial
Who should not self-host at all
Self-hosting is not the default answer. If your volume is under roughly 1M tokens a day, a managed API is cheaper than one GPU running at 5% utilization — the fixed cost of the card dominates the formula above at low throughput. If your prompts are short and bursty with long idle gaps, serverless inference or an API wins on the same arithmetic. If you need the newest frontier model the week it ships, an API is the only route: self-hosting waits for open weights. The engines in this benchmark are for teams with sustained volume, stable models, and a reason — cost, latency, data residency, or customization — to own the serving layer. Everyone else should rent tokens and spend the engineering time on the product.
Migration protocol: TGI to vLLM
- Swap the server — replace the TGI launcher with the vLLM OpenAI API server; the client side changes only the base URL.
- Re-check quantization — confirm your GPTQ, AWQ, FP8, or INT8 weights load under vLLM and re-validate accuracy on your evaluation set.
- Re-tune the KV budget — set max-model-len and gpu-memory-utilization from the KV formula above for your context and concurrency.
- Re-run the latency SLOs — time-to-first-token and tokens-per-second-per-user on your own prompts before cutting traffic over.
- Keep TGI as fallback — run both for one full traffic day, then decommission TGI once vLLM passes every SLO.
Operator checklist
- [x] KV budget computed — KV bytes per token from the layer/hidden-size formula, times your max context, times your target concurrency, against GPU memory minus weights.
- [x] Engine chosen by deployment shape — vLLM by default, TensorRT-LLM for NVIDIA-only throughput, TGI only as a migration source.
- [x] Vendor numbers re-measured — time-to-first-token and tokens-per-second-per-user on your own prompts, not the vendor's benchmark workload.
- [x] Cost per 1M tokens priced — the formula above with your own GPU hourly rate, checked against the managed-API price for the same volume.
- [x] Prefix caching enabled — shared system prompts and few-shot examples cached, verified by a hit-rate metric.
- [x] Quantization validated — FP8 or AWQ accuracy measured on your evaluation set before production, not assumed from the docs.
- [x] TGI migration planned — OpenAI-compatible API base URL swapped, quantization re-checked, latency SLOs re-run, TGI kept as fallback for one full traffic day.
- [x] Observability wired — Prometheus metrics and request tracing on, with alerts on queue depth and KV cache pressure.
The inference engine decision is a memory decision wearing a throughput costume. PagedAttention exists because the KV cache was eating 60–80% of GPU memory; TensorRT-LLM's headline numbers exist because its kernels squeeze the same cache harder on one vendor's silicon; TGI's maintenance notice exists because its ideas won and its codebase handed the future to its successors. Measure the KV budget, price the million tokens, and pick the engine that fits the fleet you actually run.

