IBRAVRA Media Network
AI Tools • 12 min read

vLLM vs TensorRT-LLM vs TGI: The Inference Serving Benchmark

Three engines serve most self-hosted LLMs, and one of them just told its users to leave. TGI is in maintenance mode and its own docs now recommend vLLM and SGLang. This benchmark compares vLLM, TensorRT-LLM, and TGI on the numbers their own projects publish — PagedAttention memory math, throughput claims, quantization, and the cost per 1M tokens at a stated GPU rate — and gives the migration decision for each deployment shape.

By Enow A. Jovial • Published 2026-11-17

Three engines serve most self-hosted large language models, and one of them just told its users to leave. Hugging Face's Text Generation Inference — the engine that defined production LLM serving in 2023 — is now in maintenance mode, and its own documentation recommends vLLM and SGLang going forward. That single notice redraws the decision for every team running TGI, and it makes the vLLM vs TensorRT-LLM comparison the real choice for new deployments.

This benchmark compares the three engines on the numbers their own projects publish, with the arithmetic exposed so you can re-run it on your own hardware. Every throughput figure below carries its source, its hardware, and its dataset, and every cost figure states the GPU rate it assumes.

New deployments should choose between vLLM and TensorRT-LLM. vLLM is the default: the widest hardware and model support, the OpenAI-compatible API, and the PagedAttention memory system that made it the standard. TensorRT-LLM is the NVIDIA-only maximum: the highest published throughput on Blackwell hardware, at the price of build complexity and a single-vendor lock-in. TGI is a migration source, not a destination — its own docs say so.

The three engines at a glance

AttributevLLMTensorRT-LLMTGI
OriginUC Berkeley Sky Computing LabNVIDIAHugging Face
StatusActively developed, 2,000+ contributorsActively developed, fully open-source since Mar 2025Maintenance mode — bug fixes and docs only
HardwareNVIDIA, AMD, CPU, TPU, Gaudi, Ascend, Apple SiliconNVIDIA GPUs onlyNVIDIA GPUs (transformers-based)
Model support200+ Hugging Face architecturesNVIDIA-optimized model set, Day-0 new-model supportMost popular open-source LLMs
APIOpenAI-compatible, Anthropic Messages, gRPCOpenAI-compatible via serving layer, Triton backendOpenAI-compatible, Server-Sent Events streaming
QuantizationFP8, MXFP8, NVFP4, INT8, INT4, GPTQ, AWQFP8, NVFP4, INT4, FP4, KV cache compressionbitsandbytes, GPT-Q
Speculative decodingEAGLE, n-gram, suffix, DFlashN-gram, multi-token prediction (MTP)Medusa, n-gram (via Guidance)
AuthorityvLLM docs + launch blogNVIDIA TensorRT-LLM READMEHugging Face TGI docs

The table already contains the decision for most teams. If you run on anything other than NVIDIA, TensorRT-LLM is not an option. If you run on NVIDIA and want the maximum published throughput, TensorRT-LLM is the ceiling. If you want the engine with the broadest deployment footprint and the simplest operations, vLLM is the default. TGI appears in the table only so existing operators can see exactly what they are migrating from.

PagedAttention: the memory math that decides throughput

LLM serving is bottlenecked by memory, not by compute. In autoregressive decoding, every input token produces attention key and value tensors — the KV cache — that stay in GPU memory while the next token is generated. For LLaMA-13B, the vLLM team measured up to 1.7GB of KV cache for a single sequence. The cache is large, and its size depends on sequence length, which is variable and unpredictable. Traditional systems pre-allocate contiguous memory per sequence and waste 60–80% of it on fragmentation and over-reservation.

PagedAttention, vLLM's core algorithm, partitions the KV cache into fixed-size blocks — pages — that do not need to be contiguous in memory, exactly like virtual memory paging in an operating system. The waste collapses to the last block of each sequence: under 4% in practice. The arithmetic is reproducible from the model architecture alone:

extKVBytesperToken=2imesextLayersimesextHiddenSizeimesextBytesperValueext{KV Bytes per Token} = 2 imes ext{Layers} imes ext{Hidden Size} imes ext{Bytes per Value}

For LLaMA-13B — 40 layers, 5,120 hidden size, FP16 at 2 bytes, times 2 for keys and values — that is 2 × 40 × 5,120 × 2 = 819,200 bytes per token, or 0.82 MB. At a 2,048-token context, one sequence needs 1.68 GB, which is the 1.7 GB the vLLM blog reports. The same formula on your own model tells you the KV budget before you buy the GPU: a 70B model at 80 layers and 8,192 hidden size needs 2 × 80 × 8,192 × 2 = 2.62 MB per token, or 5.37 GB per 2,048-token sequence — and that is before the weights, which at FP16 are 140 GB for 70B parameters and already exceed a single 80 GB card.

That memory arithmetic is the whole throughput story. More sequences batched together means higher GPU utilization; higher utilization means higher throughput. In the vLLM team's June 2023 benchmark — LLaMA-7B on an NVIDIA A10G and LLaMA-13B on an NVIDIA A100 40GB, request lengths sampled from the ShareGPT dataset — vLLM delivered up to 24x higher throughput than HuggingFace Transformers and up to 3.5x higher than TGI. Parallel sampling and beam search, which share prompt blocks through PagedAttention's copy-on-write block table, cut memory usage by up to 55% and improved throughput by up to 2.2x. In production at LMSYS, the Chatbot Arena deployment reached up to 30x higher throughput than its initial HuggingFace backend, cut the GPUs serving the same traffic by 50%, and handled an average of 30,000 requests a day with a 60,000 peak.

vLLM launch blog — up to 24x higher throughput than HuggingFace Transformers and up to 3.5x higher than TGI, measured on LLaMA-7B (A10G) and LLaMA-13B (A100 40GB) with ShareGPT request lengths
vLLM launch blog — up to 24x higher throughput than HuggingFace Transformers and up to 3.5x higher than TGI, measured on LLaMA-7B (A10G) and LLaMA-13B (A100 40GB) with ShareGPT request lengths

TensorRT-LLM: the NVIDIA ceiling

TensorRT-LLM takes the opposite approach to vLLM. Where vLLM optimizes for breadth — every accelerator, every architecture, one pip install — TensorRT-LLM optimizes for depth on one vendor's hardware: hand-written CUDA kernels, TensorRT engine compilation, and model-specific optimization passes that only make sense when the GPU is fixed. The README's headline numbers are all on Blackwell: Llama 4 at over 40,000 tokens per second on B200 GPUs, and a 1,000 tokens-per-second-per-user barrier with Meta's Llama 4 Maverick. Older but still current optimizations include 3x Llama 3.3 70B inference throughput with speculative decoding and more than 3x long-sequence throughput from multiblock attention on HGX H200. The project went fully open-source in March 2025 and ships Day-0 support for new open-weight models — GPT-OSS-120B and GPT-OSS-20B on launch day.

The cost of that ceiling is operational. TensorRT-LLM engines are compiled per model and per GPU: changing the model, the quantization, or the GPU means rebuilding the engine. vLLM loads a Hugging Face checkpoint and serves it. For a team that serves one model on a fixed fleet, the compile step is a one-time cost paid back in throughput. For a team that serves ten models across mixed hardware, it is a rebuild treadmill. The decision is not which engine is faster in the abstract — it is whether your deployment shape amortizes the build.

TGI: the maintenance-mode verdict

The TGI documentation page now opens with a notice that settles the question: text-generation-inference is in maintenance mode, accepting pull requests only for minor bug fixes, documentation improvements, and lightweight maintenance. The notice continues: TGI initiated the movement for optimized inference engines to rely on transformers model architectures, an approach now adopted by downstream engines, which the project contributes to and recommends using going forward — vLLM and SGLang — as well as local engines with inter-compatibility such as llama.cpp or MLX.

Hugging Face TGI documentation — text-generation-inference is now in maintenance mode, and the project recommends vLLM and SGLang going forward
Hugging Face TGI documentation — text-generation-inference is now in maintenance mode, and the project recommends vLLM and SGLang going forward

That is not a hostile fork or a community dispute. It is the upstream project telling its users the future is elsewhere, in writing, on its own documentation front page. TGI's feature list — continuous batching, Flash Attention, Paged Attention, tensor parallelism, Server-Sent Events streaming, Prometheus metrics, bitsandbytes and GPT-Q quantization, Safetensors loading — was the state of the art when it was written, and vLLM and TensorRT-LLM have since absorbed every item on it. Running TGI today means running an engine whose own maintainers have stopped adding features, while its two successors add them weekly.

Cost per 1M tokens: the reproducible formula

Throughput claims become a purchasing decision only when converted to cost. The formula is simple, and the only input you must supply is your own GPU hourly rate — the example below uses an illustrative $3.00 per GPU-hour for a Blackwell-class card and $1.50 for an A100-class card, and you should replace both with your invoice before deciding anything:

ext{Cost per 1M Tokens} = rac{	ext{GPU Hourly Rate} 	imes 1{,}000{,}000}{	ext{Tokens per Second} 	imes 3{,}600}

At 40,000 tokens per second — TensorRT-LLM's published Llama 4 figure on B200 — and $3.00 per GPU-hour, the cost is $3.00 × 1,000,000 / (40,000 × 3,600) = $0.0208 per 1M tokens. At 2,000 tokens per second on a $1.50-per-hour card, the same formula gives $0.2083 per 1M tokens — 10x more, for the same million tokens, because the throughput denominator is 20x smaller. The ratio is the point: a 24x throughput improvement is a 24x cost improvement at a fixed GPU price, which is why LMSYS cut its serving GPUs by 50% when it moved to vLLM. Run the formula with your own rate and your own measured tokens per second — the vendor's number is a starting point, not a quote — and check the result against our ROI Calculator with the GPU spend as the investment and the API cost avoided as the return.

Vendor benchmarks are vendor benchmarks. The vLLM 24x figure is LLaMA-7B and LLaMA-13B on ShareGPT lengths; the TensorRT-LLM 40,000 tokens-per-second figure is Llama 4 on B200. Your prompts, your context lengths, and your concurrency profile will produce different numbers — long contexts punish KV cache, short bursts punish time-to-first-token, and neither vendor's benchmark is your workload. Measure time-to-first-token and tokens-per-second-per-user on your own traffic before you commit a fleet to either engine.
Turn on prefix caching before you buy a bigger GPU. Repeated system prompts and few-shot examples share KV blocks across requests in both vLLM and TensorRT-LLM, and for agentic workloads with long shared prefixes it is the cheapest throughput you will ever get — it costs a configuration flag, not a card.

The decision matrix

Deployment shapeEngineWhy
New deployment, mixed or unknown hardwarevLLMWidest accelerator and model support, one pip install, OpenAI-compatible API, PagedAttention memory system
NVIDIA-only fleet, one or two models, throughput is the businessTensorRT-LLMHighest published throughput on Blackwell, speculative decoding and kernel-level optimization, Day-0 model support
Existing TGI deploymentMigrate to vLLMTGI is in maintenance mode; vLLM is the migration its own docs recommend, and the API surface is compatible
Long-context RAG or agentic workloadsvLLMPrefix caching and disaggregated prefill/decode; pair with the vector database comparison for the retrieval half of the stack
CPU-only or Apple Silicon edgevLLMThe only engine of the three with CPU and Apple Silicon plugins; TensorRT-LLM cannot run there at all
Regulated environment needing audit trailsvLLMOpenTelemetry tracing and Prometheus metrics are first-class in the docs; pair with the self-hosting LLM cost guide for the full bare-metal budget

For the infrastructure half of the decision — the network path your tokens travel over — the Cloudflare Workers vs Vercel edge comparison covers the latency budget that sits in front of any inference server.

Founder & CEO Insight — Enow A. Jovial

"Every team I have seen waste money on inference wasted it the same way: they bought GPUs before they measured the KV cache. The model weights are the price of admission. The KV cache is the rent, and it is charged per token, per concurrent user, forever. Do the memory math first, then pick the engine, then buy the card — in that order."
— Enow A. Jovial, Founder & Chief Executive Officer

Who should not self-host at all

Self-hosting is not the default answer. If your volume is under roughly 1M tokens a day, a managed API is cheaper than one GPU running at 5% utilization — the fixed cost of the card dominates the formula above at low throughput. If your prompts are short and bursty with long idle gaps, serverless inference or an API wins on the same arithmetic. If you need the newest frontier model the week it ships, an API is the only route: self-hosting waits for open weights. The engines in this benchmark are for teams with sustained volume, stable models, and a reason — cost, latency, data residency, or customization — to own the serving layer. Everyone else should rent tokens and spend the engineering time on the product.

Migration protocol: TGI to vLLM

  1. Swap the server — replace the TGI launcher with the vLLM OpenAI API server; the client side changes only the base URL.
  2. Re-check quantization — confirm your GPTQ, AWQ, FP8, or INT8 weights load under vLLM and re-validate accuracy on your evaluation set.
  3. Re-tune the KV budget — set max-model-len and gpu-memory-utilization from the KV formula above for your context and concurrency.
  4. Re-run the latency SLOs — time-to-first-token and tokens-per-second-per-user on your own prompts before cutting traffic over.
  5. Keep TGI as fallback — run both for one full traffic day, then decommission TGI once vLLM passes every SLO.

Operator checklist

  • [x] KV budget computed — KV bytes per token from the layer/hidden-size formula, times your max context, times your target concurrency, against GPU memory minus weights.
  • [x] Engine chosen by deployment shape — vLLM by default, TensorRT-LLM for NVIDIA-only throughput, TGI only as a migration source.
  • [x] Vendor numbers re-measured — time-to-first-token and tokens-per-second-per-user on your own prompts, not the vendor's benchmark workload.
  • [x] Cost per 1M tokens priced — the formula above with your own GPU hourly rate, checked against the managed-API price for the same volume.
  • [x] Prefix caching enabled — shared system prompts and few-shot examples cached, verified by a hit-rate metric.
  • [x] Quantization validated — FP8 or AWQ accuracy measured on your evaluation set before production, not assumed from the docs.
  • [x] TGI migration planned — OpenAI-compatible API base URL swapped, quantization re-checked, latency SLOs re-run, TGI kept as fallback for one full traffic day.
  • [x] Observability wired — Prometheus metrics and request tracing on, with alerts on queue depth and KV cache pressure.

The inference engine decision is a memory decision wearing a throughput costume. PagedAttention exists because the KV cache was eating 60–80% of GPU memory; TensorRT-LLM's headline numbers exist because its kernels squeeze the same cache harder on one vendor's silicon; TGI's maintenance notice exists because its ideas won and its codebase handed the future to its successors. Measure the KV budget, price the million tokens, and pick the engine that fits the fleet you actually run.