How vLLM Became a Major Turning Point in the Evolution of LLM Inference Engines

Over the past few years, there has been a remarkable transformation in serving large language models, which went from being a simple challenge of loading models into memory to a sophisticated orchestration of memory management, batching strategies, and hardware optimisation.

This article takes you through this evolution and explain how vLLM became a turning point in LLM inference. 

In 2020, when GPT-3 arrived, the inference landscape was simple. The serving solutions followed the simple pattern of loading the model into GPU memory, processing requests one at a time, and generating tokens sequentially. This led to low GPU utilisation, which was around 20-30%, and because requests were processed in isolation, the throughput didn’t scale with the demand.

Between 2021 and 2022, static batching came about as a response to these issues. 

With this the GPU utilisation and throughput increased but many constraints remained. Some which are:

• All sequences in a batch had to be padded to the same length.

• The batch was held hostage by the longest sequence.

• Varying sequence lengths led to wasted computation on padding tokens.

• You had to choose between optimising for throughput and optimising for latency.

The constraints led to the ORCA breakthrough, which introduced Iteration-Level Batching in 2022. According to this paper, batching could be done at the token generation level instead of at the request level. In some scenarios, ORCA reported 10x-100x throughput improvements. However, ORCA was primarily a research prototype and not a production-ready software. 

In 2023, the team at UC Berkeley’s Sky Computing Lab implemented continuous batching, which solved the memory bottleneck that was preventing these techniques from reaching their full potential. 

The core feature called PagedAttention breaks the KV cache into fixed-size blocks and manages them like virtual memory pages. This led to near-zero memory waste, dynamic memory allocation, and 2-4x higher batch sizes on the same hardware.

• Preemption and swapping: When GPU memory is tight, vLLM can preempt low-priority requests, swap their KV cache to CPU.

• Prefix caching: Common prompt prefixes are computed once and shared across requests

• Speculative decoding support: Integration with draft models for faster generation

• Chunked prefill: Long prompts are processed in chunks to maintain interactivity 

• Simplicity: It is Python-native and requires minimal configuration.

• Transparency: Memory usage is predictable and debuggable

• Flexibility: Works with any Transformer model with minimal changes

• Community: Exceptional responsiveness to issues and PRs

• Extensibility: Easy to customise attention mechanisms, sampling, etc. 

There are two key developments which are reshaping the LLM infrastructure:

• LMCache

It stores KV cache in a hierarchy and retrieve them on demand. It separates the prefill phase from the decode phase, which can maintain consistent decode latency even under variable load. It shares KV caches between different vLLM instances or even between vLLM and SGLang. This enables deployment patterns where specialised instances can handle different workload types. Overall, it contributes to impressive real-world performance with 15x throughput improvement.

• AIBrix

Developed by ByteDance and now part of the vLLM project ecosystem, AIBrix is a Kubernetes-native control plane that sits above vLLM and other inference engines. AIBrix enables KV cache sharing across different vLLM instances and even across different nodes. This can offer a 50% throughput increase and 70% latency reduction.

It can also deploy thousands of LoRA adaptors and has reported 4.7x cost reduction by efficiently packing multiple adaptors. The prefix-aware, load-aware routing maximises cache hits. It also allows SLO-driven GPU optimisation by mixing GPU types while maintaining service level objectives. 

The field is moving fast, going beyond AIBrix and LMCache. Some further directions are:

• Multi-LoRA serving at scale: This includes sophisticated scheduling that predicts which adapters to keep warm based on request patterns.

• Continuous batching for multi-modal models: Extending PagedAttention and continuous batching to vision-language models, where you’re batching both text tokens and image patches.

• ML-based scheduling: Using learned models to predict sequence lengths, optimal batch composition, and preemption decisions rather than heuristics.

• Hardware co-design: Custom accelerators (like Groq’s LPU) designed from the ground up with PagedAttention-style memory management and streaming architectures.

• Speculative decoding improvements: Better draft models, multi-candidate speculation, and learned acceptance thresholds that adapt to workload characteristics.

The convergence of AIBrixe, LMCache, and vLLM’s core inference engine creates a stack that addresses the challenges which existed in the earlier GPU.

The growth journey from naive inference to the current vLLM is one of the most impactful developments in ML engineering. The recent model lowers costs, improves user experience, offers higher throughput, and allows greater accessibility. While the evolution may not be over, offers a new baseline for solutions to be measured against.