Open-Source LLM Deployment Architecture: vLLM, Ollama & SGLang (2026)

Open-source LLMs — Llama 3, Mistral, Qwen 2.5, Gemma 3, DeepSeek V3 — have reached production-quality performance across most tasks in 2026. Running these models at scale requires purpose-built inference servers that handle batching, KV cache management, quantization, and multi-GPU tensor parallelism. The inference serving landscape has consolidated around a few dominant frameworks: vLLM for high-throughput production serving, SGLang for structured output and agentic workloads, Ollama for local development, and llama.cpp for CPU and edge deployments.

This guide covers the architecture of each framework, key concepts for your architecture diagram, and prompt templates for diagramming production open-source LLM deployments.

vLLM: high-throughput production inference

vLLM is the most widely deployed open-source LLM inference server for production workloads. Its two core innovations are PagedAttention (treating the KV cache like virtual memory with pages, eliminating fragmentation and enabling 10-24x more throughput than naive implementations) and continuous batching (dynamically grouping incoming requests into batches as they arrive, rather than waiting for a fixed batch to fill).

Key vLLM architecture components

• LLM Engine: The core scheduling and execution loop — receives requests, schedules them into batches, runs forward passes on GPU
• Block Manager: Manages the paged KV cache — allocates, swaps, and frees KV blocks across GPU and CPU memory
• Worker processes: One worker per GPU (or tensor-parallel group), each running a model shard and communicating via NCCL
• OpenAI-compatible API: vLLM exposes a REST API compatible with the OpenAI Chat Completions and Embeddings endpoints — drop-in replacement

SGLang: structured generation and agentic workloads

SGLang (Structured Generation Language) is optimized for workloads that require constrained decoding — JSON schema enforcement, regex matching, grammar-constrained generation — and for agentic use cases where the same context is reused across many generation calls. Its RadixAttention mechanism reuses KV cache prefixes shared across multiple requests, making it significantly more efficient than vLLM for multi-turn agent loops and batch evaluation tasks.

Ollama: local development and edge deployment

Ollama packages model weights, configuration, and the inference engine into a single binary with a clean REST API. It's the dominant tool for running LLMs locally (Mac, Linux, Windows with GPU or CPU), development workflows, and edge deployments where cloud API calls are not viable due to latency, cost, or privacy. Ollama supports GGUF quantized models via llama.cpp under the hood, plus a growing set of natively supported architectures.

Inference serving framework comparison

Multi-model serving with an LLM proxy

Production systems often need to route requests to different models based on task type, cost, latency, or capability. An LLM proxy (LiteLLM, AI Gateway) sits in front of multiple inference backends and provides a unified OpenAI-compatible API surface. Your architecture diagram should show the proxy as the single entry point, with routing logic and backend connections to both self-hosted vLLM/SGLang instances and external API providers.

Frequently asked questions about open-source LLM deployment

What is vLLM and how does PagedAttention work?

vLLM is an open-source LLM inference server that achieves high throughput through PagedAttention — a memory management technique that treats the KV cache (the key-value tensors the transformer computes during generation) like virtual memory in an OS. Instead of pre-allocating a contiguous memory block per request (which wastes GPU memory due to fragmentation), PagedAttention stores KV cache in fixed-size "pages" that can be allocated, shared, and reclaimed dynamically. This allows vLLM to batch far more requests in parallel, achieving 10–24× higher throughput than naive implementations.

How many GPUs do I need to serve a 70B parameter model?

A 70B parameter model in bfloat16 requires ~140GB of GPU memory for model weights alone. The minimum viable configuration is 2× H100 (80GB each, tensor parallel=2) for model weights with minimal KV cache. For production throughput (handling concurrent requests), 4× H100 or 4× A100 (80GB) is more practical, leaving enough memory for KV cache. Using 4-bit quantization (GPTQ, AWQ) reduces weights to ~35GB, making a single A100 80GB viable for development.

When should I use self-hosted LLMs vs the Anthropic or OpenAI API?

Self-hosted open-source LLMs make sense when: (1) your data cannot leave your infrastructure (healthcare, finance, government), (2) you have extremely high token volume where the cost delta justifies GPU investment (typically >$50K/month in API spend), (3) you need guaranteed SLAs without rate limits or outage dependency, or (4) you need fine-tuning on proprietary data. For most teams, commercial APIs are more cost-effective until one of these thresholds is hit.

Related guides: LLM architecture diagrams, AI gateway architecture, RAG architecture diagram, and LLM deployment use cases.