Serving Large Language Models in Production: Patterns, Tools, and Scaling Tactics

Modern enterprises are racing to deploy large language models (LLMs) like GPT-4, Llama 2, and Mistral in production. But serving these models reliably and efficiently is a distinct engineering challenge in 2024: LLMs demand high throughput, low latency, dynamic scaling, and robust observability. If you want to avoid skyrocketing inference costs and unpredictable downtime, you need a production-grade serving stack, not just a Dockerized Hugging Face model.

What Is LLM Model Serving? (With Real Config Example)

LLM model serving is the process of deploying, scaling, and managing large neural network models behind APIs for low-latency, high-reliability inference. Unlike batch offline jobs, serving must handle unpredictable request bursts, resource contention, and strict latency SLAs—often on GPUs or specialized accelerators.

Here's a minimal but production-relevant config for serving Llama 2-13B using vLLM (v0.3.2) with Hugging Face Transformers (v4.39.0) on a Kubernetes cluster with NVIDIA GPUs:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama2-vllm
spec:
  replicas: 4
  template:
    spec:
      containers:
        - name: vllm-server
          image: vllm/vllm-openai:v0.3.2-cuda12.1-py3.10
          args: ["--model", "meta-llama/Llama-2-13b-chat-hf", "--tensor-parallel-size", "2"]
          resources:
            limits:
              nvidia.com/gpu: 2
            requests:
              cpu: "8"
              memory: "64Gi"
          env:
            - name: HF_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: token
      nodeSelector:
        kubernetes.io/instance-type: "p4d.24xlarge"

This setup delivers ~50–70 tokens/sec per replica for Llama 2-13B (with FP16), assuming AWS p4d (NVIDIA A100) nodes. Inference throughput and latency vary significantly by model, quantization, and parallelism.

Key insight: Production LLM serving hinges on robust GPU scheduling, autoscaling, and resilient API interfaces, not just containerized model code.

Step 1: Choose the Right Model Serving Framework

Why Framework Choice Matters

The serving stack dictates your latency, scalability, and operational complexity. For LLMs in 2024, the dominant frameworks are:

• vLLM (0.3.x+): State-of-the-art for OpenAI-style APIs; supports advanced batching, paged attention, and tensor parallelism. Best for high QPS and multi-tenant inference.
• TGI (Text Generation Inference, v1.1.x+): Hugging Face's optimized LLM server, solid for production REST/gRPC APIs and easy integration with HF Hub.
• FastChat (v0.2.33+): Good for research and chat UIs, less robust for high-scale production.
• Triton Inference Server (23.12+): General-purpose NVIDIA stack; great for mixed models (vision, tabular, LLM), but requires more tuning.

Example: vLLM vs. TGI in Production

• vLLM delivers 30-50% higher throughput vs. TGI for Llama 2-70B under batch loads (see vLLM benchmarks here).
• TGI is easier to deploy with Hugging Face Hub integration and offers good multi-model support, but with slightly higher overhead.

Key insight: For high-throughput LLM APIs, I recommend vLLM; for multi-model serving or Hugging Face integration, TGI is a solid choice.

Step 2: Optimize Hardware Utilization and Auto-Scaling

Efficient GPU Allocation

LLMs are GPU-hungry. Deploying a 70B model in FP16 may require 2–8 A100 GPUs, depending on sharding and tensor parallelism. Idle GPU time wastes dollars. To optimize:

1. Right-size nodes. On AWS, use p4d.24xlarge (8×A100, 1.1TB RAM) or p5.48xlarge (8×H100) for larger models. For smaller LLMs (e.g. Mistral 7B), g5.2xlarge (1×A10G) or g6.xlarge (L4) suffice.
2. Enable GPU sharing using Kubernetes device plugins like NVIDIA's k8s-device-plugin (v0.14.0+) with MIG or Kubeshare for fine-grained partitioning.
3. Use Kubernetes HPA/VPA (autoscalers) with custom GPU metrics. For vLLM, expose QPS and queue length via Prometheus and autoscale deployments accordingly.

Example: GPU-Aware Horizontal Pod Autoscaler (HPA)

apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
  name: llama2-vllm-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: llama2-vllm
  minReplicas: 2
  maxReplicas: 8
  metrics:
    - type: Pods
      pods:
        metric:
          name: gpu_utilization
        target:
          type: AverageValue
          averageValue: 75

Key insight: GPU-aware autoscaling is essential—CPU/memory metrics alone are useless for LLMs.

Step 3: Implement Robust API Gateways and Request Management

API Gateway Patterns

LLM APIs are susceptible to traffic spikes, prompt injection, and multi-tenant abuse. You need:

1. API Gateway (e.g., Kong Gateway 3.x or Envoy 1.29): For rate limiting, JWT auth, and routing. Kong's rate-limiting plugin (v3.1+) supports Redis-backed global quotas, ideal for SaaS LLM APIs.
2. Queueing and Batching: vLLM and TGI support dynamic batching out of the box. For explicit control, use Redis Streams or Kafka between the gateway and model server.
3. Observability: Use Prometheus + Grafana (v9+) to track latency, QPS, model errors, and queue depth. I recommend OpenTelemetry (v1.18+) for distributed tracing.

Example: Kong Declarative Config for LLM API

_format_version: "3.0"
services:
  - name: llama2-inference
    url: http://llama2-vllm:8000/v1/completions
    routes:
      - name: completions-route
        paths:
          - /v1/completions
plugins:
  - name: rate-limiting
    config:
      second: 10
      hour: 10000
      policy: redis

Key insight: API gateways are mandatory for LLM API safety, enforceable quotas, and multi-tenant governance.

Step 4: Monitor, Profile, and Optimize Inference Latency

Real-Time Monitoring and Profiling

In production, model serving failures are often due to resource contention, batch starvation, or memory leaks. You need deep, real-time monitoring:

1. Prometheus Metrics: Scrape vLLM/TGI internals (/metrics endpoint) for inference_latency_seconds, active_requests, and gpu_memory_utilization gauges.
2. Profiling: Use NVIDIA Nsight Systems (2024.1+) for kernel-level GPU profiling, or PyTorch Profiler (2.1+) if you customize model code.
3. Alerting: Set up Grafana alerts for p95 latency > 2s, GPU utilization < 40% (waste), and OOM errors.

Example Prometheus Query

max_over_time(inference_latency_seconds{job="llama2-vllm"}[5m])

Latency Benchmarks (2024 Reference)

• Llama 2-13B: 50–70 tokens/sec per A100 GPU (vLLM, batch size 8, FP16)
• Llama 2-70B: 15–20 tokens/sec per A100 GPU (TGI, batch size 4, FP16)
• Mistral 7B: 80–120 tokens/sec per A10G (vLLM, batch size 16, INT8)

Key insight: Latency and throughput bottlenecks are almost always hardware or batch-size related — not model code bugs.

Step 5: Secure and Version LLM Models for Production

Model Versioning and Access Control

1. Model Registry: Use MLflow Model Registry (v2.10+) or S3/GCS with strict versioned folders. Example: s3://llm-models/llama2/13b/v1.0/.
2. Access Control: Restrict access to model artifacts via IAM policies, KMS encryption, and network policies (e.g., Kubernetes NetworkPolicy CRDs).
3. Supply Chain Security: Sign and verify model artifacts (see ModelScan and SLSA Level 2+ compliance).

Example: MLflow Model Registry Entry

import mlflow
mlflow.set_tracking_uri('https://mlflow-prod.company.com')
mlflow.register_model(
    "runs:/bda123456789/meta-llama/Llama-2-13b-chat-hf", "Llama2-13B-Prod"
)

Key insight: Treat LLM model artifacts as sensitive software assets—govern them with the same rigor as container images or source code.

Production LLM Serving Frameworks: Comparison Table

Framework	Best Use Case	GPU Utilization	API Support	Multi-Model	Observability	Community/Support
vLLM (0.3.x)	High-QPS LLM inference	Excellent	OpenAI-compatible	Limited	Good	Fast-growing
TGI (1.1.x)	Hugging Face integration	Very Good	REST, gRPC	Yes	Good	Official HF
Triton (23.12+)	Mixed models, vision+NLP	Good	REST, gRPC	Yes	Advanced	NVIDIA Enterprise
FastChat (0.2.33+)	R&D, chat UI prototyping	Fair	OpenAI-compatible	Yes	Basic	Community-driven

Key insight: For most production OpenAI-compatible LLM APIs, vLLM is the current throughput and efficiency leader; use TGI for Hugging Face-centric shops.

Frequently Asked Questions

Q: How do I scale LLM inference cost-effectively in the cloud? A: Use right-sized GPU nodes (e.g., AWS p4d for 13B+ models, g5/g6 for 7B), enable spot instance pools, and autoscale pods strictly by GPU utilization and queue depth to avoid overprovisioning.

Q: What are the main causes of high LLM API latency? A: The top causes are under-provisioned GPU resources, overly large batch sizes, network bottlenecks between API gateway and model server, or cold-starts from scaling down to zero.

Q: How do I secure access to production LLM models? A: Restrict artifact store access with IAM/KMS, enforce signed model binaries, audit all API traffic, and use API gateways with rate limiting and JWT authentication to control client access.

Key Takeaways

• Always choose a modern LLM serving stack (vLLM, TGI, Triton) and avoid DIY Flask or FastAPI wrappers for anything beyond prototype scale.
• Autoscale deployments strictly on GPU utilization and API queue metrics; CPU/mem-based scaling will fail for LLMs.
• Use API gateways (Kong, Envoy) for safety, rate limiting, and multi-tenant quota enforcement.
• Monitor p95/p99 latency and GPU metrics with Prometheus+Grafana; set alerts for resource saturation and OOMs.
• Store models in a versioned registry (MLflow, S3/GCS) and control access as you would any sensitive binary artifact.
• Benchmark real-world latency/throughput with your prompts and workloads — vendor benchmarks rarely match production reality.