In-House LLM Serving at Netflix
Netflix has developed an in-house LLM serving platform using vLLM and Triton to integrate machine learning models directly into their production environment, focusing on engine selection, model packaging, API design, and deployment strategies to optimize performance and flexibility.
MAIN POINTS
- Netflix runs LLMs in-house, integrating them into their production environment instead of using hosted APIs.
- The platform uses vLLM and Triton for model deployment, offering a unified interface for various ML models.
- Deployment strategies include Red-Black and Versioned, addressing coordination and cost challenges.
- Operational challenges included model caching, boot sequence, and unified metrics endpoint for observability.
TAKEAWAYS
- vLLM was chosen for its operational fit, extensibility, and familiarity among ML practitioners.
- The OpenAI-compatible API was adopted to streamline the transition from experimentation to production.
- Custom logits processors were optimized for batch-level execution to overcome scaling bottlenecks.
- Future improvements focus on system prompt compression, asynchronous scheduling, and vectorized logits processors.