AINetflix Tech Blogabout 2 hours ago
In-House LLM Serving at Netflix
12 min read
Netflix built an in-house LLM serving platform using vLLM and NVIDIA Triton Inference Server. The system runs inside Netflix's existing production environment, not a separate ML silo. Key decisions included engine selection, model packaging, API surface design, and deployment strategy. Production revealed gaps like silent response_format dropping and observability issues with vLLM metrics.
Level
Hype check
Tap to vote and see what everyone thinks.
In this storyNetflix
Summary by ByteBrief
