Token-count-based Batching: Faster, Cheaper Embedding Inference for Queries

Voyage AI by MongoDB introduced token-count-based batching for embedding inference, reducing GPU inference latency by 50% while using 3x fewer GPUs. The technique leverages padding removal in engines like vLLM and SGLang to batch short query requests efficiently, addressing memory-bound bottlenecks in search and retrieval systems.