AIXDAabout 2 hours ago
VRAM context setting doubles local AI speed
1 min read
Setting Ollama's num_ctx to 16K doubled tokens per second for Gemma 4 12B on an RTX 3070, from 14.4 at 128K to 32-33. Ollama pre-allocates VRAM for the full context window, wasting memory on small prompts.
Level
Hype check
Tap to vote and see what everyone thinks.
Summary by ByteBrief
