AIMarkTechPostabout 7 hours ago
Gigatoken Encodes Text at 24.53 GB/s
6 min read
Marcel Rød released Gigatoken, a Rust BPE tokenizer under MIT license, encoding text at 24.53 GB/s on a 144-core AMD EPYC 9565. That is 989x faster than HuggingFace tokenizers and 681x faster than tiktoken. Gains come from a hand-written SWAR pretokenizer and pretoken caching, not a faster BPE merge loop.
Level
Hype check
Tap to vote and see what everyone thinks.
Summary by ByteBrief
