← 返回 Siami 首頁

Gigatoken:比 HuggingFace 快 1000 倍的開源 Rust Tokenizer

▲ 275 💬 50
Gigatoken:比 HuggingFace 快 1000 倍的開源 Rust Tokenizer

Gigatoken:把 LLM 的第一道關卡從 MB/s 推上 GB/s

一句話總結:獨立開發者 Marcel Røed 釋出開源 Rust 函式庫 Gigatoken,宣稱在多數 tokenizer × CPU 組合下,比 HuggingFace Tokenizers 快 280~1,353 倍,對 OpenAI 的 tiktoken 也能快 50~700 倍,而且可直接當 drop-in replacement 使用。

為什麼這件事重要

Tokenizer 是大型語言模型(LLM)訓練與推論的「第一關」——每一筆輸入文字在進入模型前,都要先被切成 token。一般開發者通常不會特別關注這個環節,因為 HuggingFace Tokenizers 跟 OpenAI 的 tiktoken 已經非常方便、堪稱業界標準。

但 Marcel Røed 這次丟出的 Gigatoken 直接把這個大家習以為常的環節,從 MB/s 等級推上 GB/s 等級——而且是用單機 CPU 就能達到。在 AMD EPYC 144 核機器上對 GPT-2 tokenizer 量測到 24.53 GB/s,對比 HuggingFace Tokenizers 的 24.8 MB/s,整整快 989 倍;在 Apple M4 Max 上對 OLMo 2/3 tokenizer 達到 1,299 倍加速。這種等級的差距代表:以前要花一整天跑完的 tokenization pipeline,現在只要幾分鐘就能結束。

編按:在作者的 benchmark 裡,以 EPYC 144 核的速率計算,整個 Common Crawl(130 兆 tokens)只要 6.5 小時就能 tokenize 完——這個規模以前只有擁有大型分散式叢集的組織才有可能辦到。


開發者本人怎麼說

Gigatoken 作者 Marcel Røed 在 X 上親自發布這個專案:

「Introducing the world’s fastest tokenizer implementation, Gigatoken! 這些 baseline 本來就已經是多執行緒的 Rust 實作了!」—— @marcelroed

連史丹佛 CS336(Language Models from Scratch)授課教師 Percy Liang 也轉推表示:

「⚡! Tokenization is the first thing you do in CS336.」—— @percyliang

數據解讀 / 質疑

Gigatoken 的 benchmark 看起來驚人,但魔鬼藏在細節裡:

  • HF tokenizers 用 encode_batch_fast 只取前 100 MB、tiktoken 取前 1 GB,Gigatoken 卻是「對整個 11.9 GB 檔案直接編碼、還要自己找 split 邊界、自動做平行化」。換句話說,HF 跟 tiktoken 之所以「比較慢」,有一部分是因為它們先放棄處理大批量資料來換取通用性。Gigatoken 選擇直接 hardcode 場景換取極限效能,所以真實差距可能沒有一千倍那麼懸殊。
  • SentencePiece-based tokenizer(如 Gemma 系列、ModernBERT)在 Gigatoken 上只快 7~17 倍,相對其他 BPE tokenizer 的數百倍落差非常大。這代表如果你用的是 Google 的模型,Gigatoken 的吸引力會大幅下降。
  • 作者在 README 揭露了 AI 協助的部分,包含「最後約 4× 效能是靠消滅分支、改善 pretoken cache hierarchy」。換言之,數百倍到一千倍的主加速來自手工 SIMD 程式設計,最後的 4× 才由 AI 協助完成,這對「AI 對系統程式設計的實際貢獻」是個有意思的案例。
  • 目前不支援 Windows(建議用 WSL)、沒有 file sink、WordPiece 也還沒做。對 Windows-only 的開發者或特定模型族(BERT)來說,現在還不能直接切換過去。

Hacker News 上的討論

HN 討論串 275 點、50 個留言,圍繞「到底什麼場景會被 tokenizer 速度綁住?」展開:

  • author: “fwip” text: “What sort of setups do people have that are bounded by the speed of the tokenizer?” upvotes: 0 depth: 0
  • author: “rhdunn” text: “It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches. It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I’m not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.” upvotes: 0 depth: 1
  • author: “andersa” text: “Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!” upvotes: 0 depth: 1
  • author: “marcelroed” text: “Author here! In my case it’s mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM. From what I can tell it’s also useful for inference when considering time-to-…” upvotes: 0 depth: 1
  • author: “wren6991” text: ”> I’m not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you’re seeking for the exact token index having determi…” upvotes: 0 depth: 2

怎麼用?(最簡單示範)

pip install gigatoken
import gigatoken as gt

# 相容模式:把既有的 HuggingFace tokenizer 包起來
hf_tokenizer = ...
tokenizer = gt.Tokenizer(hf_tokenizer).as_hf()
tokens = tokenizer.encode_batch(["This is a test string", "And here is another"])

或是用原生 API(最快、但要傳檔案路徑):

import gigatoken as gt

tokenizer = gt.Tokenizer("Qwen/Qwen3-8B")
file_source = gt.TextFileSource(["owt_train.txt"], separator=b"")
tokens = tokenizer.encode_files(file_source)

主要 benchmark 數據(節錄)

TokenizerGigatokenHF tokenizersvs HF
GPT-2 (EPYC 144 核)24.53 GB/s24.8 MB/s989×
Phi-4 (EPYC 144 核)24.00 GB/s29.9 MB/s801×
OLMo 2/3 (M4 Max)7.56 GB/s5.8 MB/s1,299×
Qwen 3 (M4 Max)6.36 GB/s6.9 MB/s918×
DeepSeek V3/R1/V4 (M4 Max)5.68 GB/s7.2 MB/s788×
Gemma 4 (EPYC 144 核)4.82 GB/s334.1 MB/s14×
Mistral 7B v0.3 (EPYC 144 核)3.57 GB/s354.7 MB/s10×

完整 benchmark 涵蓋 23 種主流 tokenizer(GPT-2、Qwen 2/3/3.5/3.6、Llama 3/3.1/3.2/3.3/4、DeepSeek V3/R1/V4、GLM 4/5、Kimi K2、Phi-4 系列、Gemma 系列、Nemotron 3、ModernBERT、CodeLlama、TinyLlama 等),跨三種 CPU(AMD EPYC 144 核、Apple M4 Max 16 核、AMD Ryzen 7 9800X3D 8 核 16 緒)。


已知限制

  • Windows 測試不完整,目前建議用 WSL。
  • WordPiece 尚未支援。
  • SentencePiece-based tokenizer(Gemma、ModernBERT)優化程度遠不及 BPE 類。
  • Python iteration 使用 ABI3 binding,比專屬 CPython API 慢,未來打算對每個 Python 版本特化。

編按:本文綜合整理自 Gigatoken GitHub Repository、Hacker News 討論串、Marcel Røed 的 X 推文(@marcelroed),以及 AIWeekly 與 Hyper.ai 的二手報導,並加入 Siami 編輯部觀點與分析。


延伸閱讀

網友熱門留言 (5)

#1 fwip 0
What sort of setups do people have that are bounded by the speed of the tokenizer?
#2 rhdunn 0
It can be useful for checking input token usage before sending it to the model, e.g. preventing calls above a given token bound or grouping requests into batches. It can also be used by the LLMs to provide the input and output token counts on the different APIs, though I'm not sure if this is how llama.cpp or other OpenAI-like APIs calculate the input/output tokens of a request.
#3 andersa 0
Wait, since when does it matter whether something being hyper-optimized is useful? The computer going brrrr on an interesting problem is in itself the goal!
#4 marcelroed 0
Author here! In my case it's mostly pretraining experiments, where you might want to change your data mixture/filtering/processing of training data, and splits are usually done at a token-level instead of a text level. In this case we usually run for days on a huge number of CPUs to finish tokenizing something like DCLM. From what I can tell it's also useful for inference when considering time-to-...
#5 wren6991 0
> I'm not sure about the proprietary inference engines, but in the open source ones tokenization is done before looking up if a text sequence is present in the KV-cache Is this necessary? Tokenisation is deterministic, so for a hit/miss check you can lookup on (a hash of) the source text instead of the tokens. You only need the tokens once you're seeking for the exact token index having determi...