TGINSIGHT POST
@DataScienceArchive
Data Science Archive
来自 Huggingface 的 tokenizer,Rust 实现,确实速度惊人。https://github.com/huggingface/tokenizers