WeSearch

ztok — a fast multithreaded tokenizer in Zig that loads tiktoken / HF / SentencePiece and is 2–5× faster

· 0 reactions · 0 comments · 22 views
Original article
r/LocalLLaMA
Read full at r/LocalLLaMA →
Anonymous · no account needed
Share 𝕏 Facebook Reddit LinkedIn Threads WhatsApp Bluesky Mastodon Email

Discussion

0 comments

More from r/LocalLLaMA