1 Repositories
Rust bytepiece Libraries
The Bytepiece Tokenizer Implemented in Rust.
bytepiece Implementation of Su's bytepiece. Bytepiece is a new tokenize method, which uses UTF-8 Byte as unigram to process text. It needs little prep
11 Oct 2, 2023