Hey @guillaume-be, I'm having a little trouble using rust-tokenizers within a multi-process data loading pipeline. In particular, with the RobertaTokenizer, the issue is that it's not Send or Sync due to the use of Rc and RefCell, so currently what I have to do is instantiate the tokenizer within each worker thread after they have been spawned (see here, for example).

This is a little cumbersome, and also isn't ideal because the cache can't be shared between threads.

Is there any reason not to use something like a RwLock instead?

It would help if the tokenisers could maintain and output character offset information, with respect to the original input. This would help greatly to align information back together at the end of an NLP pipeline.

Possible points for discussion:

• Are utf-8 byte offsets or do we also want unicode graphemes? (with https://github.com/unicode-rs/unicode-segmentation for instance).

(this continues part of the discussion from guillaume-be/rust-bert#29)

This (draft) PR is the result of #14 , it implements offset support for all the tokenisers.

As a lot of the original logic consisted of simple string replacements, losing the alignment with the original text, various internal functions had to be modified or rewritten completely. This code therefore constitutes a major refactoring. All logic is rewritten to use new Token (or TokenRef, holding &str instead of String) structures. Each token holds an offset and a mask. The mask tells something about the role of the token (is it a special token? out of vocabulary? is it a continuation subtoken?).

The core of the tokenisers is formed by the following new shared functions in tokenization_utils:

• split_on_char() - split on character (takes a testing function as argument)
• split_on_substr() - split on substring (takes a testing function as argument)
• split_on_regex() - split on regular expression (takes a regex as argument)
• split_on_bpe_pairs() - split on bpe pairs, this abstracts over parts of the GPT2/Roberta/ctrl tokenizers (takes a bpe function as argument).

The various higher-order functions all use these:

• split_on_regex_with_lookahead() - abstracts a part of the GPT2/Roberta tokenizers
• split_on_punct- split on punctuation
• whitespace_tokenize- split on whitespace
• tokenize_cjk_chars()- split on CJK characters

All these function take a single Token or TokenRef (which may correspond to an entire text/sentence/whatever!) and return either Vec<Token> or Vec<TokenRef> (depending on whether I could get away with not doing any copies). The general idea is that tokens are being constantly decomposed into smaller subtokens by chaining the functions in an appropriate order.

Each of the tokenisers has a tokenize_into_tokens() -> Vec<Token> method coming from the Tokenizer trait. tokenize_with_offsets() is a small wrapper around that which decomposes the Vec<Token> into a 3-tuple (Vec<String>,Vec<Offset>,Vec<Mask>). The original tokenize() function remains for backward compatibility but simply ignores the offset and mask information (so has no performance benefits).

As for performance, though certain things got optimized and will surely help, there is also added complexity that will have some impact. Things can probably still be improved (and cleaned up) too.

Because of all the extensive changes, the deeper code no longer maps one-on-one to the original transformer's tokenizers (assuming it did in the first place?), still the behaviour should be almost identical.

This is a draft pull request at this stage, because there are still some loose ends to fix. The main issue is that I currently have failures on a few tests which I'm not sure about (I'll address it in a new comment because this one is already getting pretty long).

Master PR: #76 Blocked by: #79

Motivation

1. Not every FS path is a valid utf-8 string.
2. AsRef<Path> is strictly more acceptable with same level of correctness: you can use &str, Path and PathBuf.

Implementation

• [x] Use AsRef<Path> instead of &str.
• [x] Remove unnecessary type casts and unwraps from tests.

@veta666, this one is also on you.

This PR handles the addition of special token maps for all vocabularies and tokenizers.

• Normalizes the crate interface to special tokens
• Replaces the unknown_token field of vocabs by special_token_map (this common token can still be accessed directly via get_unknown_token())
• No longer hardcodes vocabs special tokens to static strings
• Refactors some duplicated code to read files

Todo:

• [x] Update vocabs
• [x] Update tokenizers
• [x] Add special token map constructors to tokenizers
• [x] Fix unit tests
• [ ] Fix integration tests

Hello, this is my initial NLLB tokenizer support MR!

Tokenizer config: https://huggingface.co/facebook/nllb-200-1.3B/blob/main/tokenizer_config.json

Special tokens: https://huggingface.co/facebook/nllb-200-1.3B/blob/main/special_tokens_map.json

Vocabulary + depth of unknown: https://huggingface.co/facebook/nllb-200-1.3B/blob/main/tokenizer.json

Unsolved questions:

1. ~~bos/cls/eos is hardcoded constants, but we have configuration file for them. Why doesn't rust-tokenizers use it?~~
2. ~~I copied some code from M2M module and python's source, but I am not entirely understand it. Is there some documentation for vocab alignment and JSON format?~~
3. tokenizer.json contains not only vocab, but also normalizer, pre_tokenizer, post_processor and meta information for the model. Should we do something with it?

Plan

• [x] Initial implementation.
• [x] Tests.
• [ ] Python bindings.
• [ ] Docs.

I'm not sure if this is where this is supposed to go but, at the highest level, I'm using rust-bert and trying to instantiate an NERModel with seemingly default XLMRoberta configurations like:

    let config = TokenClassificationConfig::new(
        ModelType::XLMRoberta,
        RemoteResource::from_pretrained(RobertaModelResources::XLM_ROBERTA_NER_EN),
        RemoteResource::from_pretrained(RobertaVocabResources::XLM_ROBERTA_NER_EN),
        RemoteResource::from_pretrained(RobertaConfigResources::XLM_ROBERTA_NER_EN),
        None,  //merges resource only relevant with ModelType::Roberta
        false, //lowercase
        None,  //strip_accents
        None,  //add_prefix_space
        LabelAggregationOption::Mode,
    );

    let ner_model = NERModel::new(config).unwrap();

and this is failing with:

TokenizerError("Error when loading vocabulary file, the file 
may be corrupted or does not match the expected format: incorrect tag")

That resource seems to be pointing to the sentencepiece.bpe.model file for this model and it seems to be parsed by ModelProto in this repo.

I'm not sure if:

1. I'm doing the wrong thing with rust-bert
2. The file hosted in the xlm-roberta dataset is out of date
3. This repo is out of date with a new update to the xlm-roberta dataset file

This issue is really only relevant for the last item (though I wouldn't know where to report the second item). Apologies if it's the first item but I'm wondering if you can offer any insight here!

Note I'm using the version of rust-bert on the master git branch, not the latest published crate (because I need 0.7 tch support).

Hi, thanks for the awesome libraries. I'm new to rust and the community is enthusiastic. I have some questions about the tokenizer library. How is rust-tokenizers compared to huggingface's tokenizer? (Performance, tokenized result etc.) Can I use them interchangably with rust-bert?

Dearest Maintainer,

I have been using rust-bert and my panic leads me here. I have a nonbreaking space \u{a0} https://en.wikipedia.org/wiki/Non-breaking_space . This appears to be valid and the chars are there. https://play.rust-lang.org/?version=stable&mode=debug&edition=2018&gist=865b0df5864e4d3b68004df0babd3afe

My error:

thread 'main' panicked at 'byte index 11 is not a char boundary; it is inside '\u{a0}' (bytes 10..12) of input.   We', /home/becker/.rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/src/rust/src/libcore/str/mod.rs:1920:47

it comes from 14: core::str::traits::<impl core::slice::SliceIndex for core::ops::range::Range>::index::{{closure}} 15: rust_tokenizers::preprocessing::tokenizer::tokenization_utils::split_on_regex_with_lookahead

I have looked over the code and i dont understand which index is off. it looks like it calls len_utf8 at a good spot.

example code to trigger the issue.

    let result = panic::catch_unwind(|| {
        let mut summarization_model =
            SummarizationModel::new(Default::default()).expect("summarization_model fail");
        let input = [text];
        summarization_model
            .summarize(" input.\u{a0} \u{a0}We")
            .join(" ")
    });

in the mean time i am going to preprocess \u{a0} to covert it to a space.

Any help in understanding this would be much appreciated.

Thanks again Becker

I have some issues regarding the clean_up_tokenization() function, it looks like what it basically does is strip some whitespace? I agree that the tokens itself should not contain leading or trailing whitespace. I think it could be implemented more efficiently and generically (= language independent)?

https://github.com/guillaume-be/rust-tokenizers/blob/master/main/src/preprocessing/tokenizer/base_tokenizer.rs#L130

.replace(" do not", " don't")

I don't like this part because it changes the actual text and doesn't just concern whitespace.

The current tokenizers don't have test cases where unicode chars are expanded in the normalization process with .nfkc(). This pr adds test cases and fixes the problem. Expanded unicode chars in the middle of the sentence were previously wrongly tokenized and unicode chars that were expanded at the end of a text fragment led to panics.

We've filled in the tests with the output of the tokenizers to get the tests to pass, but it's probably worth checking whether they look alright, as we pattern-matched the tests and don't have any insight as to what the output should be.

Master MR: https://github.com/guillaume-be/rust-tokenizers/pull/76

• [x] Add structural errors:
  • [x] Save source error type.
  • [x] Line location.
  • [x] Make reasonable error message.
• [ ] Fix tests ignored in tokenization_utils: they broke because line location in display message.

@veta666, could you finish this PR?

I've created a SentencePiece model using Python which results in a .model and a .vocab file. It is not possible to create a SentencePieceVocab from the later since Python does not seem to use protobuf but rather a plain text file. Here's an excerpt of my file:

<unk>	0
<s>	0
</s>	0
▁	-2.29038
s	-3.10405
l	-3.41047

I didn't find an option in the Python code for creating a protobuf vocab file so I wrote a parser. Unless I'm mistaken and did something wrong, would you like that code as a PR? I.e. something like:

impl SentencePieceVocab {
    ...
    
    pub fn from_vocab_txt_file(path: &str) -> Result<SentencePieceVocab, TokenizerError> { 
        ... 
    }
}

in rust-tokenizers/main/src/vocab/sentence_piece_vocab.rs