It would be useful to add a sentence splitter, for instance, possibilities could be,

• Puntk sentence tokenizer from NLTK (needs pre-trained model)
• Unicode sentence boundaries from https://github.com/unicode-rs/unicode-segmentation/pull/24 (doesn't need a pre-trained model)
• investigate spacy implementation (likely needs pre-trained model)

First attempt at including the UnicodeSentenceTokenizer in the Python package. I have two issues that I am unsure how to resolve:

1. After using python3 setup.py develop to compile the lib which runs without errors, when trying to import the package in Python import _lib I get the following fatal error:

SystemError: Type does not define the tp_name field.
thread '<unnamed>' panicked at 'An error occurred while initializing class UnicodeSentenceTokenizer', ...
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5

2. In python/src/tokenize_sentence.rs I have resorted to creating a second base class BaseTokenize2 (see line 14) as I am unsure how to import BaseTokenize from python/src/tokenize.rs

This updates to the lastest PyO3, which allows using lifetimes in pymethods. As result tokenization in Python is a bit faster by avoiding string copies.

On master,

python3.7 benchmarks/bench_tokenizers.py
 Tokenizing 19924 documents
         Python re.findall(r'\b\w\w+\b', ...): 2.93s [31.0 MB/s, 2450 kWPS]
                RegexpTokenizer(r'\b\w\w+\b'): 1.96s [46.5 MB/s, 3671 kWPS]
   UnicodeSegmentTokenizer(word_bounds=False): 2.97s [30.7 MB/s, 2269 kWPS]
    UnicodeSegmentTokenizer(word_bounds=True): 3.58s [25.4 MB/s, 3182 kWPS]
                         VTextTokenizer('en'): 4.11s [22.1 MB/s, 2467 kWPS]
                        CharacterTokenizer(4): 7.73s [11.8 MB/s, 5927 kWPS]

after this PR,

# Tokenizing 19924 documents
         Python re.findall(r'\b\w\w+\b', ...): 2.92s [31.2 MB/s, 2460 kWPS]
                RegexpTokenizer(r'\b\w\w+\b'): 1.40s [64.8 MB/s, 5119 kWPS]
   UnicodeSegmentTokenizer(word_bounds=False): 2.48s [36.8 MB/s, 2721 kWPS]
    UnicodeSegmentTokenizer(word_bounds=True): 2.65s [34.3 MB/s, 4292 kWPS]
                         VTextTokenizer('en'): 3.32s [27.4 MB/s, 3053 kWPS]
                        CharacterTokenizer(4): 4.47s [20.4 MB/s, 10252 kWPS]

Currently CountVectorizer and HashingVectorizer mostly perform BOW token counting without the possibility to change the tokenizer or any other parameters.

While we intentionally won't support all the parameter that scikit-learn versions does (as these meta-estimators are doing too much), additional parametrization would be preferable.

• parametrization of the tokenizer will be addressed in #48

This makes it possible to use any object implementing the Tokenizer trait in Vectorizers,

    let tokenizer = VTextTokenizer::new("en");
    let vectorizer = CountVectorizer::new(&tokenizer);

Currently, we use the MurmurHash3 hash function from the rust-fasthash (to be more similar to scikit-learn implementation). That crate also supports a number of other hash functions,

City Hash Farm Hash Metro Hash Mum Hash Sea Hash Spooky Hash T1 Hash xx Hash

I'm not convinced hashing is currently the performance bottleneck, but in any case using a faster hash function such as xxhash would not hurt.

This would involve updating the text-vectorize crate and adding hasher parameter to the HashingVectorizer python estimator.

Another use case could to use different hash functions to reduce the effect of collisions Svenstrup et. al. 2017, discussed e.g. in https://stackoverflow.com/q/53767469/1791279

UnicodeSegmentTokenizer was meant to be a shorter version of Unicode segmentation tokenizer, but the name is not very explicit. Besides UnicodeSentenceTokenizer also uses unicode-segmentation crate which adds to the confusion. Maybe UnicodeWordTokenizer would be a better name?

This aims to build vtext for the wasm32 target.

Currently a blocker is the incompatible rustc-serialize create scattered though the dependencies. In particular,

• ndarray <0.13 (required by sprs: will be fixed in https://github.com/vbarrielle/sprs/pull/175)
• num-complex < 0.2 (used in sprs: will be fixed in https://github.com/vbarrielle/sprs/pull/164)

other issues may be discovered later on..

What would you think about adding float-cmp for doing float comparisons in test?

Clippy complains about float comparisons and I've run into it with other projects...

first time I've tried out this crate to deal with float comparisons in tests.

Follow up on #48, allows to select the tokenizer used in vectorizers.

As mentioned in https://github.com/rth/vtext/pull/48#issuecomment-488223434 this will require PyO3 O.7.0 that includes https://github.com/PyO3/pyo3/pull/461 (hopefully released in the coming days) or a way to avoid using explicitly defined lifetimes in python wrapper methods for vectorizers.

Currently, tokenizers return Vec<String> where tokens are be slices of the input string. Moving to Vec<&str> would remove one memory copy and is likely to help with run time.

This should be possible with PyO3 0.7.0 (not yet released) that will allow using lifetime specifiers in pymethods.

Bumps numpy from 1.17.3 to 1.22.0.

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

This adds a NTLKWordTokenizer which implements the default tokenizer from NLTK. The test suite from NLTK passes, however we are not handling one edge case due to the lack of the lookahead functionality in regex. I don't think it's worth adding another library as a dependency to address that, and marking it as a known limitation could be a workaround for now. That regexp is an enhancement proposed by NLTK on top of the classical Penn Treebank word tokenizer.

Currently this returns Vec<String> and I have struggled with making it return an iterator due to lifetime issues so far.

It's around 3x faster than the NLTK version in Python. It is very English specific and should probably not be used in other languages.

TODO:

• [ ] improve documentation

Implemented k-skip-n-grams. Has convenience functions for bigram, trigram, ngrams, everygrams and skipgrams.

Provides the same output as the equivalent nltk functions - although nltk does generate duplicates sometimes which are omitted here. The iterator consumes the input iterator only once and holds a window of items to generate the grams. The window is stepped forward as it consumes the input. It also correctly generates left or right padding if specified.

Currently the iterator outputs Vec<Vec<&str>>. I'm unsure if this is desirable or the function should join the items into a string to output Vec<String>. e.g. Currently it does: vec![vec!["One", "Two"], vec!["Two", "Three"] ...] but it might be desirable to have vec!["One Two", "Two Three"].

The nltk implementations tend to consume the input sequence multiple times and concatenate the output (for example everygram). I wanted the struct to consume the input iterator only once and provide an output. This however considerably complicated the code. I have tried to refractor it so it is as readable as much as possible but it would be good to get a second eye on it.

There is also still the question of how to chain the different components together (which links to #21). Currently the transform method takes an input iterator and provides an iterator as output which can be consumed by the user.

Todo:

• [x] Add Python interface
• [x] Benchmark against nltk
• [ ] Add function for character ngrams
• [x] Further documentation

It can happen that the tokenization results are unsatisfactory in some way, and the question is what should be the mechanism to customize/improve them. Whether it should be by, a) adding options make these optional improvements in the tokenizer. The issue with these is that some of these might be relevant to multiple tokenizers b) add a new step later in the pipeline. That's probably the best way to allow arbitrary customization. The issue is that some steps might be specific to the previous step, and adding them in the library might be confusing.

There is probably a balance that needs to be found between the two.

For instance,

1. PunctuationTokenizer,
   • currently doesn't take into account repeated punctuation

     >>> PunctuationTokenizer().tokenize("test!!!")                                                                                                     
     ['test!', '!', '!']
     

   • will tokenize abbreviations separated by . as separate sentence

     >>> PunctuationTokenizer().tokenize("W.T.O.")
     ['W.', 'T.', 'O.']
     

   both could probably be addressed by adding an option to force sentences to be longer than some minimal length (and otherwise append them to the previous token).

2. UnicodeSentenceTokenizer, will not tokenizer sentences separated by a punctuation without space e.g.,

   >>> UnicodeSentenceTokenizer().tokenize('One sentence.Another sentence.')
   ['One sentence.Another sentence.']
   

   That's a very common occurrence in actual text, and I think a workaround should be found (e.g. using an additional tokenization pass with a regex/punctuation tokenizer).

Generally it would be good to add some evaluation benchmarks to evaluation/ for sentence tokenization to evaluation/ folder.

3. UnicodeTokenizer is currently extended in VTextTokenizer (for lack of a better name), with a few additional rules. Maybe this could have been a separate token-processing step, particularly if one imagine that more rules could be added (or potentially even using an ML model).

From https://github.com/rth/vtext/pull/78#issuecomment-644009378 by @joshlk

I think it would be sensible to identify different languages throughout the package using ISO two-letter codes (e.g. en, fr, de ...).

In particular, we should implement this for the Snowball stemmer in python which currently uses the full language names.

I am also wondering if in Rust, we should use String for the language parameter or define an Enum e.g.

use vtext::lang

let stemmer = SnowballStemmerParams::default().lang(lang::en).build()

The latter is probably simpler, but it makes it a bit harder to extend e.g. if someone designs an custom estimator for a language not in the list (e.g. some ancient infrequently used language), they would have to create a new enum.

Also just to be consistent the parameter name would be "lang" not "language", right?