Hey @christophertrml ,

I've contributed a few times to the project, and I'd really like to continue to do so. I don't see a ton of options for NLP in Rust. I'd really like to make the library more uniform, and overall more flexible. Currently the tf-idf works but isn't easy to use in a meaningful way. The Naive-Bayes classifier has been in need of some work for a while now, and a logistic regression classifier has been a near-sighted goal since I first learned of the project. I could help handle issues and PRs, as well as expanding the feature set, and improving already existing features of the project if you give me permissions for the repository.

Let me know what you think,

Travis Sturzl

Hey @christophertrml,

In order to publish changes I need access on crates.io. When you have time, could you add me? See: https://doc.rust-lang.org/cargo/reference/publishing.html#cargo-owner

Thanks!

I get the message:

thread 'main' panicked at 'index out of bounds: the len is 3 but the index is 18446744073709551615', /c
heckout/src/libcore/slice/mod.rs:2041:10

The value seems to indicate an unsigned value dropping below zero.

Hi,

Would be cool to be able to serialize/store the trained model somehow, probably would be a good idea provide some API that can access internal structures for store/restore to be able to have independent serialization/storage engines.

Regards

I use rs-natural in a few of my hobby projects, and I notice that this project isn't on crates.io. This creates a problem if I myself would like to publish one of my projects that uses rs-natural to crates.io. Is there any reason that it is not? I know rust-stem is also not on crates.io, but issue #3 seems to address this. If this is the only thing holding it back I'd be more than happy to make a PR where rust-stem is included. There shouldn't be any licensing issues since both projects are MIT licensed.

When attempting to stem the word "ion," I got this panic:

thread 'main' panicked at 'attempt to subtract with overflow', /home/paul/.cargo/registry/src/github.com-1ecc6299db9ec823/natural-0.3.0/src/stem.rs:273:29

When trying to use padded ngrams I realized that the vectors weren't symmetrical. It seems like the tokens were appended in the wrong order. Here is an example:

get_ngram_with_padding("This is a test thing for ngram padding.", 5, "-")

Output:

[
  [
    ["-", "-", "-", "-", "This"], 
    ["-", "-", "-", "This", "is"], 
    ["-", "-", "This", "is", "a"], 
    ["-", "This", "is", "a", "test"],
    ["This", "is", "a", "test", "thing"],
    ["is", "a", "test", "thing", "for"], 
    ["a", "test", "thing", "for", "ngram"],
    ["test", "thing", "for", "ngram", "padding"], 
    ["thing", "for", "ngram", "padding", "-", "-", "-", "-"], 
    ["for", "ngram", "padding", "-", "-", "-"], 
    ["ngram", "padding", "-", "-"], 
    ["padding", "-"]
  ]
]

With this PR, the output is now:

[
    [
        ["-", "-", "-", "-", "This"],
        ["-", "-", "-", "This", "is"],
        ["-", "-", "This", "is", "a"],
        ["-", "This", "is", "a", "test"],
        ["This", "is", "a", "test", "thing"],
        ["is", "a", "test", "thing", "for"],
        ["a", "test", "thing", "for", "ngram"],
        ["test", "thing", "for", "ngram", "padding"],
        ["thing", "for", "ngram", "padding", "-"]
        ["for", "ngram", "padding", "-", "-"],
        ["ngram", "padding", "-", "-", "-"],
        ["padding", "-", "-", "-", "-"],
    ]
]

This isn't a major optimization, but you can simply just remove the HashMap label_probabilities from the classifier and just store the highest running probability. This way you don't need a HashMap or another loop. You simply just store the result_label and result_probability each time the probability is higher than the last highest probability stored in result_probability. Another solution I looked at is to use a vector of tuples Vector<(String, f32)> and just allocating the vector(construct using with_capacity) with a fixed size that is exactly the size of the amount of documents(self.count) to prevent reallocation, because you don't really need to access the label by the key and you aren't really using any methods of a HashMap that you'd be missing by using a Vector instead. But this is also unnecessary, because you're only passing back the highest label rather than all the labels and probabilities. This should reduce overhead of heap allocating a HashMap, storing unused values, and reduce overhead of having a second loop.

All tests are passing on my local.

Added Levenshtein implementation based on a public domain implementation[1]. Added couple of base tests and made the existing ones pass. I'm open for feedback :)

[1] http://hetland.org/coding/python/levenshtein.py

This crate could be highly valuable to the Data Privacy world. If we could teach our systems to flag data that contains sensitive content, (e.g.: PII, NPPI, PCI), we can use Privacy by Design to implement guardrails to avoid privacy breeches or misuse of data.

A suggested enhancement for such an application would be to calculate the chance of a data string belonging to a classification.

Example

// returns 0 to 100
pub fn match_classification(data_to_validate: String, category: String) -> u8 {
    let chance = 0; 

    /*
   calculating ...
   */

    chance
}

let chance = NaiveBayesClassifier::match_classification("ssn: 003-43-7621", "NPPI");
println!("This data is {} likely to be in the NPPI category.", chance); 

• replace deprecated range pattern
• remove unused imports
• make normalized_prob not mutable
• replace deprecated trim_right
• add auto-generated comments to Cargo.lock

As of now, it is impossible to have blank-padded ngrams since internally it assumes an empty string means no padding.

I've made this possible in this branch by making the pad member of NGram an Option<&'a str>. If this feature is wanted, I can make a PR.