This isn't quite specific to lingua-rs but I've been looking into WebAssembly lately and it would be great to be able to use Lingua-rs into a wasm project. I did an initial test but it failed on a problem with bzip2-sys. I'll have to keep looking into it and let you know what I find but I thought you might be interested.

As the slightly ironic title hopefully conveys, I'm not very convinced of this idea, I'm just putting it here for the sake of discussion.

Motivation

Lingua stores language models (which really are just n-grams and their associated probabilities) as zipped JSON files in the binary. Depending on user preference either upfront at startup or lazily on demand, it will unzip, parse and then transform this data into the in-memory representation HashMap<Ngram, f64> used by part of the detection pipeline. That carries a certain computational cost, which is negligible for most use cases where the detector is reused and this initial cost can be offset over its lifetime as all following detections can use the already "warmed up" detector.

In certain environments however, for example serverless computing or when using WebAssembly in plugins for other software, the detector can not be used in such a way. In these cases our code (containing a call to Lingua) is very short-lived and no reuse over subsequent invocations is possible, meaning each time it has to be constructed from scratch. In these permanent "cold start" situations, the described startup cost ends up dominating the overall runtime needed for detection. In my case, where I invoke Lingua from C# by means of wasmtime, that results in taking 540 ms for detecting the language of a short text, most of which is spent in startup. The following flamegraph illustrates this nicely. You can see some time being spent deflating the zipped data, some more deserializing the JSON and then some constructing the hash map.

I would like to point out that this is not a weakness of Lingua. It's a problem that comes with the domain of language detection and I'm not aware of a library that has solved this. And as I mentioned, it's really only an issue in specialized use cases.

Solution

Idea

We could save a lot of time by embedding the language models into the binary in a way that is significantly cheaper to read. With rkyv's zero-copy deserialization it's possible to encode a data structure in a format that is the same as its in-memory representation. This means there's no additional work to be done, as language models can be read directly from the binary's data segment where they are stored.

Implementation

I hacked the demo together by adding a build script for language crates. It reads the JSON models for a language and writes them to binary files in their rkyv encoding. These bytes are then simply statically included and accessed at runtime. Instead of exposing a Dir<'static>, every language crate now has a [Option<&'static [u8]>; 5], an array of bytes for all 5 possible n-gram types. The bytes are just the archived representation of HashMap<Ngram, f64>.

To read the JSON models from the build script I had to move the Fraction and Ngram types out into a separate common crate used by both the build script and Lingua.

Results

I measured the C# scenario I described earlier.

• Benchmark implemented with BenchmarkDotNet on .NET 6.0
• Uses wasmtime to call a WebAssembly module built with lingua to detect the language of a single short text
• Instance of WebAssembly module (its memory space) is destroyed after each iteration, while the JIT-compiled module (bytecode) itself can be reused
• Only 4 languages were included in the binary

The effect of zero-copy deserialization of the language models in this cold start scenario is as follows:

| | before | after | change | |-------------|--------|--------|--------------| | Binary size | 8.8 MB | 33 MB | x3.75 bigger | | Time | 543 ms | 8.9 ms | x61 faster |

The new flamegraph looks much less informative, mainly because there's just not a whole lot going on anymore. Some time is spent on what I think is language runtime stuff and the rest in the actual detection, there is nothing obvious to optimize away anymore.

These results are impressive both in the positive and negative sense. The build takes much longer now because it's serializing a bunch of pretty sizable files to disk for all languages. And the actual binary grows to a size that is no longer convenient to deliver in all environments, for example if you imagine it having to be downloaded over a mobile connection to a phone. On the other hand it's pretty amazing to see that you can initialize and run a sophisticated language detection suite in just under 9 ms.

Hi,

I'm evaluating the lingua-rs libraray and I discovered a long running time of the following program:

#[macro_use]
extern crate lazy_static;

use std::env;
use std::fs;

use lingua::{Language, LanguageDetector, LanguageDetectorBuilder};

lazy_static! {
    static ref DETECTOR: LanguageDetector = {
        LanguageDetectorBuilder::from_languages(&[
            Language::Dutch,
            Language::English,
            Language::French,
            Language::German,
            Language::Hungarian,
            Language::Italian,
            Language::Portuguese,
            Language::Russian,
            Language::Spanish,
            Language::Finnish,
            Language::Swedish,
        ])
        .build()
    };
}

fn main() {
    let args: Vec<String> = env::args().collect();
    if args.len() != 2 {
        eprintln!("missing argument");
        std::process::exit(1);
    }

    let filename = args.get(1).unwrap();
    match fs::read_to_string(filename) {
        Ok(content) => match DETECTOR.detect_language_of(&content) {
            Some(lang) => println!("{},{}", filename, lang.iso_code_639_3()),
            _ => println!("{},", filename),
        },
        _ => println!("{},", filename),
    };

    std::process::exit(0);
}

First I thought the reason for the long running time could be the language detector. But even after moving this part into an lazy_static block, the runtime is very slow. Is a running time of 9.82 seconds to be expected with lingua for an article of 26,712 words? Are there ways to speed up the program?

I would welcome your response, Nico

Hi,

I am trying to embed this into my program but I am seeing a very long startup time setting up the detector. Can I ask (on the high level) why that is (other than the files being larger than other libraries due to the use of 5-grams). Anything we can do to speed up the initialization time?

Happy to help contribute if needed.

Oddly I had to add a feature flag to lingua-rs to disable wasm so that I could use it in a WebAssembly project. It was't actually disabling wasm as much as it's disabling the wasm-bindgen. I'm running it in a non-browser setting and the code generated from bindgen causes problems. Let me know if you'd like me to put in a PR. It was only a couple of minor changes.

println!("{:?} -- {} -- {:?}", i, i.iso_code_639_3(), i.iso_code_639_3());
Italian -- ita -- ITA

Debug print shouldn't be giving a different value than a normal print.

Also it seems impossible to do anything with the value returned:

30 |       if i == "Italian" { println!("Match"); }
   |            ^^ no implementation for `Language == str
   
30 |       if i.to_owned() == "Italian" { println!("Match"); }
   |                          ^^^^^^^^^ expected enum `Language`, found `&str`
   
30 |       if i.display() == "Italian" { println!("Match"); }
   |            ^^^^^^^ method not found in `&Language`

It appears that we're missing display()... Do I just need to use the strum_macros crate? Should I attempt to make a PR to add the display() trait?

The accuracy is looking really good; I've tested so far with about 50 subtitles (grabbing 10 lines evenly distributed throughout the file) and it's 100% accurate so far. Thanks for such an awesome library, and your help using it. :)

Hello,

This crate looks awesome! I plan on using it to detect languages of subtitles. Thanks for writing this crate! I added it to cargo.toml, but I get this on cargo build:

error[E0308]: mismatched types
   --> /home/michael/.cargo/registry/src/github.com-1ecc6299db9ec823/lingua-1.3.2/src/json.rs:265:32
    |
262 | fn get_language_models_directory(language: Language) -> Dir<'static> {
    |                                                         ------------ expected `include_dir::Dir<'static>` because of return type
...
265 |         Language::Afrikaans => AFRIKAANS_MODELS_DIRECTORY,
    |                                ^^^^^^^^^^^^^^^^^^^^^^^^^^ expected struct `include_dir::Dir`, found struct `include_dir::dir::Dir`
    |
    = note: perhaps two different versions of crate `include_dir` are being used?

For more information about this error, try `rustc --explain E0308`.

Here is my full cargo.toml dependency list:

clap = { version = "3.0", features = ["derive"] }
regex = "1"
tmdb = "3.0.0"
log = "0.4"
pretty_env_logger = "0.4.0"
walkdir = "2"
dirs = "3.0"
serde = { version = "1.0", features = ["derive"]   }
serde_yaml = "0.8"
md-5 = "0.9"
unicode-truncate = "0.2.0"
lazy_static = "1.4.0"
common-path = "1.0.0"
lingua = "1.3.2"

$ rustc --version
rustc 1.58.1 (db9d1b20b 2022-01-20)

I'm kind of a Rust noob so I'm not sure how to fix this...

I am trying to add a feature to dynamically add languages in lingua-node, and needed the ability to add language codes from the outside of Rust context.

I saw there is strum so I thought to add the macro into the isocode file and it works like a charm (used a regex find/replace to add this many annotations lol)! Added test make sure the features work.

Let me know if you have any questions on the PR.

Cheers!

One of the things I've been working on is an HTTP service wrapper around lingua. My implementation allows the caller to define the various pieces of a detector or just use the default for the service instance. If they specify invalid options (say, an unrecognized language or out of range minimum relative distance) I have to either validate the options manually before attempting to create the builder or catch the panic. If validation was done in LanguageDetectorBuilder::build and that returned a Result<LanguageDetector, Error> or the like I could rely on the library itself for validation.

When trying to run a file that uses lingua 1.0.2 as a dependency, compiling fails with the following error. This is a new rust project, with no other dependencies and just a println, as I just wanted to check out the library

Compiling lingua v1.0.2
error[E0603]: module `export` is private
   --> C:\Users\luana\.cargo\registry\src\github.com-1ecc6299db9ec823\lingua-1.0.2\src\ngram.rs:19:12
    |
19  | use serde::export::Formatter;
    |            ^^^^^^ private module
    |
note: the module `export` is defined here
   --> C:\Users\luana\.cargo\registry\src\github.com-1ecc6299db9ec823\serde-1.0.119\src\lib.rs:275:5
    |
275 | use self::__private as export;
    |     ^^^^^^^^^^^^^^^^^^^^^^^^^

error: aborting due to previous error

For more information about this error, try `rustc --explain E0603`.
error: could not compile `lingua`.

First, thanks for your work on Lingua.

I found a memory leak when using lingua with the tokio runtime. I think the problem comes from the fact that the language detector might be Send or Sync to multiples threads.

I tried using with_preloaded_language_models() on the builder with no success. Even when I build a new LanguageDetector in each method call I still have a memory leak.

The current WASM API does not strictly follow the builder pattern used in the Rust API. That's because I did not know back then how to accomplish it with wasm-bindgen. While working on the WASM API for grex, I found out how to implement the builder pattern. Before creating the JavaScript distribution for Lingua, these changes have to be applied first.

Currently, the language models are parsed from json files and loaded into simple maps at runtime. Even though accessing the maps is pretty fast, they consume a significant amount of memory. The goal is to investigate whether there are more suitable data structures available that require less storage space in memory, something like NumPy for Python. Perhaps it is even possible to store those data structures in some kind of binary format on disk which can be loaded faster than the current json files.

One promising candidate could be ndarray.

In addition to the current relative confidence metric, an absolute confidence metric shall be implemented which is able to say how likely it is that a given text is written in a specific language, independently from all the other languages.

Lingua's high detection accuracy comes at the cost of being noticeably slower than other language detectors. The large language models also consume significant amounts of memory. These requirements might not be feasible for systems running low on resources.

For users who want to classify mostly long texts or need to save resources, a so-called low accuracy mode will be implemented that loads only a small subset of the language models into memory. The API will be as follows:

LanguageDetectorBuilder::from_all_languages().with_low_accuracy_mode().build();

The downside of this approach is that detection accuracy for short texts consisting of less than 120 characters will drop significantly. However, detection accuracy for texts which are longer than 120 characters will remain mostly unaffected.

Hello. I am doing a cryptology library. I need to detect if the text is english or not. Could you allow confidence value for a single language please?

The library puts emphasis on lazy loading in order to be efficient in certain servless environments. My use-case requires maximum performance, and I can sacrify slower startup for better runtime performance. The LanguageDetectorBuilder has a flag with_preloaded_language_models that forces eager loading of models. However, during detection the LanguageDetector loops through every ngram and calls the function load_language_models, that has to take read lock to check whether the model is loaded. Read locks are cheap, but not free. As an experiment I completely eliminated lazy loading and stored models in the LanguageDetector. Single threaded benchmark improved by 1.2x-2x and multi-threaded by 2x-4x depending on the system.

Patch

This patch works with lazy loading design. Instead of lazy-loading model for every ngram, I load models slightly more eagerly in compute_sum_of_ngram_probabilities and in count_unigrams for the specified language, and reuse the models for ngrams in the loop. For this to work I had to use Arc instead of Box in the BoxedLanguageModel

Benchmark

For the benchmark I used the accuracy reports test data. The benchmark code is here. I tested both single-threaded and multi-threaded/parallel mode.

Results

I tested the patch on two machines

• Intel(R) Core(TM) i7-6800K CPU @ 3.40GHz on Linux
• M1 Max on Mac

The numbers in the columns Before and After are throughput as detections per second.

Single threaded benchmark

cargo run --release --bin bench -- --max-examples 30000

| Machine | Before | After | Change | | ------------- |:-------------:| -----: | -----: | | i7-6800K | 1162 | 1294 | 1.11x | | M1 Max | 1592 | 1844 | 1.15x |

Multi threaded benchmark

cargo run --release --bin bench -- --max-examples 50000 --parallel

| Machine | Before | After | Change | | ------------- |:-------------:| -----: | -----: | | i7-6800K | 4226 | 8033 | 1.9x | | M1 Max | 1383 | 5858 | 4.23x |

I am really surprised by the numbers on M1 chip. Before, single-threaded benchmark runs faster than multi-threaded, which means RwLocks on M1-Mac are slow.

I ran accuracy reports and checked against the current main branch diff -r accuracy-reports/lingua/ ../lingua-rs-main/accuracy-reports/lingua/ The command found no differences.