Lindera
A Japanese morphological analysis library in Rust. This project fork from fulmicoton's kuromoji-rs.
Lindera aims to build a library which is easy to install and provides concise APIs for various Rust applications.
Build
The following products are required to build:
- Rust >= 1.46.0
% cargo build --release
Usage
Basic example
This example covers the basic usage of Lindera.
It will:
- Create a tokenizer in normal mode
- Tokenize the input text
- Output the tokens
use lindera::tokenizer::Tokenizer;
use lindera_core::core::viterbi::Mode;
fn main() -> std::io::Result<()> {
// create tokenizer
let mut tokenizer = Tokenizer::new(Mode::Normal, "");
// tokenize the text
let tokens = tokenizer.tokenize("関西国際空港限定トートバッグ");
// output the tokens
for token in tokens {
println!("{}", token.text);
}
Ok(())
}
The above example can be run as follows:
% cargo run --example basic_example
You can see the result as follows:
関西国際空港
限定
トートバッグ
User dictionary example
You can give user dictionary entries along with the default system dictionary. User dictionary should be a CSV with following format.
<surface_form>,<part_of_speech>,<reading>
For example:
% cat userdic.csv
東京スカイツリー,カスタム名詞,トウキョウスカイツリー
東武スカイツリーライン,カスタム名詞,トウブスカイツリーライン
とうきょうスカイツリー駅,カスタム名詞,トウキョウスカイツリーエキ
With an user dictionary, Tokenizer will be created as follows:
use lindera::tokenizer::Tokenizer;
use lindera_core::core::viterbi::Mode;
fn main() -> std::io::Result<()> {
// create tokenizer
let mut tokenizer = Tokenizer::new_with_userdic(Mode::Normal, "", "resources/userdic.csv");
// tokenize the text
let tokens = tokenizer.tokenize("東京スカイツリーの最寄り駅はとうきょうスカイツリー駅です");
// output the tokens
for token in tokens {
println!("{}", token.text);
}
Ok(())
}
The above example can be by cargo run --example:
% cd lindera/lindera
% cargo run --example userdic_example
東京スカイツリー
の
最寄り駅
は
とうきょうスカイツリー駅
です
API reference
The API reference is available. Please see following URL:
12 Sep 22, 2022
5.8k Dec 30, 2022
322 Dec 26, 2022
81 Dec 6, 2022
805 Dec 28, 2022
72 Dec 16, 2022
496 Jan 8, 2023
14 Sep 5, 2022
12 Sep 27, 2022
21 Nov 30, 2021
34 Dec 20, 2022
6 Oct 3, 2022
7 Dec 30, 2022
1 Nov 24, 2021
13 Oct 31, 2022
953 Jan 3, 2023
1 Feb 5, 2022
11 Jul 29, 2022