A Japanese Morphological Analyzer written in pure Rust

Agata Naomichi

Last update: Apr 30, 2022

Related tags

Overview

Yoin - A Japanese Morphological Analyzer

yoin is a Japanese morphological analyze engine written in pure Rust.

mecab-ipadic is embedded in yoin.

:) $ yoin
すもももももももものうち
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

Build & Install

yoin is available on crates.io

CLI

:) $ cargo install yoin

# or

:) $ git clone https://github.com/agatan/yoin
:) $ cd yoin && cargo install

Library

yoin can be included in your Cargo project like this:

[dependencies]
yoin = "*"

and write your code like this:

extern crate yoin;

Usage - CLI

By default, yoin reads lines from stdin, analyzes each line, and outputs results.

:) $ yoin
すもももももももものうち
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS
そこではなしは終わりになった
そこで	接続詞,*,*,*,*,*,そこで,ソコデ,ソコデ
はなし	名詞,一般,*,*,*,*,はなし,ハナシ,ハナシ
は	助詞,係助詞,*,*,*,*,は,ハ,ワ
終わり	動詞,自立,*,*,五段・ラ行,連用形,終わる,オワリ,オワリ
に	助詞,格助詞,一般,*,*,*,に,ニ,ニ
なっ	動詞,自立,*,*,五段・ラ行,連用タ接続,なる,ナッ,ナッ
た	助動詞,*,*,*,特殊・タ,基本形,た,タ,タ
EOS

Or, reads from file.

:) $ cat input.txt
すもももももももものうち
:) $ yoin --file input.txt
すもも	名詞,一般,*,*,*,*,すもも,スモモ,スモモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
も	助詞,係助詞,*,*,*,*,も,モ,モ
もも	名詞,一般,*,*,*,*,もも,モモ,モモ
の	助詞,連体化,*,*,*,*,の,ノ,ノ
うち	名詞,非自立,副詞可能,*,*,*,うち,ウチ,ウチ
EOS

LICENSE

This software in under the MIT License and contains the MeCab-ipadic model. See LICENSE and NOTICE.txt for more details.

Rust-nlp is a library to use Natural Language Processing algorithm with RUST

nlp Rust-nlp Implemented algorithm Distance Levenshtein (Explanation) Jaro / Jaro-Winkler (Explanation) Phonetics Soundex (Explanation) Metaphone (Exp

34 Dec 20, 2022

Fast suffix arrays for Rust (with Unicode support).

suffix Fast linear time & space suffix arrays for Rust. Supports Unicode! Dual-licensed under MIT or the UNLICENSE. Documentation https://docs.rs/suff

207 Dec 26, 2022

Elastic tabstops for Rust.

tabwriter is a crate that implements elastic tabstops. It provides both a library for wrapping Rust Writers and a small program that exposes the same

212 Dec 16, 2022

An efficient and powerful Rust library for word wrapping text.

Textwrap Textwrap is a library for wrapping and indenting text. It is most often used by command-line programs to format dynamic output nicely so it l

322 Dec 26, 2022

⏮ ⏯ ⏭ A Rust library to easily read forwards, backwards or randomly through the lines of huge files.

EasyReader The main goal of this library is to allow long navigations through the lines of large files, freely moving forwards and backwards or gettin

81 Dec 6, 2022

An implementation of regular expressions for Rust. This implementation uses finite automata and guarantees linear time matching on all inputs.

regex A Rust library for parsing, compiling, and executing regular expressions. Its syntax is similar to Perl-style regular expressions, but lacks a f

2.6k Jan 8, 2023

Natural language detection library for Rust. Try demo online: https://www.greyblake.com/whatlang/

Comments

Use `u32` directly in FST execution

今までは u32 を [u8; 4] に変換して FST を構築し，実行時には 1 byte ずつバッファに詰めて accept される時に u32 に変換し直していた。これをパフォーマンス向上のために最初から u32 だけを使うように変更した。

FST のノードの状態数が増えるので、辞書サイズが大きくなるかと思ったが、4.5MB -> 4.7MB 程度だった。

現状は bytecode に落とす時に alignment を全く考えていないので、もう少し丁寧にやる余地はありそう。

opened by agatan 0