This PR suggests to unify three arrays ends, ends_full, and indices in Lattice for locality of reference.

In this modification, I added a new struct PackedNode packing three structs VNode, Node, and NodeIdx, and implemented the three arrays as one array of PackedNode.

The three arrays are often accessed simultaneously with the same index value, and the modification can improve cache efficiency in tokenization.

My microbenchmark results (with Intel i7, 16G RAM) showed that the modification shortened a tokenization time by 10%. (My microbenchmark code can be found here).

I found an input pattern which causes exception in sudachipy==0.6.0. The reproducing code below is the abstract of the Japanese tokenizer of spaCy v3.2.0.

from sudachipy import dictionary, tokenizer

def get_dtokens(tokenizer, sudachipy_tokens, need_sub_tokens):
    sub_tokens_list = get_sub_tokens(tokenizer, sudachipy_tokens) if need_sub_tokens else None
    dtokens = [
        (
            t.surface(),
            t.part_of_speech()[:4],
            t.part_of_speech()[4:],
            t.dictionary_form(),
            t.normalized_form(),
            t.reading_form(),
            sub_tokens_list[idx] if need_sub_tokens else None,
        ) for idx, t in enumerate(sudachipy_tokens) if len(t.surface()) > 0
    ]
    return dtokens

def get_sub_tokens(tokenizer, sudachipy_tokens):
    sub_tokens_list = []
    for token in sudachipy_tokens:
        sub_a = token.split(tokenizer.SplitMode.A)
        if len(sub_a) == 1:  # no sub tokens
            sub_tokens_list.append(None)
        else:
            sub_b = token.split(tokenizer.SplitMode.B)
            if len(sub_a) == len(sub_b):
                dtokens = get_dtokens(tokenizer, sub_a, False)
                sub_tokens_list.append([dtokens, dtokens])
            else:
                sub_tokens_list.append(
                    [
                        get_dtokens(tokenizer, sub_a, False),
                        get_dtokens(tokenizer, sub_b, False),
                    ]
                )
    return sub_tokens_list

tokenizer = dictionary.Dictionary().create(mode=tokenizer.Tokenizer.SplitMode.C)
sudachipy_tokens = tokenizer.tokenize("Ｔ社はｅコマース（電子商取引）を活用したリサイクル部品の取扱いを系列の部品販売店で平成１３年１０月より始めました。取り扱う部品は、ドア、フェンダー、グリル、バンパー、ランプ類などのＴ社の外装部品（「エ コロパーツ」）全１６品目と大手リサイクル部品流通事業社のＮグループ及びＢ社から供給を受ける国内全メーカーの外装・機能部品で、専用の中古部品ｅコマースサイトを開設し、自動車保有期間の長期化に伴う低価格修理の需要に応えることにしています。")
get_dtokens(tokenizer, sudachipy_tokens, True)

/home/matsuda/ginza/test.py:21: DeprecationWarning: API around this functionality will change. See github issue WorksApplications/SudachiPy#92 for more.
  sub_a = token.split(tokenizer.SplitMode.A)
/home/matsuda/ginza/test.py:25: DeprecationWarning: API around this functionality will change. See github issue WorksApplications/SudachiPy#92 for more.
  sub_b = token.split(tokenizer.SplitMode.B)
thread '<unnamed>' panicked at 'byte index 10 is not a char boundary; it is inside 'ｅ' (bytes 9..12) of `Ｔ社はｅコマース（電子商取引）を活用したリサイクル部品の取扱いを系列の部品販売店で平成１３年１０月より始めました。取り扱う部品は、ドア、フェンダー、グリル、バンパー、ラン`[...]', /github/workspace/sudachi/src/analysis/morpheme.rs:122:10
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Traceback (most recent call last):
  File "/home/matsuda/ginza/test.py", line 40, in <module>
    get_dtokens(tokenizer, sudachipy_tokens, True)
  File "/home/matsuda/ginza/test.py", line 4, in get_dtokens
    sub_tokens_list = get_sub_tokens(tokenizer, sudachipy_tokens) if need_sub_tokens else None
  File "/home/matsuda/ginza/test.py", line 27, in get_sub_tokens
    dtokens = get_dtokens(tokenizer, sub_a, False)
  File "/home/matsuda/ginza/test.py", line 5, in get_dtokens
    dtokens = [
  File "/home/matsuda/ginza/test.py", line 14, in <listcomp>
    ) for idx, t in enumerate(sudachipy_tokens) if len(t.surface()) > 0
pyo3_runtime.PanicException: byte index 10 is not a char boundary; it is inside 'ｅ' (bytes 9..12) of `Ｔ社はｅコマース（電子商取引）を活用したリサイクル部品の取扱いを系列の部品販売店で平成１３年１０月より始めました。取り扱う部品は、ドア、フェンダー、グリル、バンパー、ラン`[...]

Prereq of #52

We want a ponter-generic version of return container for the result of analysis.

Rust API wants references/mut references; Python binding wants Arc.

This project has no license, which means (by default) users do not have rights to use it.

Fix steps:

• [x] Add license file to repo
  • https://docs.github.com/en/github/building-a-strong-community/adding-a-license-to-a-repository
• [x] Add "license" field to Cargo.toml
  • https://doc.rust-lang.org/cargo/reference/manifest.html#the-license-and-license-file-fields

I suggest Apache 2.0 license like Sudachi.

Fixes #147

Add pyi file for type hints. sudachipy.pyi is enough to cover all classes. I used mypy's stubgen cli tool to generate base of pyi file.

Also fix type signatures. The annotation for positional-only parameters was wrong. ref: PEP570

#73

~WIP~: default dictionary path setting SudachiPy: dict_type > config.systemDict > sudachidict_core. sudachi.rs: arg > config.systemDict > baked dict.

Fixes #17 Fixes #19 Fixes #20 Fixes #21 Fixes #24

• Plugin loading logic is refactored, no repeated code now
• DSO plugins can now be resolved relative to executable with the syntax: $exe/plugin or relative to the configuration folder $cfg/plugin.
• It is recommended not to specify platform dependent name parts for DSO plugins: plugin instead of libplugin.so. First name will correctly resolve on Windows/Linux/macOS.
• Built-in plugins are moved into the main crate, they can be specified in the configuration by their Java names (but that needs checking).

Main design for the rework of plugin loader

We have core logic of plugin loading in a PluginLoader struct which is generic over implementations of PluginCategory trait. PluginCategory captures plugin category specific logic: providing specific types, configuration extraction, and instantiation of bundled plugins.

PluginLoader is using fully qualified trait call syntax (see previous paragraph before this link). It is required because type inference does not work if T is not present as one of arguments.

1. README上に記載されている以下のコマンドは、正しくは「pip install --upgrade 'sudachipy>=0.6.2'」ではないでしょうか？

pip install --update 'sudachipy>=0.6.2'

2. 以下のコマンドが通りません。エラー「error: found a virtual manifest at /(略)/sudachi.rs/Cargo.toml instead of a package manifest」となります。

cargo install --path .

Fixes #184

Implement out parameters for Tokenizer.tokenize() and Morpheme.split() Python API.

For the memory sharing to be actually useful, I had to refactor internal MorphemeList to allow multiple references to input data, while having distinct list of morphemes. Let's welcome Arc<RefCell<X>> in our codebase. Python MorphemeListWrapper has also changed. As a side effect there is no copy in custom pretokenizers (win). Need to document semantics of everything, but will do that as a documentation pass.

Depends on PR #176 (includes its comments)

Fixes #38 Fixes #166 #Fixes #168

Created as Dictionary.pre_tokenizer([mode]), where the default mode is C.

Implementation can be used from multiple threads, with a single PreTokenizer instance shared between all threads. It will create thread local Sudachi tokenizer and use it to perform the actual tokenization. Also, implementation releases GIL while doing the analysis, so it should achieve some speedup when used multithreaded.

#98

Generate doc:

• install sudachi.rs python binding: pip install setuptools-rust; python setup.py develop
• install sphinx and theme: pip install sphinx sphinx-rtd-theme
• run make html in docs directory
  • html will be generated under docs/build/html

I noticed you don't have wheels for aarch64 for Linux. Do you have any plans to release such wheels?

Github Actions supports this architecture, so it looks like you can probably just make a small change to the current wheel building actions.

Hi, thanks for developing this great library. I played a bit with R (my repo), but I had to struggle with this error when I tried to make a static binary for users that don't have Rust installed. It seems this error is because Config::new() cannot find the files under resources directory.

Io(Os { code: 3, kind: NotFound, message: "指定されたパスが見つかりません。" })

As the files are not very huge, is it possible to embed them somehow, e.g. using include_str macro, on compile time?

I noticed that if input is too long in Python an Exception is thrown, but it's a plain Exception, not a ValueError or something. I see in the Rust code there are a variety of specific error types.

I'm not familiar with Rust, but surely it's possible to have the Python code throw something more specific like a InputTooLongException?