A Rust library containing an offline version of webster's dictionary.

Grant Handy

Last update: Sep 27, 2022

Related tags

Text processing webster-rs

Overview

webster-rs

A Rust library containing an offline version of webster's dictionary.

Add to Cargo.toml

webster = 0.3.0

Simple example:

fn main() {
    let word = "silence";

    let definition = webster::dictionary(word).unwrap();

    println!("{} definition: {}", word, definition);
}

The definitions are not great but they'll do for simple projects if you need an open source local dictionary API.

This library uses the dictionary.json file from adambom's dictionary adapted from Webster's Unabridged English Dictionary.

Runtime Decompression

In an effort to reduce binary size (naive storage weighs 9mb), the dictionary is stored in a compressed binary format in the executable (4mb) and then decompressed upon runtime access. The runtime container provides O(log n) access complexity and access time (anecdotally) faster than a BTreeMap.

License

The works in this repository are licensed under the MIT License, with the exception of the contents of dictionary.json, which are licensed under the terms of the Project Gutenberg License:

From Project Gutenberg:

This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.net

Comments

Reduction in binary size, access time.
I have not profiled startup time, it's tricky because it only occurs once per run of the executable. Left as an exercise for the reader :)

On my machine the amortized access times have been reduced to 142ns. That's pretty good, and better than std containers like BTreeMap.

Further improvements:

compressed representation can be made smaller, gzip -9 gives better results than using libflate in the build script. Go figure.

load time can probably be improved by using mostly stack-allocated buffers for words.

Hopefully everything looks good to you.
opened by wbrickner 6
[Opt] do not parse giant JSON string for each query

Hi,

I am wondering if a PR would be accepted that changes the behavior from reparsing a giant JSON file every query to something better but more complex.

The idea is to store compressed bytes as a static, in the binary as you have done now.

The first dictionary access decompresses the bytes and stores them in a lazy_static BTree for low memory usage and high data locality and O(log n) access complexity.

This would incur a small startup cost, but this startup cost is likely similar to the cost of every single call that we have now. This solution would be very fast for subsequent accesses and reduce binary bloat.

opened by wbrickner 4

A Rust library containing an offline version of webster's dictionary.

Related tags

Overview

webster-rs

Runtime Decompression

License

You might also like...

A small rust library for creating regex-based lexers

Implementation of sentence embeddings with BERT in Rust, using the Burn library.

A command-line tool and library for generating regular expressions from user-provided test cases

A morphological analysis library.

Wrapper around Microsoft CNTK library

A lightweight library with vehicle tuning utilities.

lingua-rs Python binding. An accurate natural language detection library, suitable for long and short text alike.

Library to calculate TF-IDF (Term Frequency - Inverse Document Frequency) for generic documents.

A simple and fast linear algebra library for games and graphics

Comments

Reduction in binary size, access time.

[Opt] do not parse giant JSON string for each query

Owner

Grant Handy

Fast PDF password cracking utility equipped with commonly encountered password format builders and dictionary attacks.

Rust-nlp is a library to use Natural Language Processing algorithm with RUST

An efficient and powerful Rust library for word wrapping text.

⏮ ⏯ ⏭ A Rust library to easily read forwards, backwards or randomly through the lines of huge files.

Natural language detection library for Rust. Try demo online: https://www.greyblake.com/whatlang/

A Rust library for generically joining iterables with a separator

👄 The most accurate natural language detection library in the Rust ecosystem, suitable for long and short text alike

A fast, low-resource Natural Language Processing and Text Correction library written in Rust.

Rust wrapper for the BlingFire tokenization library

A small random number generator hacked on top of Rust's standard library. An exercise in pointlessness.