I think it makes more sense to pass a reference to the document when adding to the index as it is only a reference that is passed to the extraction function for each field.

Also I have a use case where I need to store the original documents and this prevents an unnecessary clone when adding to the index.

Thanks 🙏

Thanks for the library, I spent quite a bit of time researching the available libraries for my project and this one strikes the perfect balance for my needs, in particular the support for webassembly is critical!

It seems that if I vacuum the index then there is no need to pass removed_docs as the last argument to query?

Am i right in thinking that the removed_docs argument exists to support the case when documents have been removed but not yet vacuumed from the index and the query should ignore them?

The API feels a bit strange calling add_document_to_index(), remove_document_from_index() and vacuum_index() as they all accept &mut index as the first parameter.

Whilst we are making breaking changes I suggest using methods on Index:

impl Index {
  fn add_document(&mut self, ..) {}
  fn remove_document(&mut self, ..) {}
  fn vacuum(&mut self, ..) {}
}

I think this is more idiomatic. What do you think?

I was playing our with the library so I created a few changes I could possibly split into multiple PRs. Would like to hear you thoughts about this

Changes for improved performance

• add_document use Cow for hash keys.
• hashbrown Hashmap instead of std::collections. Faster for small hash keys. Though does not provide the same level of HashDoS resistance
• TermData query_terms property is replaced with query_terms_len to prevent unnecessary copies of all query terms during query (not benchmarked yet) (Breaking change for the ScoreCalculator trait)

Bench results on my computer add_100k_docs Master 301.19 ms

This branch 221.24 ms

API change ideas

• Filter is removed, since the Tokenizer can just be seen as a preprocessing step to indexation and query and can do anything that the Filter does. One argument less to worry about. (Breaking change)

The query API requires you to pass a mutable reference to the index. The reason for this is that vacuuming might occur (complete deletion of removed documents) which is a mutation. It would be more ideal if the the query would just take a reference to the index instead, yet still, if possible, use the trie index walk during query as an opportunity to vacuum (or as an opportunity in finding out where to vacuum later).

Based on discussions in #15

The current API only supports fields that extract &str directly from the document however there are times when we need to borrow from a collection in the document.

My use case is that I have a HashSet<String> to represent tags associated with the document.

With this PR my extract functions look like this:

// Label
fn label_extract(d: &Document) -> Vec<&str> {
    vec![d.2.label()]
}

// Tags
fn tags_extract(d: &Document) -> Vec<&str> {
    d.2.tags().iter().map(|s| &s[..]).collect()
}

I expected the extra heap allocation for the Vec to have a minor impact on performance but running the benchmarks shows a small performance improvement 🤷

In the current solution you define extraction functions and pass the as an array in add_document:

pub fn add_document<D> (
        &mut self,
        field_accessors: &[FieldAccessor<D>],
        tokenizer: Tokenizer,
        key: T,
        doc: &D,
    )
    ...

A maybe more idiomatic solution would be to create a Trait, called something like Indexable

trait Indexable
{
    fn key(&self) -> &str,
    fn extract_fields(&self) -> &[&str]
} 

Then the add_document method will look like

pub fn add_document<D: Indexable> (
        &mut self,
        doc: &D,
)

A common use case is to combine a text search with a timestamp sort. It would be interesting to see whether this library could support a numerical index in addition to a text index without too much rework, or breaking changes.

Problematic part of building this support is that we need possibly write a completely new index implementation, or make smart abstraction of some generic index. In addition the query API will become more complicated since we need to distinguish the types of the fields we query, and need to provide a different API for numbers ("lt", "lte", "ge", "geq" operators).

A silly solution to this problem would be to convert numbers to byte arrays, and see each byte as a string token i.e. This would let us do simple "lt", "lte" queries in the same way as doing prefix searches (i.e. counting three leading 0 zeros in [0,0,0,123] would let us quickly know this number is below or equal to 255).

A better solution would be to implement this thoroughly where we would build a dedicated index for numerical values.

I have a use case where I expect to have a small number of documents (several hundred to several thousand) and I would want the search to do fuzzy matching on sub-strings within terms so the search would be as greedy as possible.

This would affect performance so should be optional for this sort of "autocomplete" use case. Is this something that interests you?

Happy to do the legwork under your guidance to land this 👍

https://quantleaf.github.io/probly-search-demo/

Current behaviour garlic garlic -> 0.5 score garlic garlic garlic -> 0.5 score

Expected behaviour

garlic garlic -> 0.5 score garlic garlic garlic -> Some score< 0.5