Probably related to #897

Describe the bug In continuation of https://gitter.im/tantivy-search/tantivy?at=5ff589c503529b296bd23728 Firstly, I found memory bloating of Tantivy application. After debugging I found that merging thread have been failing every time on merging broken store in a particular segment. Example of the whole segment I've sent to your personally in Gitter.

Which version of tantivy are you using? https://github.com/tantivy-search/tantivy/commit/a4f33d3823f1bad3ff7a59877f1608615acabe6e

What happened I used poor man debugging and launched Tantivy with patched function (added print only):

    fn write_storable_fields(&self, store_writer: &mut StoreWriter) -> crate::Result<()> {
        for reader in &self.readers {
            let store_reader = reader.get_store_reader()?;
            if reader.num_deleted_docs() > 0 {
                for doc_id in reader.doc_ids_alive() {
                    let doc = store_reader.get(doc_id);
                    if let Err(ref err) = doc {
                        println!("Error: {:?}\nSegment ID: {:?}\nDocID: {}", err, reader.segment_id(), doc_id);
                    }
                    store_writer.store(&doc?)?;
                }
            } else {
                store_writer.stack(&store_reader)?;
            }
        }
        Ok(())
    }

Stdout after failed merging:

Error: IOError(Custom { kind: InvalidData, error: "Reach end of buffer while reading VInt" })
Segment ID: Seg("e6ece22e")
DocID: 53

(This is a follow up from #549)

Ideally what I need is the ability to write to index from independent threads each having own writer.

This problem is very common and hurts some Very Important Project relying on tantivy (toshi, plume). (Invoking the name @fdb-hiroshima and @hntd187 for the discussion).

Web server are typically multithreaded and requests may spawn the need to add or delete documents. Dealing with a Arc<RwLock<IndexWriter>> might feel dirty, and rust beginners may not really understand the logic behind that.

On the other hand, the IndexWriter already relies on a channel to dispatch indexing operation to its own small thread pool. Stamping is also done using Atomics. There is actually no real reason to prevent .add_document() and and .delete_term() to happen from different threads.

The problem is

• what should happen with .commit()
• what should happen with .rollback()

Especially would this ability to index from several threads confuse people on

• what a commit is in tantivy? (.commit() ensure that all operations that happened but the .commit() are processed)
• how multithreaded indexing works ? (indexing from several thread will no help. Tantivy has its own multithreading system).

Also,

• should .commit() and .rollback() block other operations? (It is technically possible to have .commit() only block other .commit() operations.)
• Should .commit() and .rollback() return futures?

This is a proof-of-concept implementation of BMW algorithm for union queries. There are still some parts missing in the algorithm itself, but also I haven't touched anything related to indexing yet.

Two things in particular I want to point out for now:

• BMW requires sorting of posting lists; I wrapped them in Box thinking that it should be faster to move than lists themselves; not yet sure this is the best way, I haven't spent too much time thinking about it so far;
• BMW needs that additional piece of feedback from the collector: the current threshold; right now, I am passing a predicate but it definitely has its disadvantages. one alternative is to have someone, say, the collector, set that between steps; again, not entirely sure what the way to do this is.

Any comments at this stage would be appreciated.

Describe the bug the snippet code I use is here

garbage_collect_files does not work the first time I use it, only the second time. First time means first call of garbage_collect_files in one thread, second means second call of garbage_collect_files in another thread

Which version of tantivy are you using? 0.9.1

To Reproduce

repo where is all the code I used, look at drop_index() function in src/bin/server.rs

I use POST request to trigger garbage_collect_files like this: POST http://localhost:8080/drop

When I post at first time all documents are deleted but ./index directory does not become empty, actually it has 36Mb data in it. Original size of index directory was 129Mb When I post at second time directory size drops to 8,0K. Nothing else but second call to garbage_collect_files happens.

I don't know, maybe it is an expected behavior, but is weird.

Hi, i'm a rust rookie. my program suffer a memory leak problem, it could been killed by kernel after running along time. Getting data from redis，and adding to tantivy with rust:

fn main() {
    let mut index_writer = index.writer_with_num_threads(num_threads, buffer).unwrap();
    index_writer.set_merge_policy(mergePolicy);
    loop {
        let mut datas = RedisService::get_data();
        let len = datas.len();
        if len == 0 {
            //sleep
        }
        for data in datas {
            let mut doc = Document::new();
            let json_value = json::parse(&data).unwrap();
            //doc.add_text,add_u64;
            index_writer.add_document(doc);
        }
        index_writer.commit();
    }
}

This patch-set implements python bindings for Tantivy.

The bindings are written using PyO3 and thus require rust nightly.

Some parts of the Tantivy API are omitted for now (like the SegmentReader) while some parts are made a little bit more pythonic using optional arguments instead of builder patterns.

The search method of the Searcher struct is incomplete and only allows a single TopCollector as its collector argument. Since the method is generic and python doesn't support generics, it has proven to be quite tricky to allow multiple differing collectors to be passed to the function in a dynamic way.

The first couple of patches add some functionality or expose some structs publicly in Tantivy that are required for the python bindings.

The last patch finally adds the python bindings as a sub-crate to Tantivy. To build the bindings a setup.py file is provided that requires setuptools-rust or pyo3-pack can be used. Both of the methods allow the bindings to be uploaded to pypi.

Closes #273

Ok, so I've started down the path, but I'm for sure lost.

1. Not sure how I would test with Automaton - still reading up on the fst code there
2. Not sure how to use the fst StreamBuilder, but it feels close on the code implementation, but I have compiler issues, which could just be about spreading the generic type love around more.

Fixes #1304.

BREAKING API CHANGE: The type DateTime is no longer (re-)exported. Instead you need to use time::OffsetDateTime explicitly.

Remarks:

• Please check the error handling and mappings. time returns results instead of panicking like chrono does. For the FastValue impl I had to use `.expect("valid UNIX timestamp").
• I had to adjust some debug strings in tests that are used for matching test results. Copy&paste from the failed test outputs. Not very handy, please verify.

TODO:

• [x] Add changelog entry

I am getting a failure starting my application

Failed purge deletes: Error(PathDoesNotExist("/SOME_PATH/c475e13ef3ca45128b1f8d9ee42fe994.term")

This is on OS X. My computer crashed recently. What's a good way to move forward here? Happy to help writing some code if that helps :)

Describe the bug The same load profile as in #969 - deletions, addings and mergings. Now it happens on querying after several hours of serving. I think the reason is basically the same. At startup and during several hours afterwards all queries were ok but after generations of merges searcher.doc started to throw VInt decoding error.

Which version of tantivy are you using? https://github.com/tantivy-search/tantivy/commit/bf6e6e8a7cc1826212ba2500b08ecb53dfcdeda1

To Reproduce Sent broken segment to you in gitter.

Is your feature request related to a problem? Please describe. I am creating an app where I would like to break different fragment onto their own lines.

What my app currently outputs:

[ueuccfcgbx] Fucshia Arch: https://blog.quarkslab.com/playing-around-with-the-fuchsia-operating-system.html
address space is context-switched by **Zircon**.Contrary to other OSes however, the IOMMU
(Input-Output MMU), plays an important role on **Zircon**: it is

Notice how **Zircon**.Contrary is concatenated.

What I would like:

[ueuccfcgbx] Fucshia Arch: https://blog.quarkslab.com/playing-around-with-the-fuchsia-operating-system.html
address space is context-switched by **Zircon**. 
Contrary to other OSes however, the IOMMU(Input-Output MMU), plays an important role on **Zircon**: it is

Currently a Snippet stores fragments (a string that has all of the text of the snippet) and a vector of HighlightSections (what parts of fragments should be highlighted). I would like to break each fragment onto its own line to make this output more readable/understandable. Currently there is no way to know where one fragment ends, and another begins.

Describe the solution you'd like Change the type of Snippet::fragments to Vec<String> and add a fragment_number member to HighlightSection.

[Optional] describe alternatives you've considered Add a new member to Snippet similar to highlighted called fragment_sections, that is a Vec of start and stop points for the different fragments

This adds a regex tokenizer which tokenizes the text by using a regex pattern to split. This is my first attempt and works for my usecase, but it's not ideal from a code and a configuration perspective.

Closes https://github.com/quickwit-oss/tantivy/issues/1670

Implementation of a RegexTokenizer for #1670.

To make this a decently flexible solution I implemented support for capture groups as well as simple find matches: The RegexTokenStream will check on construction if there are any capture groups (besides the entire match) and use either captures_read_at for capture support or find_at for simple matches. Tests and examples included.

Closes https://github.com/quickwit-oss/tantivy/issues/1670

Describe the bug

When inserting fast fields in a certain order, segments will occasionally end up with a column that matches extra documents when calling get_docids_for_value_range using nonexistant values.

The testcase at the bottom of this report illustrates a failing case, though you need to run it repeatedly to cause it to fail.

Expected behaviour: passing fastfield values/ranges to get_docids_for_value_range that cannot possibly exist in a given segment reader should return no matching documents.

Which version of tantivy are you using?

0.19.0

To Reproduce

This code will reproduce the bug, but it won't happen every time. Running it repeatedly will eventually give you a SegmentReader with the min/max values [6341073085727221 3373930417471920086], and that SegmentReader will always match document zero when querying in the invalid value 7999999999999999.

    #[test]
    fn test_gcd_bug() {
        let dir = RamDirectory::create();
        let mut schema_builder = Schema::builder();
        let url_norm_hash_field = schema_builder.add_i64_field("url_norm_hash", FAST | INDEXED);  
        let schema = schema_builder.build();
        let index = Index::create(dir, schema, IndexSettings::default()).unwrap();
        let mut writer = index.writer(50_000_000).unwrap();
        for i in [6341073085727221_i64, 6341073085727221_i64, 6341073085727221_i64, 6341073085727221_i64, 6341073085727221_i64, 6341073085727221_i64, 3373930417471920086_i64, 3373930417471920086_i64, 3373930417471920086_i64, 3373930417471920086_i64] {
            writer.add_document(doc! {
                url_norm_hash_field => i,
            }).unwrap();
        }
        writer.commit();
        let reader = index.reader().unwrap();
        let searcher = reader.searcher();
        for segment in searcher.segment_readers() {
            let field = segment.fast_fields().i64(url_norm_hash_field).unwrap();
            let (min, max) = (field.min_value(), field.max_value());
            println!("{} {}", min, max);
            if 7999999999999999_i64 >= min && 7999999999999999_i64 <= max {
                let mut vec = vec![];
                field.get_docids_for_value_range(7999999999999999_i64..=7999999999999999_i64, 0..u32::MAX, &mut vec);
                println!("{:?}", vec);
            }
        }
    }

Describe the bug

Tantivy panics when reading an index of old version with no DOC_STORE_VERSION field in the footer:

thread 'MainLoop' panicked at 'actual doc store version: 2222222222, expected: 1', tantivy-0.19.0\src\store\footer.rs:30:13

Is your feature request related to a problem? Please describe. Continuing my work on getting tantivy to work server-side with WASM (related issues #1751 #541 ), I would like to index dynamically added documents. In essence I have a large set of documents I can pre-index in a build phase, however each user also has some documents that are loaded dynamically from a database.

I would imagine that normally I could do something like this:

let index = Index::open(ram_directory).unwrap();
let mut index_writer = index.writer(3_000_000).unwrap();
index_writer.add_document(my_document).unwrap();
index_writer.commit().unwrap();

And then search the index. I realize that this might leak documents from one tenant to another, but as this index is rebuilt in-memory on each request and dropped after, this isn't a large concern.

However, as WASM is single-threaded I'm unable to actually get this to work as it seems all the IndexWriters require a thread pool of some sort.

I have tried both index.writer() and index.writer_with_num_threads with both 1 and 0 threads. I've even delved into the undocumentedSingleSegmentIndexWriter Even though it seems to suggest it's only for creating an index with a Single Segment and not adding to an existing index, I figured I would give it a go.

However trying to instantiate it gives me the following IoError

IoError(
    Error {
        kind: Unsupported,
        message: "operation not supported on this platform",
    },
)

Which I think might be threadpool-related, but I am unable to get a stacktrace to confirm.

Describe the solution you'd like I think in essence I'm asking if there's any way to accomplish what I want to do.

• Is there another way to use tantivy to index dynamic documents without using an IndexWriter and that machinery?
• If not, is there any single-threaded IndexWriter or similar I've been unable to find? Will you consider adding one?
• Alternatively, is there any way or guidance for me to implement my own single-threaded IndexWriter? From a first glance it seems like a lot of the code that deals with writing is (understandably) internal to tantivy and not exposed.