LNX: e9804944edc8a7c0af24ee3ba8397b87f1640b5f

I'm trying to figure out how to reproduce this. After the panic I reported in #18, after restarting lnx, I noticed that my index schema that was being returned from the search query was messed up. Searching seems to work as advertised (as in searching field_a:foo, will return documents where foo is set in field_a, but the schema of the search results is messed up).

For example, let's say I have schema where I am only storing 3 fields (field_a, field_b, field_c), but I am indexing 6. before the restart (example)

$ curl 'http://localhost:4040/indexes/posts/search?query=field_a:foo&mode=normal&limit=50&order_by=-ts'
{"data":{"count":40,"hits":[{"doc":{"field_a":["foo"],"field_b":[4],"field_c":[44]}, # etc

Now that same query is returning:

$ curl 'http://localhost:4040/indexes/posts/search?query=field_a:foo&mode=normal&limit=50&order_by=-ts'
{"data":{"count":40,"hits":[{"doc":{"field_d":["foo"],"field_e":[4],"field_f":[44]}, # etc

The values are correct, but the name of the keys are completely different.

However, in trying to reproduce the error, it seems like my lnx install is corrupted. I tried to to create an index like so:

{
    "name": "corrupt",

    "writer_buffer": 144000000,
    "writer_threads": 12,
    "reader_threads": 12,

    "max_concurrency": 24,
    "search_fields": [
        "field_a"
    ],

    "storage_type": "filesystem",
    "set_conjunction_by_default": true,
    "use_fast_fuzzy": false,
    "strip_stop_words": false,

    "fields": {
        "field_a": {
            "type": "text",
            "stored": true
        },
        "field_b": {
           "type": "u64",
           "stored": true,
           "indexed": true,
           "fast": "single"
        },
        "field_c": {
           "type": "u64",
           "stored": true,
           "indexed": true,
           "fast": "single"
        },
        "field_d": {
            "type": "text",
            "stored": false
        },
        "field_e": {
            "type": "text",
            "stored": false
        },
        "field_f": {
            "type": "text",
            "stored": false
        },
        "version": {
            "type": "u64",
            "stored": false,
            "indexed": true,
            "fast": "single"
        }
    },
    "boost_fields": {}
}

then index these documents:

[
    {"field_a":["foo"], "field_b":[4], "field_c":[44], "field_d":["macbook"], "field_e":["apple"], "field_f":["iphone"], "version":[1]},
    {"field_a":["bar"], "field_b":[5], "field_c":[55], "field_d":["laptop"], "field_e":["micrsoft"], "field_f":["galaxy"], "version":[2]},
    {"field_a":["redbull coke"], "field_b":[6], "field_c":[66], "field_d":["thinkpad"], "field_e":["netflix"], "field_f":["nexus"], "version":[3]},
    {"field_a":["vodka sprite"], "field_b":[7], "field_c":[77], "field_d":["ultrabook"], "field_e":["facebook"], "field_f":["blackberry"], "version":[4]},
    {"field_a":["ginger ale whiskey"], "field_b":[8], "field_c":[88], "field_d":["chomebook"], "field_e":["google"], "field_f":["oneplus"], "version":[5]}
]

When I added them I got no error:

 $ curl -X POST -d@corrupt_data.json -H "Content-Type: application/json" http://localhost:4040/indexes/corrupt/documents?wait=true
{"data":"added documents","status":200}

But in the lnx logs I saw

Aug 30 01:23:56 torako lnx[635823]: [2021-08-30][01:23:56] | engine::index::writer | INFO  - [ WRITER @ corrupt ][ TRANSACTION 0 ] completed operation ADD-DOCUMENT
Aug 30 01:23:56 torako lnx[635823]: [2021-08-30][01:23:56] | engine::index::writer | INFO  - [ WRITER @ corrupt ][ TRANSACTION 1 ] completed operation ADD-DOCUMENT
Aug 30 01:23:56 torako lnx[635823]: [2021-08-30][01:23:56] | engine::index::writer | INFO  - [ WRITER @ corrupt ][ TRANSACTION 2 ] completed operation ADD-DOCUMENT
Aug 30 01:23:56 torako lnx[635823]: [2021-08-30][01:23:56] | engine::index::writer | INFO  - [ WRITER @ corrupt ][ TRANSACTION 3 ] completed operation ADD-DOCUMENT
Aug 30 01:23:56 torako lnx[635823]: [2021-08-30][01:23:56] | engine::index::writer | INFO  - [ WRITER @ corrupt ][ TRANSACTION 4 ] completed operation ADD-DOCUMENT
Aug 30 01:23:56 torako lnx[635823]: thread 'thrd-tantivy-index3' panicked at 'Expected a u64/i64/f64 field, got Str("redbull coke") ', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tantivy-0.16.0/src/fastfield/mod.rs:208:14
Aug 30 01:23:56 torako lnx[635823]: stack backtrace:
Aug 30 01:23:56 torako lnx[635823]:    0: rust_begin_unwind
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
Aug 30 01:23:56 torako lnx[635823]:    1: std::panicking::begin_panic_fmt
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:457:5
Aug 30 01:23:56 torako lnx[635823]:    2: tantivy::indexer::index_writer::index_documents
Aug 30 01:23:56 torako lnx[635823]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aug 30 01:23:56 torako lnx[635823]: thread 'thrd-tantivy-index0' panicked at 'Expected a u64/i64/f64 field, got Str("foo") ', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tantivy-0.16.0/src/fastfield/mod.rs:208:14
Aug 30 01:23:56 torako lnx[635823]: stack backtrace:
Aug 30 01:23:56 torako lnx[635823]:    0: rust_begin_unwind
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
Aug 30 01:23:56 torako lnx[635823]:    1: std::thread 'panickingthrd-tantivy-index1::' panicked at 'begin_panic_fmtExpected a u64/i64/f64 field, got Str("bar")
Aug 30 01:23:56 torako lnx[635823]: ',              at /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tantivy-0.16.0/src/fastfield/mod.rs/rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs::208457::145
Aug 30 01:23:56 torako lnx[635823]:    2: tantivy::indexer::index_writer::index_documents
Aug 30 01:23:56 torako lnx[635823]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aug 30 01:23:56 torako lnx[635823]: stack backtrace:
Aug 30 01:23:56 torako lnx[635823]:    0: rust_begin_unwind
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
Aug 30 01:23:56 torako lnx[635823]:    1: std::panicking::begin_panic_fmt
Aug 30 01:23:56 torako lnx[635823]:              at thread '/rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rsthrd-tantivy-index4:' panicked at '457Expected a u64/i64/f64 field, got Str("vodka sprite") :', 5/root/.cargo/registry/src/github.com-1ecc6299db9ec823/tantivy-0.16.0/src/fastfield/mod.rs
Aug 30 01:23:56 torako lnx[635823]: :208: 14
Aug 30 01:23:56 torako lnx[635823]: 2: tantivy::indexer::index_writer::index_documents
Aug 30 01:23:56 torako lnx[635823]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aug 30 01:23:56 torako lnx[635823]: stack backtrace:
Aug 30 01:23:56 torako lnx[635823]: thread 'thrd-tantivy-index2' panicked at 'Expected a u64/i64/f64 field, got Str("ginger ale whiskey") ', /root/.cargo/registry/src/github.com-1ecc6299db9ec823/tantivy-0.16.0/src/fastfield/mod.rs:208: 14
Aug 30 01:23:56 torako lnx[635823]:  0: rust_begin_unwind
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
Aug 30 01:23:56 torako lnx[635823]:    1: std::panicking::begin_panic_fmt
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:457:5
Aug 30 01:23:56 torako lnx[635823]:    2: tantivy::indexer::index_writer::index_documents
Aug 30 01:23:56 torako lnx[635823]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Aug 30 01:23:56 torako lnx[635823]: stack backtrace:
Aug 30 01:23:56 torako lnx[635823]:    0: rust_begin_unwind
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:515:5
Aug 30 01:23:56 torako lnx[635823]:    1: std::panicking::begin_panic_fmt
Aug 30 01:23:56 torako lnx[635823]:              at /rustc/a178d0322ce20e33eac124758e837cbd80a6f633/library/std/src/panicking.rs:457:5
Aug 30 01:23:56 torako lnx[635823]:    2: tantivy::indexer::index_writer::index_documents
Aug 30 01:23:56 torako lnx[635823]: note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

I'm not sure if these two errors are related, but it seems lnx's understanding of the fields and tanvity's aren't in sync.

So in this bug, you are right that was the cause. I copied this example from the book and assumed I was hitting the same issue. What I am actually seeing is that when I index a document with a date field, I am no longer able to index any more documents.

# curl -X DELETE  'http://localhost:4040/indexes/my-index'
{"data":"index deleted","status":200}# 
# cat a.json
{
  "name": "my-index",
  "writer_buffer": 6000000,
  "writer_threads": 1,
  "reader_threads": 1,
  "max_concurrency": 10,
  "search_fields": [
    "title"
  ],
  "storage_type": "memory",
  "use_fast_fuzzy": false,
  "strip_stop_words": false,
   "set_conjunction_by_default": false,
  "fields": {
    "title": {
      "type": "text",
      "stored": true
    },
    "description": {
      "type": "text",
      "stored": true
    },
    "id": {
      "type": "u64",
      "indexed": true,
      "stored": true,
      "fast": "single"
    },
 "ts": {
            "type": "date",
            "stored": false,
            "indexed": true,
            "fast": "single"
        }
  },
  "boost_fields": {}
}
# curl -X POST [email protected] -H "Content-Type: application/json" http://127.0.0.1:4040/indexes
{"data":"index created","status":200}
# cat c.json
{
    "title": ["Hello, World2"],
"id":[4]
}
# curl -X POST [email protected] -H "Content-Type: application/json" http://localhost:4040/indexes/my-index/documents?wait=true
{"data":"added documents","status":200}
# curl -X POST 'http://localhost:4040/indexes/my-index/commit'
{"data":"changes committed","status":200}
# curl 'http://localhost:4040/indexes/my-index/search?query=*&mode=normal'
{"data":{"count":1,"hits":[{"doc":{"id":[4],"title":["Hello, World2"]},"document_id":"8295453496340348446","ratio":1.0}],"time_taken":0.0001392010017298162},"status":200}
# cat b.json
{
    "title": ["Hello, World2"],
"id":[4],
"ts":[1630097583]
}
# curl -X POST [email protected] -H "Content-Type: application/json" http://localhost:4040/indexes/my-index/documents?wait=true
{"data":"added documents","status":200}
# curl -X POST 'http://localhost:4040/indexes/my-index/commit'
{"data":"changes committed","status":200}
# curl 'http://localhost:4040/indexes/my-index/search?query=*&mode=normal'
{"data":{"count":1,"hits":[{"doc":{"id":[4],"title":["Hello, World2"]},"document_id":"8295453496340348446","ratio":1.0}],"time_taken":0.0001936009939527139},"status":200}

Adding a document with a date field doesn't produce an error, but seems to corrupt the index. In my original setup, I always had a date field, and I wasn't seeing any documents get indexed, which is why I assumed the two errors were the same. Once this happens even documents without the ts field fail to be indexed.

Originally posted by @miyachan in https://github.com/lnx-search/lnx/issues/14#issuecomment-907478178

This issue is not present in 0.7.0, but present in 0.8.0

Error log detail

2022-01-16T14:30:18.000995Z ERROR error during lnx runtime: failed to load existing indexes due to error Failed to acquire Lockfile: LockBusy. Some("Failed to acquire index lock. If you are using a regular directory, this means there is already an `IndexWriter` working on this `Directory`, in this process or in a different process.")

Reproduce step: Restart pod or recreate pod. k8s will send SIGTERM to pod. But this issue is not present in 0.7.0 In k8s I can add preStop hook to execute kill -SIGINT $(pgrep lnx) to send CTRL-C to lnx process, but if lnx is paniced or crashed, how can I unlock the writer index? For example, if there is a lock file could be removed before lnx start: rm -rf index/some-lock-file && lnx

LNX Version: 8d38d386de3ecdb1bced4f55032673860feaccf4

This is an issue that happened between e980494...8d38d386de3ecdb

$ cat a.json
{
  "name": "my-index",
  "writer_buffer": 6000000,
  "writer_threads": 1,
  "reader_threads": 1,
  "max_concurrency": 10,
  "search_fields": [
    "title"
  ],
  "storage_type": "memory",
  "use_fast_fuzzy": false,
  "strip_stop_words": false,
   "set_conjunction_by_default": false,
  "fields": {
    "title": {
      "type": "text",
      "stored": true
    },
    "description": {
      "type": "text",
      "stored": true
    },
    "id": {
      "type": "u64",
      "indexed": true,
      "stored": true,
      "fast": "single"
    },
 "ts": {
            "type": "date",
            "stored": true,
            "indexed": true,
            "fast": "single"
        }
  },
  "boost_fields": {}
}
$ curl -X POST [email protected] -H "Content-Type: application/json" http://127.0.0.1:4040/indexes
{"data":"index created","status":200}
$ cat d.json
{
    "id": {"type": "u64", "value": [4]}
}
$ curl -X DELETE [email protected] -H "Content-Type: application/json" http://localhost:4040/indexes/my-index/documents?wait=true
{"data":"invalid JSON body: Failed to parse the request body as JSON","status":400}

Logs:

Aug 30 19:01:42 torako lnx[866518]: [2021-08-30][19:01:42] | tantivy::indexer::segment_updater | INFO  - Running garbage collection
Aug 30 19:01:42 torako lnx[866518]: [2021-08-30][19:01:42] | tantivy::directory::managed_directory | INFO  - Garbage collect
Aug 30 19:01:42 torako lnx[866518]: [2021-08-30][19:01:42] | engine::index::writer | INFO  - [ WRITER @ my-index ][ TRANSACTION 4 ] completed operation COMMIT
Aug 30 19:01:49 torako lnx[866518]: [2021-08-30][19:01:49] | engine::index::reader | INFO  - [ SEARCH @ my-index ] took 120.259µs with limit=20, mode=Normal and 1 results total
Aug 30 19:07:26 torako lnx[866518]: [2021-08-30][19:07:26] | lnx::routes | WARN  - rejecting request due to invalid body: InvalidJsonBody(Error { inner: Error("invalid type: map, expected a string or u32", line: 1, column: 13) })
Aug 30 19:07:51 torako lnx[866518]: [2021-08-30][19:07:51] | lnx::routes | WARN  - rejecting request due to invalid body: InvalidJsonBody(Error { inner: Error("invalid type: map, expected a string or u32", line: 1, column: 13) })

Lnx Version: e9804944edc8a7c0af24ee3ba8397b87f1640b5f

I'm trying to somehow reproduce this as I'm not sure how it occurred. I have a system which adds documents to the index and commits every 10s. I was executing searches against the system (specifically I was testing which queries might cause an error, not sure if this is related):

$ curl 'http://localhost:4040/indexes/posts/search?query=text:f^oobar&mode=normal&limit=50&order_by=-ts'
{"data":"Syntax Error","status":400}
$ curl 'http://localhost:4040/indexes/posts/search?query=text:f`oobar&mode=normal&limit=50&order_by=-ts'
{"data":"channel closed","status":400}

logs:

Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | tantivy::directory::managed_directory | INFO  - Deleted "8ae9f9e93c674678ae3e7ab694752231.fast"
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | tantivy::directory::managed_directory | INFO  - Deleted "31b0bab77e014d539022907d36eac93c.fieldnorm"
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | tantivy::directory::managed_directory | INFO  - Deleted "2dbb35c78423479290186b0fccb9b48e.fieldnorm"
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | tantivy::directory::managed_directory | INFO  - Deleted "90fcfd004ee34f3892332a95d9c260e1.fast"
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210320997 ] completed operation DELETE-TERM
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210320998 ] completed operation ADD-DOCUMENT
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210320999 ] completed operation ADD-DOCUMENT
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | tantivy::directory::file_watcher | INFO  - Meta file "./lnx/index-data/posts/meta.json" was modified
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210321000 ] completed operation DELETE-TERM
Aug 30 00:52:18 torako lnx[307715]: [2021-08-30][00:52:18] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210321001 ] completed operation ADD-DOCUMENT
Aug 30 00:52:19 torako lnx[307715]: [2021-08-30][00:52:19] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210321002 ] completed operation DELETE-TERM
Aug 30 00:52:19 torako lnx[307715]: [2021-08-30][00:52:19] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210321003 ] completed operation ADD-DOCUMENT
Aug 30 00:52:19 torako lnx[307715]: [2021-08-30][00:52:19] | engine::index::writer | INFO  - [ WRITER @ posts ][ TRANSACTION 210321004 ] completed operation ADD-DOCUMENT
Aug 30 00:52:19 torako lnx[307715]: thread 'index-posts-worker-0' panicked at 'get executor', /root/lnx/engine/src/index/reader.rs:264:44
Aug 30 00:52:19 torako lnx[307715]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aug 30 00:52:19 torako lnx[307715]: [2021-08-30][00:52:19] | lnx::routes | WARN  - rejecting search index operation due to bad request: channel closed

My index settings are like so:

{
"writer_buffer": 144000000,
"writer_threads": 6,
"reader_threads": 6,

"max_concurrency": 12
}

Is it possible I exceeded the amount of concurrent requests allowed?

I can't seem to get any results from lnx. I'm using commit e9804944edc8a7c0af24ee3ba8397b87f1640b5f. I built lnx using cargo build --release, then starting it with /usr/local/bin/lnx -p 4040.

# cat a.json
{
  "name": "my-index",
  "writer_buffer": 6000000,
  "writer_threads": 1,
  "reader_threads": 1,
  "max_concurrency": 10,
  "search_fields": [
    "title"
  ],
  "storage_type": "memory",
  "use_fast_fuzzy": true,
  "strip_stop_words": true,
  "fields": {
    "title": {
      "type": "text",
      "stored": true
    },
    "description": {
      "type": "text",
      "stored": true
    },
    "id": {
      "type": "u64",
      "indexed": true,
      "stored": true,
      "fast": "single"
    }
  },
  "boost_fields": {
    "title": 2,
    "description": 0.8
  }
}
# curl -X POST [email protected] -H "Content-Type: application/json" http://127.0.0.1:4040/indexes
{"data":"index created","status":200}
# cat b.json
{
    "title": ["Hello, World"],
    "description": ["Welcome to the next generation system."]
}
# curl -X POST -H "Content-Type: application/json" [email protected] http://localhost:4040/indexes/my-index/documents?wait=true
{"data":"added documents","status":200}
# curl -X POST 'http://localhost:4040/indexes/my-index/commit'
{"data":"changes committed","status":200}
# curl 'http://localhost:4040/indexes/my-index/search?query=*'
{"data":{"count":0,"hits":[],"time_taken":0.001682035974226892},"status":200}
# curl 'http://localhost:4040/indexes/my-index/search?query=Hello'
{"data":{"count":0,"hits":[],"time_taken":0.00014333099534269422},"status":200}

I can't figure out what I'm doing wrong here.

Hiya!

I’m attempting to run LNX on Kubernetes as a stateful set, but running into an issue - the docs/example code suggests mounting a volume to etc/lnx but doing this in Kubernetes causes the contents of that path to be replaced with the attached volume, which means the binary can’t be found.

Is there a way to parametise the storage path so that it doesn’t collide with binary path? 🙂

Attempting to perform a search where the order_by field is a date leads to an error:

{"data":"Schema error: 'Field \"ts\" is of type I64!=Date'","status":400}

It looks this is because the FieldValue is implied to be i64:

https://github.com/lnx-search/lnx/blob/8d38d386de3ecdb1bced4f55032673860feaccf4/engine/src/index/reader.rs#L549-L552

Using master/0.9 beta and a INDEX with "use_fast_fuzzy": true results in corrupted data.

minimum example

1. create index with use_fast_fuzzy set to true:

{
    "override_if_exists": true,
    "index": {
        "name": "products",
        "storage_type": "tempdir",
        "fields": {
            "title": {
                "type": "text",
                "stored": true
            }
        },
        "search_fields": [],
        "boost_fields": {},
        "reader_threads": 1,
        "max_concurrency": 1,
        "writer_buffer": 300000,
        "writer_threads": 1,
        "set_conjunction_by_default": false,
        "use_fast_fuzzy": true,
        "strip_stop_words": false,
        "auto_commit": 0
    }
} 

2. send a document missing one of the defined fields, or having the vaule a empty sting or "-" or "_"

e.g: POST /indexes/products/documents with body {"title":""}

3.

POST /indexes/products/commit I get a error message: {"status":400,"data":"Data corrupted: 'Data corruption: : Failed to open field "title"'s term dictionary in the compos (truncated...)

workaround / recover

to recover from this I rebuild the index and add the document with a dummy value or I rebuild the index without using "use_fast_fuzzy".

Use case

The example above is not the real use case, usually titles on all my documents are set but they have some optional fields that are optional.

For example:

{
  "status": 200,
  "data": {
    "hits": [
      {
        "doc": {
          "author": [
            "248b2e6a-7c36-4da3-bcc4-55a979eb57dc"
          ],
          "id": [
            18
          ],
          "title": [
            "title 01"
          ],
          "uuid": [
            "06dbf5c7-d313-413d-8f65-49aed93e4031"
          ]
        },
        "document_id": "1628525110829290421",
        "score": 1.542423
      },
      {
        "doc": {
          "author": [
            "248b2e6a-7c36-4da3-bcc4-55a979eb57dc"
          ],
          "id": [
            19
          ],
          "title": [
            "title 02"
          ],
          "uuid": [
            "8da05387-8727-4a27-baa7-265af7558c0c"
          ]
        },
        "document_id": "1493516234521670736",
        "score": 1.542423
      },
      {
        "doc": {
          "author": [
            "248b2e6a-7c36-4da3-bcc4-55a979eb57dc"
          ],
          "id": [
            20
          ],
          "title": [
            "title 03"
          ],
          "uuid": [
            "3bf64ee1-f2ac-46ce-8e45-0d25956b195c"
          ]
        },
        "document_id": "9603160257558085701",
        "score": 1.542423
      }
    ],
    "count": 3,
    "time_taken": 0.000578893
  }
}

I think it would make much more sense to show the doc as it has been posted.

Reading through the docs and the source code it seems like you can specify which fields you can search a specific term for, so you can issue a query like:

{
    "query": [
         {"term": {"ctx": "Harry Potter", "fields": ["role"]}, "occur": "must"},
         {"term": {"ctx": "Daniel Radcliffe", "fields": ["actor"]}, "occur": "must"}
     ]
}

It would be really neat if it were possible to do the same for fuzzy queries so that something like this would be possible:

{
   "query": [
        {"fuzzy": {"ctx": "Barry Potter", "fields": ["role"]}, "occur": "must"},
        {"fuzzy": {"ctx": "Daniel Radclif", "fields": ["actor"]}, "occur": "must"}
    ]
}

To put this in a little more perspective in terms of a use case, suppose documents of movies, actors, and roles. I might have heard that the lead character of a movie is called Harry Potter but have no idea what movie this character belongs in, but I do want to know who the actor is.

If I were to create an index with the fuzzy method, I could create an index across all 3 fields, but when I search for Harry Potter I will get a bunch of results of actors on account of the movie being called Harry potter and the ...

Alternatively I could create separate indexes for each of these and search the individual index, but then I run into the problem that once I do have more information (like say movie name or actor name), I would have to compute a likelihood score myself from results of searching multiple indexes.

We should mark everything that gets serialized and deserialized with rkyv with the C layout repr, otherwise this could cause us some issues later on if rust's memory layout changes.

Now that we have the ability to write and produce index segments, we want to create a reader which can open a segment, and handle the deletes and query execution.

Currently, we can only combine two or more segments into one another, which although works, can make our index inefficient.

We should create a directory that can read from the segment, and split it out into a temp directory (this can be done with the writer directory) then tell tantivy to process all of the deletes marked within the index (providing it is safe to do so, see below) and then re-export the directory to a new segment.

Issue notes

• Deletes are not attached to the index that actually contains the documents intending to be deleted so we cannot blindly remove deletes.
• Some deletes may occur after a new document has been inserted and should not affect the new documents.