Getting random data corruption when using SQLite backend to run tests concurrently. Limiting test threads to 1 makes these errors disappear.

An example failure looks like:

thread 'data::treewalker::exec_test::set_queries' panicked at 'internal error: entered unreachable code', rdb-analyzer/src/data/treewalker/exec_test.rs:254:10

---- data::treewalker::asm::asm_test::basic_exec stdout ----
compile took 64.591668ms
TwScript { graphs: [TwGraph { name: "main", exported: false, nodes: [(LoadParam(0), [], None), (GetField(0), [0], None), (LoadConst(0), [], None), (LoadConst(1), [], None), (CreateMap, [], None), (InsertIntoMap(4), [3, 4], None), (InsertIntoMap(3), [2, 5], None), (BuildTable(2), [6], None), (InsertIntoTable(1), [7, 1], None), (LoadConst(2), [], None), (InsertIntoTable(5), [9, 1], None)], output: None, param_types: [0], output_type: None }], entry: 0, consts: [Primitive(Int64(1)), Primitive(Int64(2)), Primitive(String("test_name"))], idents: ["some_item", "duration", "Duration<int64>", "start", "end", "name"], types: [Schema] }
tyck took 123.214µs
exec took 821.355µs
Ok(None)
compile took 47.039006ms
TwScript { graphs: [TwGraph { name: "main", exported: false, nodes: [(LoadParam(0), [], None), (GetField(0), [0], None), (GetField(1), [1], None), (GetField(2), [1], None), (GetField(3), [1], None), (LoadConst(0), [], None), (Eq, [3, 5], None), (GetField(4), [4], Some(6)), (LoadConst(1), [], Some(6)), (Not, [6], None), (GetField(5), [4], Some(9)), (LoadConst(2), [], Some(9)), (Select, [7, 10], None), (Select, [8, 11], None), (GetField(6), [0], None), (LoadConst(3), [], None), (GetSetElement, [15, 14], None), (GetField(2), [16], None), (LoadConst(4), [], None), (GetSetElement, [18, 14], None), (GetField(2), [19], None), (LoadConst(5), [], None), (GetSetElement, [21, 14], None), (GetField(2), [22], None), (GetField(9), [1], None), (IsNull, [17], None), (Not, [25], None), (LoadConst(6), [], Some(25)), (Nop, [17], Some(26)), (Select, [27, 28], None), (CreateMap, [], None), (InsertIntoMap(13), [29, 30], None), (InsertIntoMap(12), [23, 31], None), (InsertIntoMap(11), [20, 32], None), (InsertIntoMap(10), [17, 33], None), (InsertIntoMap(9), [24, 34], None), (InsertIntoMap(8), [13, 35], None), (InsertIntoMap(7), [12, 36], None), (InsertIntoMap(2), [3, 37], None), (InsertIntoMap(1), [2, 38], None)], output: Some(39), param_types: [0], output_type: Some(1) }], entry: 0, consts: [Primitive(String("test")), Primitive(String("start")), Primitive(String("end")), Primitive(String("xxx")), Primitive(String("yyy")), Primitive(String("zzz")), Primitive(String("<unknown>"))], idents: ["some_item", "id", "name", "duration", "start", "end", "many_items", "value", "kind", "altname", "set_member_name_1", "set_member_name_2", "set_member_name_3", "set_member_name_1_nonnull"], types: [Schema, Map(RedBlackTreeMap { root: Some(Node { entry: Entry { key: "name", value: Primitive(String) }, color: Black, left: Some(Node { entry: Entry { key: "id", value: Primitive(String) }, color: Black, left: Some(Node { entry: Entry { key: "altname", value: Primitive(String) }, color: Red, left: None, right: None }), right: Some(Node { entry: Entry { key: "kind", value: Primitive(String) }, color: Red, left: None, right: None }) }), right: Some(Node { entry: Entry { key: "set_member_name_2", value: Primitive(String) }, color: Red, left: Some(Node { entry: Entry { key: "set_member_name_1", value: Primitive(String) }, color: Black, left: None, right: Some(Node { entry: Entry { key: "set_member_name_1_nonnull", value: Primitive(String) }, color: Red, left: None, right: None }) }), right: Some(Node { entry: Entry { key: "value", value: Primitive(Int64) }, color: Black, left: Some(Node { entry: Entry { key: "set_member_name_3", value: Primitive(String) }, color: Red, left: None, right: None }), right: None }) }) }), size: 9 })] }
tyck took 186.836µs
exec took 2.659092ms
Ok(None)
thread 'data::treewalker::asm::asm_test::basic_exec' panicked at 'called `Option::unwrap()` on a `None` value', rdb-analyzer/src/data/treewalker/asm/asm_test.rs:174:21

---- data::treewalker::asm::asm_test::list_ops stdout ----
compile took 285.62346ms
TwScript { graphs: [TwGraph { name: "main", exported: false, nodes: [(LoadParam(0), [], None), (LoadConst(0), [], None), (LoadConst(1), [], None), (LoadConst(2), [], None), (LoadConst(3), [], None), (LoadConst(4), [], None), (CreateList(2), [], None), (PrependToList, [5, 6], None), (PrependToList, [4, 7], None), (PrependToList, [3, 8], None), (PrependToList, [2, 9], None), (PrependToList, [1, 10], None), (CreateMap, [], None), (LoadConst(5), [], None), (Reduce(3, false), [12, 13, 11], None), (LoadConst(6), [], None), (Call(2), [15], None), (CreateMap, [], None), (LoadConst(5), [], None), (Reduce(3, false), [17, 18, 16], None), (CreateMap, [], None), (InsertIntoMap(3), [19, 20], None), (InsertIntoMap(2), [16, 21], None), (InsertIntoMap(1), [14, 22], None), (InsertIntoMap(0), [11, 23], None), (GetField(5), [0], None), (CreateMap, [], None), (CreateList(3), [], None), (Reduce(1, false), [26, 27, 11], None), (BuildSet, [28], None), (InsertIntoTable(4), [29, 25], None), (GetField(5), [0], None), (CreateMap, [], None), (CreateList(3), [], None), (Reduce(1, false), [32, 33, 16], None), (BuildSet, [34], None), (InsertIntoTable(6), [35, 31], None)], output: Some(24), param_types: [0], output_type: Some(1) }, TwGraph { name: "transform_numbers", exported: false, nodes: [(LoadParam(0), [], None), (LoadParam(1), [], None), (LoadParam(2), [], None), (CreateMap, [], None), (InsertIntoMap(8), [2, 3], None), (BuildTable(7), [4], None), (PrependToList, [5, 1], None)], output: Some(6), param_types: [4, 5, 2], output_type: Some(5) }, TwGraph { name: "gen_numbers", exported: false, nodes: [(LoadParam(0), [], None), (LoadConst(5), [], None), (Eq, [0, 1], None), (CreateList(2), [], Some(2)), (Not, [2], None), (LoadConst(4), [], Some(4)), (Sub, [0, 5], Some(4)), (Call(2), [6], Some(4)), (PrependToList, [0, 7], Some(4)), (Select, [3, 8], None)], output: Some(9), param_types: [2], output_type: Some(6) }, TwGraph { name: "sum", exported: false, nodes: [(LoadParam(0), [], None), (LoadParam(1), [], None), (LoadParam(2), [], None), (Add, [1, 2], None)], output: Some(3), param_types: [4, 2, 2], output_type: Some(2) }], entry: 0, consts: [Primitive(Int64(5)), Primitive(Int64(4)), Primitive(Int64(3)), Primitive(Int64(2)), Primitive(Int64(1)), Primitive(Int64(0)), Primitive(Int64(20))], idents: ["list1", "sum1", "list2", "sum2", "collection_a", "store", "collection_b", "Number<>", "value"], types: [Schema, Map(RedBlackTreeMap { root: Some(Node { entry: Entry { key: "list2", value: List(VmListType { ty: Primitive(Int64) }) }, color: Black, left: Some(Node { entry: Entry { key: "list1", value: List(VmListType { ty: Primitive(Int64) }) }, color: Black, left: None, right: None }), right: Some(Node { entry: Entry { key: "sum1", value: Primitive(Int64) }, color: Black, left: None, right: Some(Node { entry: Entry { key: "sum2", value: Primitive(Int64) }, color: Red, left: None, right: None }) }) }), size: 4 }), Primitive(Int64), Table(VmTableType { name: "Number<>" }), Map(RedBlackTreeMap { root: None, size: 0 }), List(VmListType { ty: Table(VmTableType { name: "Number<>" }) }), List(VmListType { ty: Primitive(Int64) })] }
tyck took 367.834µs
exec took 8.56912ms
Ok(Some(Map(VmMapValue { elements: RedBlackTreeMap { root: Some(Node { entry: Entry { key: "sum1", value: Primitive(Int64(15)) }, color: Black, left: Some(Node { entry: Entry { key: "list2", value: List(VmListValue { member_ty: Primitive(Int64), node: List { head: Some(Node { value: Primitive(Int64(20)), next: Some(Node { value: Primitive(Int64(19)), next: Some(Node { value: Primitive(Int64(18)), next: Some(Node { value: Primitive(Int64(17)), next: Some(Node { value: Primitive(Int64(16)), next: Some(Node { value: Primitive(Int64(15)), next: Some(Node { value: Primitive(Int64(14)), next: Some(Node { value: Primitive(Int64(13)), next: Some(Node { value: Primitive(Int64(12)), next: Some(Node { value: Primitive(Int64(11)), next: Some(Node { value: Primitive(Int64(10)), next: Some(Node { value: Primitive(Int64(9)), next: Some(Node { value: Primitive(Int64(8)), next: Some(Node { value: Primitive(Int64(7)), next: Some(Node { value: Primitive(Int64(6)), next: Some(Node { value: Primitive(Int64(5)), next: Some(Node { value: Primitive(Int64(4)), next: Some(Node { value: Primitive(Int64(3)), next: Some(Node { value: Primitive(Int64(2)), next: Some(Node { value: Primitive(Int64(1)), next: None }) }) }) }) }) }) }) }) }) }) }) }) }) }) }) }) }) }) }) }), last: Some(Primitive(Int64(1))), length: 20 } }) }, color: Black, left: Some(Node { entry: Entry { key: "list1", value: List(VmListValue { member_ty: Primitive(Int64), node: List { head: Some(Node { value: Primitive(Int64(5)), next: Some(Node { value: Primitive(Int64(4)), next: Some(Node { value: Primitive(Int64(3)), next: Some(Node { value: Primitive(Int64(2)), next: Some(Node { value: Primitive(Int64(1)), next: None }) }) }) }) }), last: Some(Primitive(Int64(1))), length: 5 } }) }, color: Red, left: None, right: None }), right: None }), right: Some(Node { entry: Entry { key: "sum2", value: Primitive(Int64(210)) }, color: Black, left: None, right: None }) }), size: 4 } })))
compile took 63.291593ms
TwScript { graphs: [TwGraph { name: "main", exported: false, nodes: [(LoadParam(0), [], None), (CreateMap, [], None), (CreateList(2), [], None), (GetField(1), [0], None), (GetField(0), [3], None), (Reduce(1, false), [1, 2, 4], None), (CreateMap, [], None), (CreateList(2), [], None), (GetField(1), [0], None), (GetField(2), [8], None), (Reduce(1, false), [6, 7, 9], None), (CreateMap, [], None), (InsertIntoMap(4), [10, 11], None), (InsertIntoMap(3), [5, 12], None)], output: Some(13), param_types: [0], output_type: Some(1) }, TwGraph { name: "decompose_numbers", exported: false, nodes: [(LoadParam(0), [], None), (LoadParam(1), [], None), (LoadParam(2), [], None), (GetField(5), [2], None), (PrependToList, [3, 1], None)], output: Some(4), param_types: [3, 4, 5], output_type: Some(4) }], entry: 0, consts: [], idents: ["collection_a", "store", "collection_b", "list1", "list2", "value"], types: [Schema, Map(RedBlackTreeMap { root: Some(Node { entry: Entry { key: "list1", value: List(VmListType { ty: Primitive(Int64) }) }, color: Black, left: None, right: Some(Node { entry: Entry { key: "list2", value: List(VmListType { ty: Primitive(Int64) }) }, color: Red, left: None, right: None }) }), size: 2 }), Primitive(Int64), Map(RedBlackTreeMap { root: None, size: 0 }), List(VmListType { ty: Primitive(Int64) }), Table(VmTableType { name: "Number<>" })] }
tyck took 289.364µs
exec took 968.593µs
Ok(Some(Map(VmMapValue { elements: RedBlackTreeMap { root: Some(Node { entry: Entry { key: "list2", value: List(VmListValue { member_ty: Primitive(Int64), node: List { head: None, last: None, length: 0 } }) }, color: Black, left: Some(Node { entry: Entry { key: "list1", value: List(VmListValue { member_ty: Primitive(Int64), node: List { head: None, last: None, length: 0 } }) }, color: Red, left: None, right: None }), right: None }), size: 2 } })))
thread 'data::treewalker::asm::asm_test::list_ops' panicked at 'assertion failed: `(left == right)`
  left: `0`,
 right: `5`', rdb-analyzer/src/data/treewalker/asm/asm_test.rs:475:67


failures:
    data::treewalker::asm::asm_test::basic_exec
    data::treewalker::asm::asm_test::list_ops
    data::treewalker::exec_test::basic_exec
    data::treewalker::exec_test::set_queries

test result: FAILED. 30 passed; 4 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.13s

Welcome to Renovate! This is an onboarding PR to help you understand and configure settings before regular Pull Requests begin.

🚦 To activate Renovate, merge this Pull Request. To disable Renovate, simply close this Pull Request unmerged.

Detected Package Files

• rdb-analyzer/Cargo.toml (cargo)
• rdb-pgsvc/Cargo.toml (cargo)
• rdb-proto/Cargo.toml (cargo)
• rdb-server/Cargo.toml (cargo)
• rdbctl/Cargo.toml (cargo)
• rdb-server.Dockerfile (dockerfile)
• .github/workflows/ci.yml (github-actions)

Configuration

🔡 Renovate has detected a custom config for this PR. Feel free to ask for help if you have any doubts and would like it reviewed.

Important: Now that this branch is edited, Renovate can't rebase it from the base branch any more. If you make changes to the base branch that could impact this onboarding PR, please merge them manually.

What to Expect

With your current configuration, Renovate will create 27 Pull Requests:

🚸 Branch creation will be limited to maximum 2 per hour, so it doesn't swamp any CI resources or spam the project. See docs for prhourlylimit for details.

❓ Got questions? Check out Renovate's Docs, particularly the Getting Started section. If you need any further assistance then you can also request help here.

This PR has been generated by Mend Renovate. View repository job log here.

Given a schema and its associated database, there is currently no way to navigate through the data other than writing a RefineAsm program and invoking it to do the query. It would be great if we have a user interface (GUI/TUI) to view and modify the structured data without writing any code.

No one is currently working on this. Please comment if you're interested!

HSE is a fast embeddable key-value store designed for SSDs and persistent memory built by Micron. It would be nice to support HSE as a KV backend in RefineDB.

Given a table like:

type PostInfo {
  author: string,
  title: string,
  content: string,
  create_time: string,
}

It's often useful to store all fields in a single key-value entry to reduce query overhead.

That's what the @packed attribute is for:

type Post {
  @primary
  id: string,
  @packed
  info: PostInfo,
}

However currently the @packed attribute is only implemented on the syntax level, and not pushed down to either the storage plan or TreeWalker.

Changes needed for @packed:

• Storage planner: Treat packed fields as a leaf node.
• TreeWalker: Keep track of the packed/unpacked kind of any set and table DFG nodes.
• TreeWalker: Lens-like structure for looking inside a packed field.