Due to the way signed integers are encoded (two's complement), even if we flip MSB to be 0 for negative and 1 for positive, it would "still incorrectly compare negative INTs, for example: 0b00000001 (-1) 0b00000010 (-2) Using LT? and GT? with this encoding would mean that -2 > -1 (the same problem we have right now with INTs), and changing LT? and GT? to take the sign bit into account would break comparisons between unsigned INTs, because 0b01111111(unsigned 127) would be interpreted as -127." @Matt8898 (see #265)

It is an important issue as order key navigation is fairly critical to overall usefulness of PumpkinDB as a database engine.

Proposed solution (long term): explore the direction of maintaining multiple b-trees or collation sequencing for key ranges.

Any other, less invasive solutions?

Basically, it means that after a restart values returned by HLC will be less than those generated before.

Proposed solution: let timestamp module persist last generated HLC.

Letting it store in the same database is rather intricate: there might be no write txn, the write txn might never succeed, etc. So I propose that PumpkinDB maintains a separate "meta" database within the environment with its own transactions.

Any better solution (that would involve I/O penalty?)

Solution: This PR introduces a memory mapped meta database that is used by modules to persist whatever state they might need to reconstruct their previous state after a restart. For now this is only done for HLCs (see 77b2506).

Additionally, remove excessive usage of lazy_static (06e1156, 5200d65, 4c7065f).

Compiling PumpkinDB takes a while, sometimes making it pretty annoying to work on.

There are a few possible reasons as to why this could be:

1 Lots of macros being used; 1 Lots of manual in-lining; 1 Code size.

Maybe compile time can be brought down by breaking the project into sub-crates? I'm not sure how cargo goes about compiling then.

It is not a problem right now, but soon enough multi-VM setups will be a thing (use all the cores!) and they need to synchronize their access to writing, since lmdb only permits one active writer

Technically speaking, loaded words can define their own words but they will leak into the remainder of the program, which less than ideal.

Using stack alone requires a significant amount of juggling that makes writing code incredibly frustrating. I personally believe that the value of stack based programming languages is in concatenative abilities (composition via stack), not just being able to do everything by juggling items in the stack.

Solution: make all SETs and DEFs done within any closure local to that closure.

This means that if eventually we'll need to do code injection that's not injecting what effectively amounts to closures, we'll need to have a separate pass result type that can indicate that.

Wherever possible, typing conventions in PumpkinScript enforce lexicographical value sorting, i.e. if a value is considered to be smaller, it should appear earlier during cursor iteration.

Recent changes introduced in #258 break this convention:

PumpkinDB> -100i8 +1i8 LT?.
0x00 (incorrect)
PumpkinDB> -100i8 -1i8 LT?.
0x01 (correct)
PumpkinDB> -2i8 -0i8 LT?.
0x00 (incorrect)

Proposed solution:

I am not sure if there is a good solution to this, given how signed integers are encoded, but I think at the very least it is worth to highlight this fact (if we don't find any reasonable solution, we should explicitly document this).

It might also be valuable to add INTxx/LT? and GT? since INT/LT? and GT? don't work on sized ints and LT and GT? don't produce desirable solution.

Any other thoughts?

cc @Matt8898

One has to make sure that they are hexadecimal and little-endian.

Proposed solution: prohibit words to start with - and 0..9 and allow these to be interpreted as bigint/biguints.

Solution: Implement HLC/OBSERVE instruction

Addresses #199

NOTE This commit uses a fork of the hybrid-clocks repo. Changes to it are necessary for tests to pass.

Every contributor is mentioned in a dedicated file anyway, so maintaining a list of contributors in each toml file is rather cumbersome and redundant.

Solution: don't mention contributors in Cargo.toml's.

This means potential binary distribution challenges as some libraries might not be available.

Proposed solution: use Rust capabilities where possible (= Linux at the moment)

See https://blog.rust-lang.org/2016/05/13/rustup.html

Hello :crab:, we (Rust group @sslab-gatech) found a memory-safety/soundness issue in this crate while scanning Rust code on crates.io for potential vulnerabilities.

Issue Description

https://github.com/PumpkinDB/PumpkinDB/blob/f37b0b586357cafd2fd0eba8bea9072665af0b00/pumpkindb_client/src/packet.rs#L50-L55

PacketReader::read() method creates an uninitialized buffer and passes it to user-provided Read implementation. This is unsound, because it allows safe Rust code to exhibit an undefined behavior (read from uninitialized memory).

This part from the Read trait documentation explains the issue:

It is your responsibility to make sure that buf is initialized before calling read. Calling read with an uninitialized buf (of the kind one obtains via MaybeUninit<T>) is not safe, and can lead to undefined behavior.

How to fix the issue?

The Naive & safe way to fix the issue is to always zero-initialize a buffer before lending it to a user-provided Read implementation. Note that this approach will add runtime performance overhead of zero-initializing the buffer.

As of Jan 2021, there is not yet an ideal fix that works in stable Rust with no performance overhead. Below are links to relevant discussions & suggestions for the fix.

• Well-written document regarding the issue
• Rust RFC 2930
• nightly feature std::io::Initializer
• Discussion in Rust Internals Forum

Hi, I worked on making PumpkinDB compile on Linux again. I also removed deprecation warnings along the way and adopted stabilized features regarding slice patterns in Rust. As I'm not very familiar with contributing to foreign code bases please check and provide me feedback in case this pull request is strange or wrong.

I would like to see this pull request merge thus giving PumpkinDB kind of revival. What do you think?

Thank you.

Kind regards

Tobias

BTW I also checked build with Travis beforehand.

Running cargo bench acker in the pumpkindb_engine crate:

nightly-2017-06-20: (last known "good" nightly)

test script::tests::ackermann                                                  ... bench:  32,562,056 ns/iter (+/- 4,313,209)
test script::tests::ackermann_stack                                            ... bench:  24,391,250 ns/iter (+/- 4,758,534)

Most likely "first offender"

nightly-2017-06-21: (https://github.com/rust-lang/rust/commit/445077963)

test script::tests::ackermann                                                  ... bench:  99,398,843 ns/iter (+/- 5,556,874)
test script::tests::ackermann_stack                                            ... bench:  51,625,752 ns/iter (+/- 3,971,108)

Random builds:

2017-07-24:

test script::tests::ackermann                                                  ... bench:  49,617,433 ns/iter (+/- 6,028,955)
test script::tests::ackermann_stack                                            ... bench:  36,797,438 ns/iter (+/- 4,442,438)

2017-07-01:

test script::tests::ackermann                                                  ... bench:  97,953,353 ns/iter (+/- 5,756,554)
test script::tests::ackermann_stack                                            ... bench:  51,599,955 ns/iter (+/- 3,756,541)

2017-07-12:

test script::tests::ackermann                                                  ... bench:  50,183,660 ns/iter (+/- 7,364,549)
test script::tests::ackermann_stack                                            ... bench:  39,244,258 ns/iter (+/- 6,607,862)

2017-06-23:

test script::tests::ackermann                                                  ... bench: 111,698,229 ns/iter (+/- 17,495,313)
test script::tests::ackermann_stack                                            ... bench:  58,807,694 ns/iter (+/- 4,591,336)

Last build:

2017-08-02:

test script::tests::ackermann                                                  ... bench:  49,253,683 ns/iter (+/- 9,255,754)
test script::tests::ackermann_stack                                            ... bench:  35,636,600 ns/iter (+/- 5,089,613)

Proposed solution: figure out what happened in the first place, why did it get so much slower?

Currently, builtins! uses lazy_static! to prepare BUILTINS map of instructions. This means that until handle_builtins is first called, nothing is going to get parsed. So if there's, for example, a parsing mistake, the whole engine will crash.

Proposed solution: find a way to parse builtins during compile time

For most end-user cases, PumpkinDB is too "close to the metal" to be easily plugged into a business application, especially if "event sourcing" type of scenarios are considered.

Proposed solution: implement a higher level database on top of PumpkinDB