A scalable, distributed, collaborative, document-graph database, for the realtime web

Describe the bug

OS: Windows 10 x64

PS C:\WINDOWS\system32> iwr https://windows.surrealdb.com -useb | iex

 .d8888b.                                             888 8888888b.  888888b.
d88P  Y88b                                            888 888  'Y88b 888  '88b
Y88b.                                                 888 888    888 888  .88P
 'Y888b.   888  888 888d888 888d888  .d88b.   8888b.  888 888    888 8888888K.
    'Y88b. 888  888 888P'   888P'   d8P  Y8b     '88b 888 888    888 888  'Y88b
      '888 888  888 888     888     88888888 .d888888 888 888    888 888    888
Y88b  d88P Y88b 888 888     888     Y8b.     888  888 888 888  .d88P 888   d88P
 'Y8888P'   'Y88888 888     888      'Y8888  'Y888888 888 8888888P'  8888888P'

Fetching the latest database version...
Fetching the host system architecture...
Installing surreal-v1.0.0-beta.5
 for windows-amd64...
Invoke-WebRequest : The remote server returned an error: (403) Forbidden.
At line:54 char:5
+     Invoke-WebRequest $DownloadUrl -OutFile $Executable -UseBasicPars ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-WebRequest], WebExc
   eption
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand

PS C:\WINDOWS\system32>

Steps to reproduce

Run the installation script

iwr https://windows.surrealdb.com -useb | iex

Expected behaviour

It's supposed to install SurrealDB.

SurrealDB version

1.0.0-beta.5

Contact Details

[email protected]

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Describe the bug

I get the following error while using future records in a certain way:

thread 'tokio-runtime-worker' has overflowed its stack
fatal runtime error: stack overflow
[1]    12935 abort      surreal start --user root --pass root --bind 0.0.0.0:3001 file:testdb

Steps to reproduce

Define the fields as such:

DEFINE FIELD managers ON TABLE organisation TYPE array;
DEFINE FIELD organisations ON TABLE manager VALUE <future> {(SELECT id FROM organisation WHERE managers CONTAINS $parent.id)};

Insert some data:

CREATE manager:john SET name="John Doe";
CREATE organisation:doefoundation SET managers = [manager:john];

Select the data:

SELECT * FROM manager;

Expected behaviour

For it not to crash :D

SurrealDB version

Compiled from source on M1 macbook

Contact Details

[email protected]

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

What is the motivation?

Currently building from source can be frustrating, especially when one wants to enable either TiKV or FoundationDB. This is an attempt to make builds easier and provide a more consistent user experience across different systems. Building from source should be simple and painless regardless of the user's experience with Rust and C/C++.

Ideally, one shouldn't even need to know what the build dependencies of this project are or what language it's written in in order to be able to compile it. Once they install the build tool, running it or building it from source should just work and be a single command away without even needing to clone the repo first.

After cloning this repo, Rust developers too should be able to start making changes after running only a single command that prepares their environment for all the dependencies they need to build successfully. This is, of course, assuming they already have our build tool installed and configured. Both of which should, theoretically at least, be simple and straight-forward steps. It's possible to eliminate that single command entirely and have the environment be prepared automatically once one cds into this project.

I know these are hefty promises :smile: On paper, the build tool I'm proposing comes with all these features and more. However whilst I have been using it since 2014 and I love and swear by it, I have never used it outside Linux distros so your mileage may vary if you use a different system.

What does this change do?

This changeset packages this repo using the Nix package manager. More specifically it turns this repo into a Nix flake.

• [X] Add a basic Nix flake (POC)
• [x] Add support for cross compiling to different targets
• [x] Add support for building static binaries
• [x] Add a test in the CI
• [x] Add documentation

Once this PR lands one can run the server without even needing to clone the repo first by running

nix run github:surrealdb/surrealdb

This will download the latest commit on main, compile and run it. Specific branches, tags or even commits can be specified easily too, for example nix run github:surrealdb/surrealdb/v1.0.0-beta.8. If we add this project to the Flake registry, that command can even be shortened to nix run surrealdb or nix run surreal.

To just install it in one's PATH

nix profile install github:surrealdb/surrealdb

Developers will simply need to run nix develop to prepare their environment so that cargo build just works (rustup won't be needed as Nix takes over the functions of installing the Rust toolchain and targets).

What is your testing strategy?

Made sure that all tests still pass and added more CI jobs to test building with Nix.

Is this related to any issues?

It directly addresses https://github.com/surrealdb/surrealdb/issues/97. It will also help with https://github.com/surrealdb/surrealdb/issues/52 though it requires that nix develop be run first before cargo build.

Have you read the Contributing Guidelines?

• [X] I have read the Contributing Guidelines

Describe the bug

Selecting a record that contains a MultiPolygon crashes the database.

Steps to reproduce

Run the following documented query:

UPDATE university:oxford SET locations = {
	type: "MultiPolygon",
	coordinates: [
		[
			[ [10.0, 11.2], [10.5, 11.9], [10.8, 12.0], [10.0, 11.2] ]
		],
		[
			[ [9.0, 11.2], [10.5, 11.9], [10.3, 13.0], [9.0, 11.2] ]
		]
	]
};

Select the record or export the database:

SELECT * FROM university:oxford;

The database crashes with the following error message:

thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Syntax("unknown variant `Polygons`, expected one of `Point`, `Line`, `Polygon`, `MultiPoint`, `MultiLine`, `MultiPolygon`, `Collection`")', lib/src/sql/value/value.rs:97:60

Expected behaviour

Selects should return the record successfully and exporting should work as expected. I have already identified the source of the bug and will prepare and submit a pull request shortly.

SurrealDB version

surreal 1.0.0-beta.7 for linux on x86_64

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Thank you for submitting this pull request! We appreciate you spending the time to work on these changes.

What is the motivation?

The flake is amazing, however it doesn't build on darwin (macos)

What does this change do?

This fixes the build for darwin platforms. tested only on x86_64 though.

What is your testing strategy?

run nix build .#packages.x86_64-darwin.default on a darwin machine and verify the output of ./result/bin/surreal version

Is this related to any issues?

If this pull request is related to any other pull request or issue, or resolves any issues - then link all related pull requests and issues here.

Have you read the Contributing Guidelines?

• [X] I have read the Contributing Guidelines

Thank you for submitting this pull request! We appreciate you spending the time to work on these changes.

What is the motivation?

While I have yet to actually use this method for anything, according to the Contribution Guide "Query execution time" is very important and this will lower it exponentially in the use of this method.

However, the time taken for hashing, might be slower for small Arrays with large items, but I think the time will be negligible. Or we could add a condition for < 10 Values to use the O(n^2) method.

What does this change do?

Added #[derive(Hash)] to almost every struct under lib/src/sql (anything that linked back to Value). And changed the Uniq<Array> implementation to use a HashSet rather than having a nested loop.

Some things I'm not sure are the best I've had to do, such as the implementation for Hash on Number is as follows

https://github.com/surrealdb/surrealdb/blob/84be58dda0d44b3670c30a649b4e056cef54ffb4/lib/src/sql/number.rs#L323-L331

And according to what I've researched, this works, except has unintuitive behavior for NaNs

Then, for Geometry, the only sensible way I could tell to do this is with

https://github.com/surrealdb/surrealdb/blob/84be58dda0d44b3670c30a649b4e056cef54ffb4/lib/src/sql/geometry.rs#L511-L515

And while this might make it a bit slower for Values containing Geometry Values, I think it should still work as the method is idempotent.

What is your testing strategy?

1. Ran cargo test and it passed
2. Built and ran SurrealDB
3. Created a record with 10k values each a random number between 0 and 2999
4. Ran SELECT array::distinct(v) FROM test:test;
5. Did steps 2, 3 and 4 on the main release build of SurrealDB
6. Compared the output of steps 4 and 5 to verify they are the same and that 4 was faster

Is this related to any issues?

Fixes #1309

Have you read the Contributing Guidelines?

• [X] I have read the Contributing Guidelines

Is your feature request related to a problem?

I just came across SurrealDB and was curious about the underlying system in place. However, very soon I realized that calling Surreal a database is extremely misleading, especially from an academic point of view (and since I research database systems this is very incorrect).

What you're building is essentially a "query engine" or "data API" on top of other actual databases. It is like what Hasura and Timescale are to Postgres. And they clearly don't say that they are "ultimate databases" in their own right. It is like saying a REST API providing access to SQLite is a database.

Describe the solution

I like the idea of being able to choose a database engine, but please reconsider your framing of what Surreal actually is to avoid confusion. "End to end" and "ultimate" are very misleading.

Alternative methods

NA

SurrealDB version

1.0.0-beta.7

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

What is the motivation?

Since surrealdb can be embedded into surrealdb-rs, it makes sense to combine these two crates so we can expose a single API that works for both embedded and remote databases. This eliminates confusion as to which crate to use.

Previously the answer to that question would be “use surrealdb if you want an embedded database or use surrealdb-rs if you want to connect to a remote one”. Now the answer to this question is more nuanced.

What does this change do?

It merges surrealdb-rs into surrealdb, making the former the public interface of the latter.

What is your testing strategy?

Add more automated tests.

Is this related to any issues?

It fixes https://github.com/surrealdb/surrealdb.rs/issues/1 though it's not specifically created to address it.

Have you read the Contributing Guidelines?

• [x] I have read the Contributing Guidelines

Is your feature request related to a problem?

Mainly, just to have concise API, where any expression can be standard SRQL or use embedded JS as an escape hatch for even more awesomeness.

Currently, trying to define a function for at least WHEN clause, throws a Parse error.

Describe the solution

Essentially, allow us to:

DEFINE EVENT whatever ON TABLE sometable WHEN function () { return this.event == "CREATE" } THEN (
    CREATE whatever;
)

Of course, with the power of #1320, we might as well:

DEFINE EVENT whatever ON TABLE sometable WHEN function () {
    if (this.event == "CREATE") {
        const otherCount = await query("SELECT count() FROM other GROUP BY ALL");
        return otherCount > 0;
    } 
    return false;
}

Should be possible to define a function in THEN clause as well:

DEFINE EVENT whatever ON TABLE sometable WHEN $event = "CREATE" THEN function () {
    const dependency = await query("SELECT * FROM dependency");
    const result = await query(`CREATE target CONTENT { dependencies: ${JSON.stringify(dependency) }`);
    -- or better yet, introduce `create` etc. methods
}

Alternative methods

Not applicable here, as this is a generic feature request, where the API is inconsistent.

In case there are other places where embedded JS is not usable, this in theory should track that as well.

SurrealDB version

surreal 1.0.0-beta.8+20220930.c246533 for macos on aarch64

Contact Details

@tomsseisums

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

What is the motivation?

The current version reported, for example 1.0.0-beta.7 does not carry enough information to pinpoint the exact commit built when building from source. SemVer allows for adding additional build metadata by adding it after the + sign. This can be very helpful for debugging.

What does this change do?

It makes it possible to specify optional build metadata using the SURREAL_BUILD_METADATA environment variable. Build tools like Nix can set this variable to something like 20220926.a72052a. The first part is the date on which the commit being built was made. The second part is the short form of the commit. In this case the version reported by surreal version will be 1.0.0-beta.7+20220926.a72052a.

What is your testing strategy?

Made sure tests still pass and tested the functionality in a separate PR.

Is this related to any issues?

Extracted from https://github.com/surrealdb/surrealdb/pull/100.

Have you read the Contributing Guidelines?

• [x] I have read the Contributing Guidelines

Describe the bug

I was in the process of writing some benchmarks with Benchmark.js & nodemon to restart my tests on save when I came across an error I believe should be handled in RUST.

Basically, in between a running a benchmark for the update query, I updated the source causing nodemon to restart the process. In effect this sudden disconnection of the client caused SurrealDB to completely panic and die.

thread 'tokio-runtime-worker' panicked at 'called `Result::unwrap()` on an `Err` value: Io(Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" })', src/net/rpc.rs:64:37
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Aborted (core dumped)

I think we should handle the Result rather than unwrap to avoid killing the database process.

Steps to reproduce

1. Run Surrealdb NodeJs client query on a loop (as would happen in a bechmark)
2. Then suddenly kill the client to abruptly disconnect from the server.
3. The Tokio thread dies in a panic.

Expected behaviour

The match expression should be used to handle the Result instead of Result::unwrap() which currently causes the database process to panic and terminate.

SurrealDB version

1.0.0-beta.7

Contact Details

[email protected]

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Describe the bug

It seems that the sql! macro is not correctly parsing queries with functions.

Steps to reproduce

Running the following query in rust:

db.query(sql!(SELECT * FROM time::now())).await?.take(0)?

Throws the following error

Parse error on line 1 at character 19 when parsing ':: now() ;'

Expected behaviour

It should work. Tested via a rest client directly to the database and the query works as expected.

SurrealDB version

23be3353be0974dfaaf52c4a77a5e38817c0aa62

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Is your feature request related to a problem?

Testing distributed systems is very hard.

Describe the solution

Incorporate https://github.com/tokio-rs/turmoil in our test suite.

Alternative methods

N/A

SurrealDB version

N/A

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

Is your feature request related to a problem?

There is no simple way to access parameters defined within SurrealQL from within the context of a JavaScript function.

This is related to #1322.

Describe the solution

There are three potential solutions here:

1. We make all parameters available on a global params variable...

DEFINE EVENT js_when_srql_then ON sometable WHEN fn() { return params.event == "CREATE" } THEN (CREATE some_event_result);

2. We make all parameters available as own variable names prefixed with a $ symbol...

DEFINE EVENT js_when_srql_then ON sometable WHEN fn() { return $event == "CREATE" } THEN (CREATE some_event_result);

3. We make all parameters available with a function. This is the most performant, but the worst in terms of experience...

DEFINE EVENT js_when_srql_then ON sometable WHEN fn() { return param('event') == "CREATE" } THEN (CREATE some_event_result);

Passing the parameters on the this context isn't really an option, because the this always refers to the current record being processed, and therefore this.event would refer to the event field on the current record.

Alternative methods

Currently this can be achieved by passing in the required parameters as function arguments.

DEFINE EVENT js_when_srql_then ON sometable WHEN function ($event) { return arguments[0] == "CREATE" } THEN (CREATE some_event_result);

SurrealDB version

surreal 1.0.0-beta.8+20220930.c246533 for macos on aarch64

Contact Details

No response

Is there an existing issue for this?

• [X] I have searched the existing issues

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

What is the motivation?

The CI is taking longer and longer to run as we add more tests.

What does this change do?

It improves CI build times by splitting it into more jobs that can be run in parallel and carefully considering the features enabled.

What is your testing strategy?

Ensure the CI still runs correctly.

Is this related to any issues?

No.

Have you read the Contributing Guidelines?

• [x] I have read the Contributing Guidelines

What is the motivation?

Currently the database error variant leads to messages like

There was a problem with the database: Database error: Specify a namespace to use

What does this change do?

It shortens the message to something like

Database error: Specify a namespace to use

What is your testing strategy?

Ensure CI passes.

Is this related to any issues?

No.

Have you read the Contributing Guidelines?

• [x] I have read the Contributing Guidelines

Thank you for submitting this pull request! We appreciate you spending the time to work on these changes.

What is the motivation?

See #1393 - Illegal datetimes panic, such as November 31st (2022-11-31T12:00:00.000Z)

What does this change do?

Unifies the parsing of datetimes into one function that errors when invalid dates are entered. The errors comply with how nom expects it to and will try a new branch when one does occur; this results in nom parsing the invalid datetime as a string.

Additionally, I have changed the ranges for seconds and hours to 0..=60 and 0..=23 respectively. The 60 for seconds is for leap seconds, and 24 is an invalid hour in the ISO 8601 spec (and RFC 3339)

What is your testing strategy?

Created a new test, date_time_illegal_date, in lib/src/sql/datetime.rs

Is this related to any issues?

Fixes #1393

Have you read the Contributing Guidelines?

• [x] I have read the Contributing Guidelines