Description of bug Persistence is broken since the last tag. A more concrete reproduction is yet to be produced.

Steps to reproduce Steps to reproduce the behavior:

1. Run skyd
2. Launch skysh and run SET somekey somevalue
3. Close skysh and then close skyd
4. Launch skyd
5. Launch skysh and run GET somekey
6. You'll get a (Nil) printed on the screen

Expected behavior "somevalue" should have been printed

Meta

• Release tag: v0.5.2
• Branch: NA
• Commit ID: NA
• Operating system: Indepedent

Additional context None

Tasks

• [x] Backport fix
• [x] Tag fix into release channel
• [x] Merge fix into next

Description Since we plan to support more data structures (and more data models), beyond just keys and values - we need to support arrays in queries and/or responses.

Tasks

• [x] New QueryBuilder impl
• [x] New ResponseBuilder impl
• [x] New query packet parser
• [x] New response packet parser
• [x] New Display impl for array responses
• [x] Deprecate old interfaces and start using new interface
• [x] Test

Edit: The new protocol's specification can be found here

Skyhash is the next protocol that Skytable will be using. Refer to the current spec here: https://docs.skytable.io/next/protocol/skyhash. This will close #148

New output:

skysh 0.8.0
Sayan Nandan <[email protected]>
The Skytable Shell (skysh)

Usage: skysh [OPTIONS]

Options:
  -C, --sslcert <CERT>          Sets the PEM certificate to use for SSL connections
  -e, --eval [<EXPRESSION>...]  Run one or more expressions without REPL
  -h, --host <HOST>             Sets the remote host to connect to [default: 127.0.0.1]
  -p, --port <PORT>             Sets the remote port to connect to [default: 2003]
      --help                    Print help information
  -V, --version                 Print version information

old output:

Skytable Shell 0.8.0
Sayan N. <[email protected]>
The Skytable Shell (skysh)

USAGE:
    skysh [OPTIONS]

FLAGS:
        --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -C, --sslcert <cert>          Sets the PEM certificate to use for SSL connections
    -e, --eval <expression>...    Run an expression without REPL
    -h, --host <host>             Sets the remote host to connect to
    -p, --port <port>             Sets the remote port to connect to

✔️ By submitting this pull request, I agree to the CLA at: https://cla.skytable.io/skytable/skytable

After this PR merges, the following Debian packages will be generated additionally on subsequent releases:

• x86_64-linux-gnu
• x86_64-linux-musl
• i686-linux-gnu
• aarch64-linux-gnu

Description If you've been closely following development, you'd have noticed that we have made three iterations of the protocol, with two revisions this month alone. Not only has this led to confusion but it is also preventing us from implementing new features. This issue tracks how far we are from stabilizing Terrapipe.

• [x] Rewrite docs (terrabasedb/docs#2)
• [x] Rewrite the parser
• [x] Remove libtdb deps from server
• [x] Clarify response types (Okay, Nil, Response Codes, ...)
• [x] Implement Writable traits. (We had already discussed this)
• [x] Upgrade the k/v engine (#13)
• ~Write tests across all modules (#16)~ Left to #16
• [x] Create new Query/Response builders
• [x] New Display impl for CLI
• [x] Stabilize

Unsolved problems:

• ~Feasibility of the Except type~ Solved: The except type can get very problematic when handling large number of keys, we will just return the number of keys set for mset
• ~Have a dedicated MSET/MGET action?~ Solved: we now have MGET and MSET actions

This issue is an 'umbrella' issue that is used to track the state of Skytable's native support for Apple Silicon aka the Apple M1 (arm64-v8).

• [x] T2 Support (native support via rustup; see rust-lang/rust#73908)
• [x] Build passes
• [x] OpenSSL builds with static linking
• [x] Test suite passes
• [x] CI Runner Developed/Ported
  • [x] Secure environment (admins only) [refer to #130]
  • [x] Registry cache
  • [x] Artifact cache
  • [x] Invalidate unused dependencies
• [x] Automated release of bundles ~~- [ ] T1 Support (rustc)~~

This issue will be updated regularly to track the current level of support

A multi model fast and resource efficient database written in Rust is a great idea.

The world is moving towards graph databases,I particularly like Terminusdb https://github.com/terminusdb/terminusdb-server specially their data modeling and querying language WOQL https://terminusdb.com/blog/2020/08/07/terminusdb-graph-basics/ but currently Terminusdb is not suitable for web/mobile applications their focus is more on version control and data science.

Is there any plan to support graph queries with a query language like WOQL.

Description Support multiple keys in actions.

Preliminary tasks

• [x] Multiple SET
• [x] Multiple GET
• [x] Multiple EXIST
• [x] Multiple DEL
• [x] Multiple UPDATE
• [x] Upgrade existing interfaces
• [x] Test

Optimization

• [x] Refactor into kvengine
• [x] DB ops should return primitives
• ~Use the ExceptFor type~

Final tasks

• ~Implement Display for ExceptFor~
• [x] Benchmark

Also, from now on, we should be supporting multiple keys in the actions, wherever applicable.

Description With the last commit, we slipped in a bug and our automated testing suite failed to identify it. This is because our code coverage is very poor at this moment. We need to include tests for almost all the components to ensure that such a thing doesn't happen again. We also need to make sure that these tests are non-deterministic with timeouts.

Tasks:

• [x] GET
• [x] SET
• [x] UPDATE
• [x] DEL
• [x] EXISTS
• [x] MGET
• [x] MSET
• [x] MUPDATE

Hi.

After reading the documentation it is not clear for me, how upgrade and downgrade processes should be performed over the database. Could you please describe somewhere in the documentation:

• How to upgrade the database to the newer version (hopefully with no downtime)
• How to downgrade the database to the older version (hopefully with no downtime)
• Maybe some compatibility policies and notes between versions

Thanks in advance!

Description I suggest adding PGO (+ BOLT) support to the project. According to my local tests, I have reached with sky-bench -q1000000 -r20 the following results (local runs on Apple Macbook M1 Pro): Without PGO (default release build):

===========RESULTS===========
SET    140810.697866/sec
UPDATE 146618.948096/sec
GET    137469.544488/sec
=============================

With PGO:

===========RESULTS===========
SET    153383.657977/sec
UPDATE 156045.351506/sec
GET    159065.787814/sec
=============================

Results are a little bit unstable but the PGO version is always more performant than non-PGO. BOLT (LLVM BOLT) also could help here but I didn't test it yet.

Suggested solutions

• Optimize provided to the users binaries with PGO and BOLT (if any)
• Write a note in the project about optimizing Skytable with PGO for gaining even more performance

Hi, i am using skytable as a subprocess on my desktop application (Tauri application).

it looks like this:

use tauri::{
  api::process::{Command, CommandEvent},
  Window,
};

// sidecar doc   https://tauri.app/v1/guides/building/sidecar
let command = Command::new_sidecar("skyd").expect("Cannot create sidercar");

  let (mut rx, mut child) = command
    .args(["--noart", "-h", "127.0.0.1", "-p", "19999"])
    .current_dir(scoped_dir)
    .spawn()
    .expect("Failed to spawn skyd");


  while let Some(event) = rx.recv().await {
     match event {
      CommandEvent::Stderr(msg) => {
       println!("Stderr:{}", &msg);
        window
          .emit("stderr", Some(format!("'{}'", msg)))
          .expect("failed to emit event");
      }
      CommandEvent::Stdout(line) => {
       println!("Stdout:{}", &line);
        window
          .emit("message", Some(format!("'{}'", line)))
          .expect("failed to emit event");
      }
      CommandEvent::Terminated(t) => {
        println!("{t:?}");
      }
      _ => unimplemented!(),
    };
  }

Stderr:[2022-12-02T01:15:28Z INFO  skyd] Skytable v0.7.6 | https://github.com/skytable/skytable
Stderr:[2022-12-02T01:15:28Z INFO  skyd] Using settings from supplied configuration
Stderr:[2022-12-02T01:15:28Z WARN  skyd::config::feedback] CLI warnings:
Stderr:        - Running in `user` mode. Set mode to `prod` in production
Stderr:[2022-12-02T01:15:28Z INFO  skyd::dbnet] Server started on skyhash://127.0.0.1:19999

When skytable started , i got the ouput from CommandEvent::Stderr. isn't should be output to CommandEvent::Stdout?

Due to internal infrastructure changes, we stopped releasing M1 builds since 0.7.4. However, we can resume builds via cross-compilation (since M1 targets can be cross-compiled from x86_64 Mac machines) so we can use the existing x86_64 Mac builders. However, testing on real M1 hardware is still not achieved until we get actual M1 hosts. For now, the task should be pretty simple:

• [ ] Add M1 to the build matrix (releases)
• [ ] Ensure builds actually work

It's almost like we've been talking about Skytable 0.8 forever, but you haven't been seeing any releases (until a small preview release yesterday). The reason behind it lies in the fact that 0.8 is a convention change to how Skytable works, starting at the protocol layer, to the query language, the memory engines and the storage engine. It's almost like the entire stack was redesigned from the ground up, and with these changes I expect Skytable to become more flexible and powerful than ever.

I will go into more details about the convention change in a future post on Skytable's blog. This issue tracks the release of 0.8, with tasks and very precise ETAs. While the alpha.1 has been released, it is about 1% of what has changed across the codebase and the other changes remain on my local copy as I wait for Rust 1.65 to be released so that we can stop depending on nightly (the current tree is on stable, and when I merge my changes they'll also compile on the stable channel).

ETAs

Expected releases and features to expect:

• [x] Skytable 0.8.0-alpha.1 (ETA October 31, 2022):
  • Skyhash 2.0 preview
  • BlueQL 0.8 DDL preview
• [ ] Skytable 0.8.0-beta.1 (ETA ~~November 23, 2022 December 8, 2022~~ TBA):
  • BlueQL 0.8 final spec (DDL, DML)
  • Full data modeling features
  • New internal engines (mem, disk)
• [ ] Skytable 0.8.0 (ETA ~~November 30, 2022~~ TBA):
  • The final release
  • Skyhash 2.0 final spec
  • Advanced query expressions

Tasks

• [ ] Skyhash 2.0 final spec (#255)
• [x] BlueQL final spec (#263)
• [ ] HS-memory engine (#247)
• [ ] New storage engine drivers
  • [ ] TS
  • [ ] LS
• [ ] Make installation more accessible (#293)
• [ ] Include BlueQL inside docs
• [ ] Upgrade client

This is a lot of work, but let's get to it. Not listing what has already been done :)

Description It would be great to be able to run 'cargo install skytable' to install skysh, skydb etc. Also eliminates the need for maintaining packages for linux, macosx and windows.