Skybase is an extremely fast, secure and reliable real-time NoSQL database with automated snapshots and SSL

Overview

Skybase

The next-generation NoSQL database

GitHub Workflow Status GitHub Workflow Status Status: Alpha Development GitHub release (latest SemVer including pre-releases) Docs Contribute Now

What is Skybase?

Skybase (or SkybaseDB/SDB) is an effort to provide the best of key/value stores, document stores and columnar databases, that is, simplicity, flexibility and queryability at scale. The name 'Skybase' exemplifies our vision to create a database that has limitless possibilities. Skybase was previously known as TerrabaseDB.

Skybase is curently in an alpha stage, but can be used as a performant, secure and persistent key-value store.

Getting started πŸš€

  1. Download a bundle for your platform from here ⬇️
  2. Unzip the bundle
  3. Make the files executable (run chmod +x sdb skysh on *nix systems)
  4. First run sdb to start the database server and then run skysh to start the interactive shell
  5. Run commands like: SET foo bar , GET bar , UPDATE cat mitten or DEL proprietary πŸ€ͺ on skysh !

You can learn more about installation here

Features

  • Insanely fast: Scale to millions of queries per second per node
  • Key/value store: GET , SET , UPDATE and all that stuff
  • Snapshots: Automated (and tunable) snapshots for stress-free backups
  • Secure: Secure connections are built into Skybase with SSL/TLS
  • Multithreaded: Designed to exploit all CPU cores
  • Resource friendly: The database server doesn't need more than 1MB to run
  • Convenient: Without the setup hassle and system-specific dependencies

(... and more on the way)

Clients πŸ”Œ

Until we release an official driver, you'll have to write your own clients β€” all you need to do is implement the simple and performant Terrapipe protocol spec.

Community πŸ‘

A project which is powered by the community believes in the power of community! If you get stuck anywhere - here are your options!

Platforms πŸ’»

Linux supported macOS supported Windows supported

Versioning

This project strictly follows semver, however, since this project is currently in the development phase (0.x.y), the API may change unpredictably

Contributing

Contribute Now

Yes - we need you! Be it a typo, a bizarre idea, a dirty bug 🐞 or an amazing patch - you're welcome to contribute to Skybase! Beginner friendly issues are marked with the label. Read the guide here.

Contributors

You can see a full list of contributors here

License

First, know your rights! This project is licensed under the AGPL-3.0 License.

Comments
  • Server: Persistence is broken since tag v0.5.2

    Server: Persistence is broken since tag v0.5.2

    Description of bug Persistence is broken since the last tag. A more concrete reproduction is yet to be produced.

    Steps to reproduce Steps to reproduce the behavior:

    1. Run skyd
    2. Launch skysh and run SET somekey somevalue
    3. Close skysh and then close skyd
    4. Launch skyd
    5. Launch skysh and run GET somekey
    6. You'll get a (Nil) printed on the screen

    Expected behavior "somevalue" should have been printed

    Meta

    • Release tag: v0.5.2
    • Branch: NA
    • Commit ID: NA
    • Operating system: Indepedent

    Additional context None

    Tasks

    • [x] Backport fix
    • [x] Tag fix into release channel
    • [x] Merge fix into next
    C-bug D-server C-storage A-independent C-reliability C-Stability S-investigation-needed P-critical regression-into-release-noprev 
    opened by ohsayan 14
  • Feature: Supporting array queries/responses

    Feature: Supporting array queries/responses

    Description Since we plan to support more data structures (and more data models), beyond just keys and values - we need to support arrays in queries and/or responses.

    Tasks

    • [x] New QueryBuilder impl
    • [x] New ResponseBuilder impl
    • [x] New query packet parser
    • [x] New response packet parser
    • [x] New Display impl for array responses
    • [x] Deprecate old interfaces and start using new interface
    • [x] Test

    Edit: The new protocol's specification can be found here

    C-enhancement required-improvement tracking-issue P-high C-protocol S-superseded 
    opened by ohsayan 11
  • Implement, stabilize and migrate to Skyhash

    Implement, stabilize and migrate to Skyhash

    Skyhash is the next protocol that Skytable will be using. Refer to the current spec here: https://docs.skytable.io/next/protocol/skyhash. This will close #148

    C-enhancement required-improvement D-client D-server D-workflow C-performance P-high C-protocol C-tests L-Hard C-reliability C-actions D-cli C-Model C-Stability D-libsky 
    opened by ohsayan 9
  • Upgrading clap version to 4.x for skysh

    Upgrading clap version to 4.x for skysh

    New output:

    skysh 0.8.0
    Sayan Nandan <[email protected]>
    The Skytable Shell (skysh)
    
    Usage: skysh [OPTIONS]
    
    Options:
      -C, --sslcert <CERT>          Sets the PEM certificate to use for SSL connections
      -e, --eval [<EXPRESSION>...]  Run one or more expressions without REPL
      -h, --host <HOST>             Sets the remote host to connect to [default: 127.0.0.1]
      -p, --port <PORT>             Sets the remote port to connect to [default: 2003]
          --help                    Print help information
      -V, --version                 Print version information
    

    old output:

    Skytable Shell 0.8.0
    Sayan N. <[email protected]>
    The Skytable Shell (skysh)
    
    USAGE:
        skysh [OPTIONS]
    
    FLAGS:
            --help       Prints help information
        -V, --version    Prints version information
    
    OPTIONS:
        -C, --sslcert <cert>          Sets the PEM certificate to use for SSL connections
        -e, --eval <expression>...    Run an expression without REPL
        -h, --host <host>             Sets the remote host to connect to
        -p, --port <port>             Sets the remote port to connect to
    

    βœ”οΈ By submitting this pull request, I agree to the CLA at: https://cla.skytable.io/skytable/skytable

    C-enhancement D-client hacktoberfest-accepted 
    opened by sanjayts 8
  • Automatically package Debian packages on release

    Automatically package Debian packages on release

    After this PR merges, the following Debian packages will be generated additionally on subsequent releases:

    • x86_64-linux-gnu
    • x86_64-linux-musl
    • i686-linux-gnu
    • aarch64-linux-gnu
    D-server D-build D-workflow D-cargo 
    opened by ohsayan 8
  • Tracking issue for stabilization of the Terrapipe Protocol

    Tracking issue for stabilization of the Terrapipe Protocol

    Description If you've been closely following development, you'd have noticed that we have made three iterations of the protocol, with two revisions this month alone. Not only has this led to confusion but it is also preventing us from implementing new features. This issue tracks how far we are from stabilizing Terrapipe.

    • [x] Rewrite docs (terrabasedb/docs#2)
    • [x] Rewrite the parser
    • [x] Remove libtdb deps from server
    • [x] Clarify response types (Okay, Nil, Response Codes, ...)
    • [x] Implement Writable traits. (We had already discussed this)
    • [x] Upgrade the k/v engine (#13)
    • ~Write tests across all modules (#16)~ Left to #16
    • [x] Create new Query/Response builders
    • [x] New Display impl for CLI
    • [x] Stabilize

    Unsolved problems:

    • ~Feasibility of the Except type~ Solved: The except type can get very problematic when handling large number of keys, we will just return the number of keys set for mset
    • ~Have a dedicated MSET/MGET action?~ Solved: we now have MGET and MSET actions
    C-docs required-improvement C-performance tracking-issue P-high C-protocol 
    opened by ohsayan 8
  • Native support for Apple Silicon (arm64-v8)

    Native support for Apple Silicon (arm64-v8)

    This issue is an 'umbrella' issue that is used to track the state of Skytable's native support for Apple Silicon aka the Apple M1 (arm64-v8).

    • [x] T2 Support (native support via rustup; see rust-lang/rust#73908)
    • [x] Build passes
    • [x] OpenSSL builds with static linking
    • [x] Test suite passes
    • [x] CI Runner Developed/Ported
      • [x] Secure environment (admins only) [refer to #130]
      • [x] Registry cache
      • [x] Artifact cache
      • [x] Invalidate unused dependencies
    • [x] Automated release of bundles ~~- [ ] T1 Support (rustc)~~

    This issue will be updated regularly to track the current level of support

    D-build OS-macOS A-arm64 C-reliability C-Stability 
    opened by ohsayan 7
  • Terrabasedb as a graph database

    Terrabasedb as a graph database

    A multi model fast and resource efficient database written in Rust is a great idea.

    The world is moving towards graph databases,I particularly like Terminusdb https://github.com/terminusdb/terminusdb-server specially their data modeling and querying language WOQL https://terminusdb.com/blog/2020/08/07/terminusdb-graph-basics/ but currently Terminusdb is not suitable for web/mobile applications their focus is more on version control and data science.

    Is there any plan to support graph queries with a query language like WOQL.

    C-enhancement C-Model M-Graph G-LT F-disposition roadmap-2021-candidate 
    opened by ansarizafar 7
  • Feature: Support multiple keys in actions

    Feature: Support multiple keys in actions

    Description Support multiple keys in actions.

    Preliminary tasks

    • [x] Multiple SET
    • [x] Multiple GET
    • [x] Multiple EXIST
    • [x] Multiple DEL
    • [x] Multiple UPDATE
    • [x] Upgrade existing interfaces
    • [x] Test

    Optimization

    • [x] Refactor into kvengine
    • [x] DB ops should return primitives
    • ~Use the ExceptFor type~

    Final tasks

    • ~Implement Display for ExceptFor~
    • [x] Benchmark

    Also, from now on, we should be supporting multiple keys in the actions, wherever applicable.

    C-enhancement required-improvement P-high C-actions 
    opened by ohsayan 7
  • Testing: Add tests across all modules

    Testing: Add tests across all modules

    Description With the last commit, we slipped in a bug and our automated testing suite failed to identify it. This is because our code coverage is very poor at this moment. We need to include tests for almost all the components to ensure that such a thing doesn't happen again. We also need to make sure that these tests are non-deterministic with timeouts.

    Tasks:

    • [x] GET
    • [x] SET
    • [x] UPDATE
    • [x] DEL
    • [x] EXISTS
    • [x] MGET
    • [x] MSET
    • [x] MUPDATE
    help wanted D-build D-workflow C-tests L-Easy 
    opened by ohsayan 6
  • Docs: upgrade and downgrade processes

    Docs: upgrade and downgrade processes

    Hi.

    After reading the documentation it is not clear for me, how upgrade and downgrade processes should be performed over the database. Could you please describe somewhere in the documentation:

    • How to upgrade the database to the newer version (hopefully with no downtime)
    • How to downgrade the database to the older version (hopefully with no downtime)
    • Maybe some compatibility policies and notes between versions

    Thanks in advance!

    C-docs 
    opened by zamazan4ik 5
  • Performance: PGO applicability

    Performance: PGO applicability

    Description I suggest adding PGO (+ BOLT) support to the project. According to my local tests, I have reached with sky-bench -q1000000 -r20 the following results (local runs on Apple Macbook M1 Pro): Without PGO (default release build):

    ===========RESULTS===========
    SET    140810.697866/sec
    UPDATE 146618.948096/sec
    GET    137469.544488/sec
    =============================
    

    With PGO:

    ===========RESULTS===========
    SET    153383.657977/sec
    UPDATE 156045.351506/sec
    GET    159065.787814/sec
    =============================
    

    Results are a little bit unstable but the PGO version is always more performant than non-PGO. BOLT (LLVM BOLT) also could help here but I didn't test it yet.

    Suggested solutions

    • Optimize provided to the users binaries with PGO and BOLT (if any)
    • Write a note in the project about optimizing Skytable with PGO for gaining even more performance
    C-performance 
    opened by zamazan4ik 0
  • I'm running into a weird issue with subprocess output

    I'm running into a weird issue with subprocess output

    Hi, i am using skytable as a subprocess on my desktop application (Tauri application).

    it looks like this:

    use tauri::{
      api::process::{Command, CommandEvent},
      Window,
    };
    
    // sidecar doc   https://tauri.app/v1/guides/building/sidecar
    let command = Command::new_sidecar("skyd").expect("Cannot create sidercar");
    
      let (mut rx, mut child) = command
        .args(["--noart", "-h", "127.0.0.1", "-p", "19999"])
        .current_dir(scoped_dir)
        .spawn()
        .expect("Failed to spawn skyd");
    
    
      while let Some(event) = rx.recv().await {
         match event {
          CommandEvent::Stderr(msg) => {
           println!("Stderr:{}", &msg);
            window
              .emit("stderr", Some(format!("'{}'", msg)))
              .expect("failed to emit event");
          }
          CommandEvent::Stdout(line) => {
           println!("Stdout:{}", &line);
            window
              .emit("message", Some(format!("'{}'", line)))
              .expect("failed to emit event");
          }
          CommandEvent::Terminated(t) => {
            println!("{t:?}");
          }
          _ => unimplemented!(),
        };
      }
    
    
    Stderr:[2022-12-02T01:15:28Z INFO  skyd] Skytable v0.7.6 | https://github.com/skytable/skytable
    Stderr:[2022-12-02T01:15:28Z INFO  skyd] Using settings from supplied configuration
    Stderr:[2022-12-02T01:15:28Z WARN  skyd::config::feedback] CLI warnings:
    Stderr:        - Running in `user` mode. Set mode to `prod` in production
    Stderr:[2022-12-02T01:15:28Z INFO  skyd::dbnet] Server started on skyhash://127.0.0.1:19999
    

    When skytable started , i got the ouput from CommandEvent::Stderr. isn't should be output to CommandEvent::Stdout?

    opened by takkuumi 1
  • Infra: Resume M1 builds

    Infra: Resume M1 builds

    Due to internal infrastructure changes, we stopped releasing M1 builds since 0.7.4. However, we can resume builds via cross-compilation (since M1 targets can be cross-compiled from x86_64 Mac machines) so we can use the existing x86_64 Mac builders. However, testing on real M1 hardware is still not achieved until we get actual M1 hosts. For now, the task should be pretty simple:

    • [ ] Add M1 to the build matrix (releases)
    • [ ] Ensure builds actually work
    C-enhancement D-build C-tests L-Easy C-infra 
    opened by ohsayan 1
  • Tracking issue: Skytable 0.8.0

    Tracking issue: Skytable 0.8.0

    It's almost like we've been talking about Skytable 0.8 forever, but you haven't been seeing any releases (until a small preview release yesterday). The reason behind it lies in the fact that 0.8 is a convention change to how Skytable works, starting at the protocol layer, to the query language, the memory engines and the storage engine. It's almost like the entire stack was redesigned from the ground up, and with these changes I expect Skytable to become more flexible and powerful than ever.

    I will go into more details about the convention change in a future post on Skytable's blog. This issue tracks the release of 0.8, with tasks and very precise ETAs. While the alpha.1 has been released, it is about 1% of what has changed across the codebase and the other changes remain on my local copy as I wait for Rust 1.65 to be released so that we can stop depending on nightly (the current tree is on stable, and when I merge my changes they'll also compile on the stable channel).

    ETAs

    Expected releases and features to expect:

    • [x] Skytable 0.8.0-alpha.1 (ETA October 31, 2022):
      • Skyhash 2.0 preview
      • BlueQL 0.8 DDL preview
    • [ ] Skytable 0.8.0-beta.1 (ETA ~~November 23, 2022 December 8, 2022~~ TBA):
      • BlueQL 0.8 final spec (DDL, DML)
      • Full data modeling features
      • New internal engines (mem, disk)
    • [ ] Skytable 0.8.0 (ETA ~~November 30, 2022~~ TBA):
      • The final release
      • Skyhash 2.0 final spec
      • Advanced query expressions

    Tasks

    • [ ] Skyhash 2.0 final spec (#255)
    • [x] BlueQL final spec (#263)
    • [ ] HS-memory engine (#247)
    • [ ] New storage engine drivers
      • [ ] TS
      • [ ] LS
    • [ ] Make installation more accessible (#293)
    • [ ] Include BlueQL inside docs
    • [ ] Upgrade client

    This is a lot of work, but let's get to it. Not listing what has already been done :)

    release-track 
    opened by ohsayan 3
  • Feature: Support 'cargo install' for the Skytable binaries

    Feature: Support 'cargo install' for the Skytable binaries

    Description It would be great to be able to run 'cargo install skytable' to install skysh, skydb etc. Also eliminates the need for maintaining packages for linux, macosx and windows.

    C-enhancement 
    opened by shnads 0
Releases(v0.8.0-alpha.1)
Owner
Skybase
Skybase is a community-led project β€” working to build an extremely fast realtime NoSQL database
Skybase
πŸ“Έ Voting Escrow Snapshots: Historical snapshots of previous voting escrow balances.

snapshots Voting Escrow Snapshots: Historical snapshots of previous voting escrow balances. Motivation There are several instances in which one may wa

Saber 13 Jun 7, 2022
Next-generation developer-first NoSQL database

Next-generation developer-first NoSQL database. AnnaDB moves familiar programming languages' patterns into the databases world to solve the problem of

Roman Right 40 Dec 28, 2022
Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

Stuart 2 Apr 5, 2022
The spatial message broker and database for real-time multiplayer experiences. Official Rust implementation.

WorldQL Server Rust implementation of WorldQL, the spatial message broker and database for real-time multiplayer experiences Setup Instructions ⚠️ Thi

null 214 Jan 2, 2023
A simple library for Firebase real-time database

Firerust A very simple library to implement the Firebase real-time database in your code with the best performance Instalation Add this to your Cargo.

Daniel Dimbarre 1 Apr 15, 2022
Simple document-based NoSQL DBMS from scratch

cudb (a.k.a. cuda++) Simple document-based noSQL DBMS modelled after MongoDB. (Has nothing to do with CUDA, has a lot to do with the Cooper Union and

Jonathan Lam 3 Dec 18, 2021
The rust client for CeresDB. CeresDB is a high-performance, distributed, schema-less, cloud native time-series database that can handle both time-series and analytics workloads.

The rust client for CeresDB. CeresDB is a high-performance, distributed, schema-less, cloud native time-series database that can handle both time-series and analytics workloads.

null 12 Nov 18, 2022
Simple and handy btrfs snapshoting tool. Supports unattended snapshots, tracking, restoring, automatic cleanup and more. Backed with SQLite.

Description Simple and handy btrfs snapshoting tool. Supports unattended snapshots, tracking, restoring, automatic cleanup and more. Backed with SQLit

Eduard Tolosa 27 Nov 22, 2022
Seed your development database with real data ⚑️

Seed Your Development Database With Real Data ⚑️ Replibyte is a blazingly fast tool to seed your databases with your production data while keeping sen

Qovery 3.4k Jan 2, 2023
Materialize simplifies application development with streaming data. Incrementally-updated materialized views - in PostgreSQL and in real time. Materialize is powered by Timely Dataflow.

Materialize is a streaming database for real-time applications. Get started Check out our getting started guide. About Materialize lets you ask questi

Materialize, Inc. 4.7k Jan 8, 2023
A tool for automated migrations for PostgreSQL, SQLite and MySQL.

Models Models is an implementation for a SQL migration management tool. It supports PostgreSQL, MySQL, and SQLite. Quick Start install the CLI by runn

null 45 Nov 16, 2022
A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

Datafuse Labs 5k Jan 9, 2023
An implementation of the tz database for the time-rs Rust crate.

time-tz An implementation of the tz database for the time-rs Rust crate. This implementation is based off of chrono-tz

null 12 Jul 27, 2022
tzdb β€” Time Zone Database

tzdb β€” Time Zone Database Static time zone information for tz-rs. This crate provides all time zones found in the Time Zone Database, currently in the

RenΓ© Kijewski 33 Dec 15, 2022
A high-performance, distributed, schema-less, cloud native time-series database

CeresDB is a high-performance, distributed, schema-less, cloud native time-series database that can handle both time-series and analytics workloads.

null 1.8k Dec 30, 2022
Engula empowers engineers to build reliable and cost-effective databases.

Engula is a storage engine that empowers engineers to build reliable and cost-effective databases with less effort and more confidence. Engula is in t

Engula 706 Jan 1, 2023
Databend aimed to be an open source elastic and reliable serverless data warehouse,

An elastic and reliable Serverless Data Warehouse, offers Blazing Fast Query and combines Elasticity, Simplicity, Low cost of the Cloud, built to make the Data Cloud easy

Datafuse Labs 5k Jan 3, 2023
AgateDB is an embeddable, persistent and fast key-value (KV) database written in pure Rust

AgateDB is an embeddable, persistent and fast key-value (KV) database written in pure Rust. It is designed as an experimental engine for the TiKV project, and will bring aggressive optimizations for TiKV specifically.

TiKV Project 535 Jan 9, 2023
tectonicdb is a fast, highly compressed standalone database and streaming protocol for order book ticks.

tectonicdb crate docs.rs crate.io tectonicdb tdb-core tdb-server-core tdb-cli tectonicdb is a fast, highly compressed standalone database and streamin

Ricky Han 525 Dec 23, 2022