A high-performance, distributed, schema-less, cloud native time-series database

Overview

CeresDB

License OpenIssue ClosedIssue Stars Forks

中文

CeresDB is a high-performance, distributed, schema-less, cloud native time-series database that can handle both time-series and analytics workloads.

Status

The project is currently under rapid development. This early stage is not production ready and may incur data corruptions.

RoadMap

See our RoadMap

Get started

Clone the repository

Clone this repository by git and enter it:

git clone [email protected]:CeresDB/ceresdb.git
cd ceresdb

Run ceresdb in docker

Ensure that docker is installed in your development environment, and then build the image with the provided Dockerfile:

docker build -t ceresdb .

Start the ceresdb container using the built docker image:

docker run -d -t --name ceresdb -p 5440:5440 -p 8831:8831 ceresdb

Compile and run CeresDB

Install dependencies

In order to compile CeresDB, some relevant dependencies(including Rust toolchain) should be installed.

Dependencies(Ubuntu20.04)

Assuming the development environment is Ubuntu20.04, execute the following command to install the required dependencies:

apt install git curl gcc g++ libssl-dev pkg-config cmake

It should be noted that the compilation of the project actually has version requirements for some dependencies such as cmake, gcc, g++, etc. If your development environment is an old Linux distribution, it is necessary to manually install these dependencies of a higher version.

Rust

Rust can be installed by rustup. After installing rustup, when entering the CeresDB project, the specified Rust version will be automatically downloaded according to the rust-toolchain file.

After execution, you need to add environment variables to use the Rust toolchain. Basically, just put the following commands into your ~/.bashrc or ~/.bash_profile:

source $HOME/.cargo/env

Compile and run

Compile CeresDB by the following command:

cargo build --release

Then you can run CeresDB using the default configuration file provided in the codebase.

./target/ceresdb-server --config ./docs/example.toml

Write and read data

CeresDB supports custom extended SQL protocol. Currently, you can create tables and read/write data with SQL statements through http service.

Create table

curl --location --request POST 'http://127.0.0.1:5440/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "CREATE TABLE `demo` (`name` string TAG, `value` double NOT NULL, `t` timestamp NOT NULL, TIMESTAMP KEY(t)) ENGINE=Analytic with (enable_ttl='\''false'\'');"
}'

Write data

curl --location --request POST 'http://127.0.0.1:5440/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "INSERT INTO demo(t, name, value) VALUES(1651737067000, '\''ceresdb'\'', 100)"
}'

Read data

curl --location --request POST 'http://127.0.0.1:5440/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "select * from demo"
}'

Show create table

curl --location --request POST 'http://127.0.0.1:5440/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "show create table demo"
}'

Drop table

curl --location --request POST 'http://127.0.0.1:5440/sql' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "DROP TABLE demo;"
}'

Contributing

Any contribution is welcome!

Read our Contributing Guide and make your first contribution!

Architecture and Technical Documentation

Our technical documents(still under writing and polishing) describes critical parts of ceresdb in the docs.

Acknowledgment

Some design of CeresDB references influxdb_iox, and some specific module implementations reference tikv and other excellent open source projects, thanks to InfluxDB, TiKV, and any other referenced great open source projects.

Licensing

CeresDB is under Apache License 2.0.

Community

Comments
  • feat: migrate current harness tests to sqlness

    feat: migrate current harness tests to sqlness

    Which issue does this PR close?

    Closes #397

    Rationale for this change

    migrate the harness tests to the sqlness.

    • add a new dependency: sqlness = { git = "https://github.com/CeresDB/sqlness.git", rev = "c077b17d73ab25460c152dc34e8f80f904522a57"}
    • update tokio version '1.0' to '1.21'

    What changes are included in this PR?

    the tests dir. I create a new module use sqlness.

    Are there any user-facing changes?

    none

    How does this change test

    opened by dust1 14
  • Can Ceresdb support default value for columns?

    Can Ceresdb support default value for columns?

    Describe This Problem

    Can Ceresdb support default value for columns something like timestamp = now()

    Proposal

    • In Oceanbase, I can record data insert time, like gmt_create = now().

    Additional Context

    No response

    enhancement help wanted 
    opened by MachaelLee 13
  • Migrate current harness tests to sqlness

    Migrate current harness tests to sqlness

    Describe This Problem

    sqlness has been created as a separated repo, but ceresdb doesn't use it now.

    Proposal

    Let's migrate the harness tests to the sqlness.

    Additional Context

    No response

    enhancement good first issue help wanted A-test 
    opened by ShiKaiWi 11
  • build(image): :hammer: replace entrypoint with bash, drop supervisor

    build(image): :hammer: replace entrypoint with bash, drop supervisor

    Which issue does this PR close?

    Closes https://github.com/CeresDB/ceresdb/issues/145

    Rationale for this change

    What changes are included in this PR?

    1. replace entrypoint.py with entrypoint.sh
    2. use user ceres instead of admin in docker container
    3. drop supervisor

    Are there any user-facing changes?

    use user ceres instead of admin in docker container

    How does this change test

    I have successfully created a new image, but not yet had a chance to test it due to M1 chip, will test it tomorrow when I reach my intel one

    enhancement 
    opened by zwpaper 11
  • feat: support show database and show tables

    feat: support show database and show tables

    Which issue does this PR close?

    Closes #66

    Rationale for this change

    What changes are included in this PR?

    In order to satisfy the current three Show statements: show create table, show tables and show database, I modified the Plan part where show create table is located and combined it into a ShowPlan.

    Are there any user-facing changes?

    For users, two new sql statements are added to find schema and table in ceresDB.

    Effect

    SHOW TABLES

    First, use show tables to view existing tables: show tables; image Next, use create table sql to create a new table:

    CREATE TABLE `hello_show` (`name` string TAG, `value` double NOT NULL, `t` timestamp NOT NULL, TIMESTAMP KEY(t)) ENGINE=Analytic with (enable_ttl='false');
    

    image Last view effect show tables; image

    SHOW DATABASE

    exec show database; image

    How does this change test

    add 2 unit test:

    1. test_show_tables_statement_to_plan
    2. test_show_database_statement_to_plan
    opened by dust1 9
  • feat: add mysql server support

    feat: add mysql server support

    Which issue does this PR close?

    Closes #43

    Rationale for this change

    Convenient for mysql client to use

    What changes are included in this PR?

    • [x] add mysql server structure
    • [x] implement sql execute

    Are there any user-facing changes?

    there are user-facing changes that needs to update the documentation or configuration.

    a new configuration items have been added to the server::config.rs file:

    1. mysql_port: the mysql server listening port, default 3307

    How does this change test

    todo

    Note

    what I tried at first was opensrv, but I am not very proficient in tokio and related concurrent operations. since ceresDB needs to pass in Instance<C, Q> when executing sql, when I try to use AsyncMysqlIntermediary to run the service, I am troubled by the asynchrony of these two generic types. It requires me to declare the specific types of C and Q, such as: impl<C: Catalog, Q: Query>, but in this case, the server.rs needs to be changed to this:

    pub struct Server<C: Catalog, Q: Query> {
        http_service: Service<C, Q>,
        rpc_services: RpcServices,
        mysql_service: mysql::MysqlHandler<C, Q>,
    }
    

    I think this modification is not very good, so I finally use msql-srv

    How to use it?

    First, we can start ceresdb as usual. We can see that there is an extra entry in the startup log:

    2022-06-22 21:16:10.980 INFO [server/src/mysql/handler.rs:54] Mysql service listens on 127.0.0.1:3307
    

    Mysql service starts on port 3307 by default, you can customize it by modifying the ·mysql_port· option in the configuration file

    Next, you should install a mysql-client and connect to mysql by typing:

    mysql -P3307 -h127.0.0.1
    

    We will enter the mysql console. image

    Finally, Use ceresdb like mysql.

    Example

    Create table

    CREATE TABLE `hello_mysql` (`name` string TAG, `value` double NOT NULL, `t` timestamp NOT NULL, TIMESTAMP KEY(t)) ENGINE=Analytic with (enable_ttl='false');
    

    image

    Write data

    INSERT INTO hello_mysql(t, name, value) VALUES(1651737067000, 'ceresdb', 100);
    

    image

    Read data

    select * from hello_mysql;
    

    image

    Show create table

    show create table hello_mysql;
    

    image

    Drop table

    DROP TABLE hello_mysql;
    

    image

    enhancement 
    opened by dust1 9
  • Query fields are sorted by creation

    Query fields are sorted by creation

    Description ceresDB doesn't seem to require strict ordering of query result fields.

    Proposal

    Use the order in which fields are created.

    Additional context

    when I create table use:

    CREATE TABLE `hello_mysql` (`name` string TAG, `value` double NOT NULL, `t` timestamp NOT NULL, TIMESTAMP KEY(t)) ENGINE=Analytic with (enable_ttl='false');
    

    and select it. his field order seems to be random? I found that the query result is wrapped in HashMap. image

    This doesn't seem to have any effect on normal usage, so I don't know if this situation needs improvement.

    enhancement 
    opened by dust1 8
  • Integration tests are not stable

    Integration tests are not stable

    Describe this problem

    For create table test,

    https://github.com/CeresDB/ceresdb/blob/25557c73adaa6815875aaea3ccd5b61bf54e291f/tests/cases/local/05_ddl/create_tables.sql#L64

    it will report error like:

    Failed to execute query, err: Server(ServerError { code: 500, msg: "Failed to execute interpreter, query: CREATE TABLE `05_create_tables_t8`(c1 int, t1 timestamp NOT NULL TIMESTAMP KEY) ENGINE = Analytic with (storage_format= 'unknown');. Caused by: Failed to execute create table, err:Failed to create table, name:05_create_tables_t8, err:Failed to create table, err:Invalid arguments, err:Invalid options, space_id:2, table:05_create_tables_t8, table_id:2199023255725, err:Unknown storage format. value:\"unknown\"." })
    

    The table_id in error message is not stable in my local pc, it will change if I run tests again.

    The issue maybe is in test config, we use Local storage as the server config for integration tests now.

    cc: @waynexia @jiacai2050

    Steps to reproduce

    Run tests again.

    Expected behavior

    The output should the same as before when we rerun tests.

    Additional Information

    No response

    bug help wanted 
    opened by ygf11 7
  • Column order is not what user define when create table

    Column order is not what user define when create table

    Describe this problem

    Table' columns order is not what user define when create table

    Steps to reproduce

    Execute those SQL

    CREATE TABLE test_order (
        host string tag,
        v2 int,
        ts timestamp NOT NULL,
        v1 int,
        timestamp KEY (ts)
    )ENGINE = Analytic WITH (
        enable_ttl = 'false'
    );
    
    desc test_order;
    

    desc will output

    ts	timestamp	1	0	0
    tsid	uint64	1	0	0
    host	string	0	1	1
    v1	int	0	1	1
    v2	int	0	1	0
    

    Expected behavior

    The column order should be same with the order when created.

    Additional Information

    https://github.com/CeresDB/ceresdb/blob/3e826ffbb98d2c5351e63b28dd9dd4644e7d4c1e/sql/src/planner.rs#L265

    The root cause is currently we use a BTreeMap when build create table plan, it should use a vector to keep the ordering

    ref #154

    bug A-SQL 
    opened by jiacai2050 7
  • Release v0.3

    Release v0.3

    Description

    We prepare to release v0.3 at the end of Aug. Here is the feature list:

    • Release multi-language client. Include Java, Rust and Python.
    • Support static cluster mode. And keep pushing toward a full-featured dynamic distributed version (related project: Distributed CeresDB).
    • Extend supported SQLs (tag: A-SQL).
    • Implement the hybrid storage format. And support reading from two formats.

    Feel free to suggest or discuss other features you would like to add :heart:

    tracking issue 
    opened by waynexia 7
  • Simplify SQL syntax that specifies timestamp key column

    Simplify SQL syntax that specifies timestamp key column

    Description

    TIMESTAMP KEY is a special column in analytic engine. It need to be set to a column with Timestamp type and with NOT NULL constrain.

    CREATE TABLE demo (
        name string TAG,
        value double NOT NULL,
        t timestamp NOT NULL,
        TIMESTAMP KEY(t)
    ) ENGINE=Analytic;
    

    It must have one and only one TIMESTAMP KEY column in a table.

    Proposal there are a few ways to simplify it.


    • Specify TIMESTAMP KEY without create a column Considering the constraints, user is actually giving a name to the TIMESTAMP KEY column.
    CREATE TABLE demo (
        name string TAG,
        value double NOT NULL,
        TIMESTAMP KEY(t)
    ) ENGINE=Analytic;
    

    • Move TIMESTAMP KEY to column definition. This makes TIMESTAMP KEY act like primary key.
    CREATE TABLE demo (
        name string TAG,
        value double NOT NULL,
        t timestamp NOT NULL TIMESTAMP KEY
    ) ENGINE=Analytic;
    

    • Do not need to define it and let CeresDB infers the TIMESTAMP KEY Allow user to not specify TIMESTAMP KEY in create table SQL. CeresDB will try to find a column with Timestamp type and NOT NULL. Report error if zero or multiple columns are found.
    CREATE TABLE demo (
        name string TAG,
        value double NOT NULL,
        t timestamp NOT NULL
    ) ENGINE=Analytic;
    

    Additional context

    enhancement A-SQL A-analytic-engine 
    opened by waynexia 7
  • feat: disable do_snapshot when recover table data

    feat: disable do_snapshot when recover table data

    Which issue does this PR close?

    Closes #

    Rationale for this change

    Since the read-only partition table and table(follower role) need to be opened in multi servers, the server cannot do snapshot of the manifest when the table is opened.

    What changes are included in this PR?

    • Disable do_snapshot when recover table data.

    Are there any user-facing changes?

    No.

    How does this change test

    No need.

    opened by chunshao90 0
  • fix: Revert

    fix: Revert "feat: support bloom filter in hybrid format (#479)"

    Which issue does this PR close?

    Closes #

    Rationale for this change

    After #479, sst file size is larger than before, so revert this commit

    $ du -sh data-with-flush/store/2 data-without-flush/store/2
    371M    data-with-flush/store/2
    328M    data-without-flush/store/2
    

    What changes are included in this PR?

    Are there any user-facing changes?

    No

    How does this change test

    Existing UT

    opened by jiacai2050 0
  • feat: add cached router for remote engine client

    feat: add cached router for remote engine client

    Which issue does this PR close?

    Closes # Part of #492

    Rationale for this change

    Now, we do route work in each request, that make performance too bad. So, this pr add a cached router in remote engine client.

    What changes are included in this PR?

    Add cached router for remote engine client.

    Are there any user-facing changes?

    None.

    How does this change test

    Test manually.

    opened by Rachelint 0
  • support schema header for http/grpc service

    support schema header for http/grpc service

    Which issue does this PR close?

    Closes #

    Rationale for this change

    Currently, it is unreasonable to use tenant as the schema when accessing grpc/http service.

    What changes are included in this PR?

    • Add new header x-ceresdb-schema;
    • For compatibility, tenant is still used as schema if schema is not set.

    Are there any user-facing changes?

    User now can use x-ceresdb-schema header to set schema instead of x-ceresdb-access-tenant.

    How does this change test

    Existing tests.

    opened by ShiKaiWi 0
  • feat: add open_table_on_shard and close_table_on_shard in meta_event_service

    feat: add open_table_on_shard and close_table_on_shard in meta_event_service

    Which issue does this PR close?

    Closes #

    Rationale for this change

    Support closing and opening table.

    What changes are included in this PR?

    • Implement open_table_on_shard in meta_event_service.
    • Implement close_table_on_shard in meta_event_service.

    Are there any user-facing changes?

    No.

    How does this change test

    No need.

    opened by chunshao90 0
  • feat: add permission check for sub tables in table partition

    feat: add permission check for sub tables in table partition

    Which issue does this PR close?

    Closes #

    Rationale for this change

    Common user shouldn't have right to operate the sub tables in table partition directly, and only admin can do this. So this pr add a permission check for reaching this target.

    What changes are included in this PR?

    Add permission check in interpreter creating stage.

    Are there any user-facing changes?

    None.

    How does this change test

    Test by ut.

    opened by Rachelint 0
Releases(v1.0.0-alpha02)
  • v1.0.0-alpha02(Dec 30, 2022)

    Major features

    • Table Partition
      • Support key partition(MySQL-like syntax).
      • Implement PartitionTable to support query and write for partitioned tables.
    • Improve query performance
      • Add disk based object store cache. (unstable)
      • Implement parallel get_byte_ranges for ObjectStoreReader.
      • Scan row groups in one sst file parallelly.
      • Support bloom filter in hybrid format.
      • Support MergeIterator to pull data concurrently.
    • Support auto query forwarding for grpc.
    • Normalize the case of SQL and make clear that all SQL cases are sensitive.
    • Chore
      • Migrate current harness tests to sqlness. by @dust1
      • Support memory usage limit on background compaction.
      • Make the bloom filter optional in sst meta.
    • Bug fix
      • Fix wrong primary key when define tsid and timestamp key as primary key.
      • Fix wrong path in the result from StoreWithPrefix.
      • Fix lru-weighted-cache` memory leak.
      • Fix some bugs in background compaction.
      • Fix wrong profile output os path for heap profiling.

    What's Changed

    • feat: convert table name to lowercase when not quoted by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/444
    • fix: object_store config by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/443
    • feat: add disk based object store cache by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/420
    • feat: add dynamic setting log level by @ZuLiangWang in https://github.com/CeresDB/ceresdb/pull/445
    • feat: add disk cache value crc by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/447
    • refactor: make http max body size configurable and modify default value by @Rachelint in https://github.com/CeresDB/ceresdb/pull/451
    • chore: use default value for StorageOptions by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/455
    • chore: modify some config in workflows by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/457
    • fix: wrong primary key when define tsid and timestamp key as primary key by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/453
    • feat: impl parallel get_byte_ranges for ObjectStoreReader by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/450
    • feat: convert table name to quoted style by @Rachelint in https://github.com/CeresDB/ceresdb/pull/454
    • refactor: remove enable_tsid_primary_key by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/459
    • chore: bump oss sdk by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/458
    • feat: make rocksdb wal compatible with other distributed implementation by @Rachelint in https://github.com/CeresDB/ceresdb/pull/460
    • fix: kill ceresdb-server if run harness failed by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/461
    • feat: support prefix for object store by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/463
    • fix: wrong path in the result from StoreWithPrefix by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/465
    • feat: make initializing of buffered streams in MergeIterator concurrent by @Rachelint in https://github.com/CeresDB/ceresdb/pull/466
    • chore: add metrics oss by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/478
    • chore: bump oss sdk by @ZuLiangWang in https://github.com/CeresDB/ceresdb/pull/475
    • feat: support memory usage limit on background compaction by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/476
    • fix: lru-weighted-cache mem leak by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/480
    • chore: support Cargo.toml format check by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/481
    • feat: support bloom filter in hybrid format by @Rachelint in https://github.com/CeresDB/ceresdb/pull/479
    • fix: compact table in background scheduler by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/482
    • fix: wrong profile output os path for heap profiling by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/484
    • feat: add max_input_sstable_size by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/483
    • feat: scan row groups in one sst file parallelly by @Rachelint in https://github.com/CeresDB/ceresdb/pull/474
    • feat: migrate current harness tests to sqlness by @dust1 in https://github.com/CeresDB/ceresdb/pull/473
    • feat: make the bloom filter optional in sst meta by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/488
    • feat: support store with readonly cache by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/490
    • feat: support ignore bloomfilter when compaction by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/491
    • fix: remove EncodingWriter by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/493
    • feat: support create partition table by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/485
    • feat: support parsing partition table creating by @Rachelint in https://github.com/CeresDB/ceresdb/pull/487
    • feat: support query limit by rule by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/494
    • chore: adjust usage of sst type by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/499
    • feat: introduce ObjectStorePicker to replace the two object stores by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/496
    • feat: define remote table engine trait by @Rachelint in https://github.com/CeresDB/ceresdb/pull/502
    • feat: define partition rule trait by @Rachelint in https://github.com/CeresDB/ceresdb/pull/501
    • refactor: separate object store from parquet sst async reader by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/503
    • refactor: recovery in standalone mode by @Rachelint in https://github.com/CeresDB/ceresdb/pull/414
    • feat: support parse key partition by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/506
    • chore: define remote_engine grpc service by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/505
    • feat: impl key partition rule by @Rachelint in https://github.com/CeresDB/ceresdb/pull/507
    • feat: auto forward query for grpc by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/511
    • feat: impl remote_engine grpc service by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/508
    • feat: Impl remote sdk by @Rachelint in https://github.com/CeresDB/ceresdb/pull/509
    • feat: upgrade sqlness to latest version by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/517
    • feat: update ceresdb proto & adapter to partition info by @ZuLiangWang in https://github.com/CeresDB/ceresdb/pull/515
    • fix: fix conversion by avro by @Rachelint in https://github.com/CeresDB/ceresdb/pull/519
    • fix: fix convert record batch by @ZuLiangWang in https://github.com/CeresDB/ceresdb/pull/521
    • feat: impl PartitionTable by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/512
    • fix: fix partition table affected_rows by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/522
    • chore: add more context for query log by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/523
    • chore: bump sqlness to 0.1.1 by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/524
    • fix: fix integration test's bad case by @Rachelint in https://github.com/CeresDB/ceresdb/pull/527

    Full Changelog: https://github.com/CeresDB/ceresdb/compare/v1.0.0-alpha01...v1.0.0-alpha02

    Source code(tar.gz)
    Source code(zip)
  • v1.0.0-alpha01(Dec 1, 2022)

    Major features

    • Improve query performance
      • Utilize the bloom filter stored in the sst meta data to do filtering on the row groups according to the provided predicate
      • Refactor sst cache based on weighted LRU.
      • Add a new async parquet reader.
      • Support get_range api in aliyun oss to avoid OOM when sst is large.
      • Add memory based cache for object store.
    • Improve CeresDB cluster mode
      • Implement distributed wal based on kafka.(unstable)
      • Implement expansion and transferLeader control with ceresmeta.(unstable)
      • Refactor cluster metadata management in ceresemta.
    • Bug fix
      • Fix some bugs in WAL on OBKV.
      • Fix encoding bug in hybrid storage format.
      • Correct the order of the columns to be consistent with the order in which the user created the table. by @dust1
    • SDK

    What's Changed

    • chore: fix error message when table is not found by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/348
    • docs: Table operation doc by @MachaelLee in https://github.com/CeresDB/ceresdb/pull/318
    • fix: adopt new parquet api by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/351
    • refactor: utilize datafuison to support filter before merge procedure by @ygf11 in https://github.com/CeresDB/ceresdb/pull/326
    • Refactor replace protobuf with prost by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/352
    • feat: region meta in wal manager on kafka by @Rachelint in https://github.com/CeresDB/ceresdb/pull/345
    • docs: proposal of roadmap by @archerny in https://github.com/CeresDB/ceresdb/pull/356
    • feat: add async parquet reader by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/355
    • feat: encoding in wal's message queue impl. by @Rachelint in https://github.com/CeresDB/ceresdb/pull/358
    • feat: add tools sst-convert by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/361
    • fix: apply missing filtering on parquet reader by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/360
    • docs: update dynamic cluster mode doc by @ZuLiangWang in https://github.com/CeresDB/ceresdb/pull/357
    • feat: replace ParquetExec with ParquetRecordBatchStream by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/368
    • feat: region in wal's message queue impl. by @Rachelint in https://github.com/CeresDB/ceresdb/pull/362
    • feat: support equal pruner by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/372
    • feat: add block list by @MachaelLee in https://github.com/CeresDB/ceresdb/pull/359
    • feat: enhancement to mq's kafka implementation by @Rachelint in https://github.com/CeresDB/ceresdb/pull/375
    • fix: add ignore to test on message queue's kafka implementation. by @Rachelint in https://github.com/CeresDB/ceresdb/pull/377
    • fix:Disallow defining timestamp key column as Tag column by @QuintinTao in https://github.com/CeresDB/ceresdb/pull/374
    • feat: add table id to LogEntry by @Rachelint in https://github.com/CeresDB/ceresdb/pull/379
    • feat: add bloom filter when write sst by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/370
    • feat: support like syntax in show tables statement by @QuintinTao in https://github.com/CeresDB/ceresdb/pull/331
    • feat: support filter_row_groups by EqPruner by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/380
    • refactor: build row groups from record batch stream by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/381
    • refactor: show tables by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/382
    • refactor: wal unit tests. by @Rachelint in https://github.com/CeresDB/ceresdb/pull/387
    • feat: filter row groups by bloom filter by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/383
    • feat: implement region recovery by @Rachelint in https://github.com/CeresDB/ceresdb/pull/385
    • feat: read all bytes out before do async bench by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/390
    • feat: meta cache for both custom and parquet metadata by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/392
    • feat: sst-convert support convert to hybrid format by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/398
    • feat: support get_range in aliyun oss by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/395
    • refactor: support multiple writing in a specific table by @Rachelint in https://github.com/CeresDB/ceresdb/pull/393
    • fix: add meta client timeout by @ZuLiangWang in https://github.com/CeresDB/ceresdb/pull/399
    • chore: update obkv-client to latest version by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/401
    • fix: wrong cargo lock by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/405
    • feat: correct column order by @dust1 in https://github.com/CeresDB/ceresdb/pull/340
    • chore: update the ceresdbproto to latest by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/406
    • chore: remove manual installation of protoc by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/407
    • feat: refacor existing wal implementations to support scanning logs of shard by @Rachelint in https://github.com/CeresDB/ceresdb/pull/400
    • chore: bump client in test harness by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/409
    • docs: update http api by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/410
    • test: add case sensitive demo by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/411
    • feat: add memory based cache for object store by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/415
    • feat: Impl wal on mq namespace by @Rachelint in https://github.com/CeresDB/ceresdb/pull/404
    • feat: Impl wal on mq setup by @Rachelint in https://github.com/CeresDB/ceresdb/pull/416
    • fix: cleaner bugs of wal on obkv by @Rachelint in https://github.com/CeresDB/ceresdb/pull/418
    • feat: replay from flushed sequence by @Rachelint in https://github.com/CeresDB/ceresdb/pull/423
    • feat: make manifest operations more configurable by @Rachelint in https://github.com/CeresDB/ceresdb/pull/424
    • feat: setup MemCacheStore by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/421
    • docs: fix broken conventional commit document link(#429) by @Huachao in https://github.com/CeresDB/ceresdb/pull/430
    • fix: fix grpc service metrics by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/431
    • fix: purging buckets in initializing bug by @Rachelint in https://github.com/CeresDB/ceresdb/pull/432
    • fix: load last sequence in obkv wal by @Rachelint in https://github.com/CeresDB/ceresdb/pull/434
    • fix: drop bug of wal on mq, can't not drop runtime by @Rachelint in https://github.com/CeresDB/ceresdb/pull/433
    • fix: hybrid storage encode bug in multi record batch by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/426
    • chore: add logs about replaying by @Rachelint in https://github.com/CeresDB/ceresdb/pull/438
    • chore: return schema not found when schema not exists by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/439
    • refactor: manifest wal config by @Rachelint in https://github.com/CeresDB/ceresdb/pull/437
    • fix: bug in distributed wal(obkv, kafka) by @Rachelint in https://github.com/CeresDB/ceresdb/pull/422
    • chore: bump version to 1.0.0-alpha01 by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/440

    New Contributors

    • @MachaelLee made their first contribution in https://github.com/CeresDB/ceresdb/pull/318
    • @ZuLiangWang made their first contribution in https://github.com/CeresDB/ceresdb/pull/357
    • @Huachao made their first contribution in https://github.com/CeresDB/ceresdb/pull/430

    Full Changelog: https://github.com/CeresDB/ceresdb/compare/v0.4.0...v1.0.0-alhpa01

    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Oct 25, 2022)

    Major features

    • Improve CeresDB cluster mode
      • Refactor the Catalog and Cluster module for better interaction with CeresMeta
      • Refactor the create/drop table procedure
      • Support open/close shards on the ceresdb-server controlled by CeresMeta
    • Support default value option of column by @ygf11
    • Replace grpc-io with tonic
    • Upgrade to datafusion 12 and arrrow 23
    • Improve the CI
      • Support nightly benchmark with TSBS
      • Reduce the size of CeresDB binary compiled with release profile
      • Fix the random failure in the integration tests
    • Bug fix
      • Avoid extra flushes when triggering one flush on table by @dust1

    What's Changed

    • refactor: remove SchemaIdAlloc and TableIdAlloc by @waynexia in https://github.com/CeresDB/ceresdb/pull/238
    • chore: remove stale docs by @waynexia in https://github.com/CeresDB/ceresdb/pull/239
    • chore: correct integration test's dir by @waynexia in https://github.com/CeresDB/ceresdb/pull/242
    • fix:fix license check by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/243
    • fix: flush of one table might be triggered multiple times by @dust1 in https://github.com/CeresDB/ceresdb/pull/236
    • chore: fix submodule init by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/244
    • feat: add scan batch size options by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/248
    • feat: primary key definition support specify tsid column by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/254
    • refactor: use TableManager to implement CatalogManager and SchemaManager by @waynexia in https://github.com/CeresDB/ceresdb/pull/260
    • feat: collapsible columns supports variable length type by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/241
    • feat: base implementation of default value by @ygf11 in https://github.com/CeresDB/ceresdb/pull/246
    • feat: add tsbs CI by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/264
    • fix: null buffer assert by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/262
    • feat: Implement some cluster commands by @waynexia in https://github.com/CeresDB/ceresdb/pull/265
    • feat: persist collapsible info in parquet meta_data by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/261
    • chore: update parquet-testing by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/266
    • test: make harness test stable by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/270
    • chore: Upgrade to datafusion 12 & arrow 23 by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/269
    • feat: replace grpcio with tonic by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/267
    • chore: make use of workspace dependencies by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/274
    • build: fix test_panic_hook UT by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/275
    • docs: update example config for static deployment by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/276
    • test: add test case upgraded arrow&datafusion by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/272
    • docs: update complie and run chapter by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/281
    • feat: create schema in static routing by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/285
    • chore: adapt to latest ceresdbproto by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/286
    • refactor: remove event handling from cluster by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/289
    • feat: enhance sql handle, accept both json and text request by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/287
    • feat: support flush tables periodically by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/290
    • chore: fix CI cache key, release mode should not mix with debug by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/292
    • feat: support meta event service by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/293
    • feat: bump object store to 0.5.1, same with datafusion by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/296
    • feat: support default value option of column like mysql by @ygf11 in https://github.com/CeresDB/ceresdb/pull/278
    • feat: remove cluster catalog and refactor volatile catalog by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/295
    • refactor: modify exist wal by @Rachelint in https://github.com/CeresDB/ceresdb/pull/288
    • chore: rename ShardTables to TablesOfShard by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/300
    • ci: support check disk quota by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/301
    • feat: add TabelCreator/TableDroper for executing create/drop table by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/303
    • fix: avoid running blocked-on task when building meta client by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/305
    • feat: support open&close shards by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/307
    • refactor: centralize the logic of choosing worker by @QuintinTao in https://github.com/CeresDB/ceresdb/pull/311
    • chore: support set log file when running test harness by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/314
    • test: set correct exit code for test harness by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/319
    • feat: do create drop table by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/310
    • chore: minimize release binary size by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/322
    • feat: impl create drop table service by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/323
    • build: set correct version and let member packages inherit it by @archerny in https://github.com/CeresDB/ceresdb/pull/325
    • chore: keep SQL original ident by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/328
    • feat: add async inner to BatchLogIteratorAdapter by @Rachelint in https://github.com/CeresDB/ceresdb/pull/320
    • chore: add quick release profile by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/330
    • feat: separate status code between storage service and meta event service by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/332
    • fix: use right table manipulator when setup by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/333
    • feat: message queue and its kafka impl by @Rachelint in https://github.com/CeresDB/ceresdb/pull/306
    • refactor: make mq interface simpler. by @Rachelint in https://github.com/CeresDB/ceresdb/pull/337
    • fix: make open shard idempotent by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/338
    • fix: add missing version for meta request by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/339

    New Contributors

    • @QuintinTao made their first contribution in https://github.com/CeresDB/ceresdb/pull/311

    Full Changelog: https://github.com/CeresDB/ceresdb/compare/v0.3.1...v0.4.0

    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Aug 29, 2022)

    Major features

    • Implement hybrid format. Check document below to see how to use
      • https://docs.ceresdb.io/analytic_engine/options.html
    • Implement distributed cluster

    What's Changed

    • build(ci): :hammer: not run the build and push CI on forks by @zwpaper in https://github.com/CeresDB/ceresdb/pull/148
    • build(ci): Test docker image build by @zwpaper in https://github.com/CeresDB/ceresdb/pull/150
    • build(image): :hammer: replace entrypoint with bash, drop supervisor by @zwpaper in https://github.com/CeresDB/ceresdb/pull/152
    • chore: Add build guidance for mac users by @tianlinzx in https://github.com/CeresDB/ceresdb/pull/169
    • chore: add docker hub description workflow by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/130
    • chore: add function tests by @Rachelint in https://github.com/CeresDB/ceresdb/pull/219
    • chore: bump rust toolchain edition to 2021 by @waynexia in https://github.com/CeresDB/ceresdb/pull/180
    • chore: bump sqlparser to 0.19 by @Rachelint in https://github.com/CeresDB/ceresdb/pull/172
    • chore: change GH issue template to yaml format for better experience by @waynexia in https://github.com/CeresDB/ceresdb/pull/181
    • chore: check Cargo.lock in CI by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/223
    • chore: fix some grammar errors in docs by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/200
    • chore: remove components/tracing and components/tracing-example by @waynexia in https://github.com/CeresDB/ceresdb/pull/184
    • chore: switch datafusion fork by @waynexia in https://github.com/CeresDB/ceresdb/pull/203
    • chore: update slog-global dep to https://github.com/tikv/slog-global by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/195
    • chore: update the async_trait used by wal to 0.1.53 by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/199
    • docs: :hammer: fix slack dead link by @zwpaper in https://github.com/CeresDB/ceresdb/pull/141
    • docs: Minor Grammar and Concision Changes in README.md by @jakemcf22 in https://github.com/CeresDB/ceresdb/pull/142
    • docs: Update user guide summary by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/149
    • docs: add data model in user guide by @chunshao90 in https://github.com/CeresDB/ceresdb/pull/132
    • docs: add integration test section to CONTRIBUTING.md by @waynexia in https://github.com/CeresDB/ceresdb/pull/160
    • docs: add quick start by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/133
    • docs: add sql chapter for user guide by @waynexia in https://github.com/CeresDB/ceresdb/pull/136
    • docs: add user guide workflow to generate html pages by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/137
    • docs: fix build dir by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/197
    • docs: improve options section by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/222
    • docs: move architecture to user-guide by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/186
    • docs: update Roadmap.md by @waynexia in https://github.com/CeresDB/ceresdb/pull/176
    • docs: user guide about static routing by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/134
    • feat: Simplify SQL syntax that specifies timestamp key column by @dust1 in https://github.com/CeresDB/ceresdb/pull/140
    • feat: RoleTable trait and LeaderTable implementation by @waynexia in https://github.com/CeresDB/ceresdb/pull/188
    • feat: WalReplicator skeleton implementation by @waynexia in https://github.com/CeresDB/ceresdb/pull/179
    • feat: add new table_option storage format by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/218
    • feat: add wal write benchmark by @ygf11 in https://github.com/CeresDB/ceresdb/pull/178
    • feat: meta client supports route_tables by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/214
    • feat: persist AlterSchema and AlterOptions to data WAL by @waynexia in https://github.com/CeresDB/ceresdb/pull/166
    • feat: read hybrid format by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/207
    • feat: read hybrid format by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/208
    • feat: setup in different deployment mode by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/190
    • feat: support double quoted string. by @Rachelint in https://github.com/CeresDB/ceresdb/pull/196
    • feat: support route in cluster mode by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/215
    • feat: write hybrid storage format by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/185
    • feat: support to route for missing tables by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/221
    • fix: ci cache dependencies by @jiacai2050 in https://github.com/CeresDB/ceresdb/pull/193
    • fix: deadlock when reconnect heartbeat channel by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/217
    • fix: insert negative number value by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/212
    • fix: select single quoted literal string. by @Rachelint in https://github.com/CeresDB/ceresdb/pull/173
    • fix:select system.public.tables will return error by @dust1 in https://github.com/CeresDB/ceresdb/pull/174
    • refactor: avoid validation for table constraints during parse by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/164
    • refactor: build Payload in encode_batch for LogBatchEncoder by @ygf11 in https://github.com/CeresDB/ceresdb/pull/209
    • refactor: enhance cluster module based on meta_client_v2 by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/100
    • refactor: make read_batch object safe in WalManager by @ygf11 in https://github.com/CeresDB/ceresdb/pull/119
    • refactor: make write method object safe in WalManager by @ygf11 in https://github.com/CeresDB/ceresdb/pull/159
    • refactor: remove Meta and Wal type params in Instance by @ygf11 in https://github.com/CeresDB/ceresdb/pull/163
    • refactor: remove old meta_client crate by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/205
    • refactor: replace ColumnOption::DialectSpecific with ColumnOption::Comment by @baojinri in https://github.com/CeresDB/ceresdb/pull/167
    • refactor: store SpaceId and TableId in TableImpl by @waynexia in https://github.com/CeresDB/ceresdb/pull/202
    • refactor: try to replace dynamic-dispatch with pre-code in wal write by @ygf11 in https://github.com/CeresDB/ceresdb/pull/189
    • refactor: volatile implementation for catalog by @ShiKaiWi in https://github.com/CeresDB/ceresdb/pull/157

    New Contributors

    • @zwpaper made their first contribution in https://github.com/CeresDB/ceresdb/pull/141
    • @jakemcf22 made their first contribution in https://github.com/CeresDB/ceresdb/pull/142
    • @tianlinzx made their first contribution in https://github.com/CeresDB/ceresdb/pull/169
    • @Rachelint made their first contribution in https://github.com/CeresDB/ceresdb/pull/172
    • @baojinri made their first contribution in https://github.com/CeresDB/ceresdb/pull/167

    Full Changelog: https://github.com/CeresDB/ceresdb/compare/v0.2.0...v0.3.0

    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Jul 22, 2022)

    Overview In the v0.2.0 version, CeresDB implemented a static distributed deployment solution and completed some preparatory work for a cloud-native distributed cluster solution. In addition, related documents have been improved to facilitate developers to use and understand CeresDB.

    Features

    • Supports Aliyun OSS #20
    • WAL implementation based on OBKV #62
    • Distributed version supports static topology(only supports routing in grpc protocol)
    • Supports MySQL communication protocol #56
    • Improve the integration testing framework #86

    Thanks @dust1 @ygf11 @messense @GoGim1 @li-jin-gou @infdahai @VirrageS

    Source code(tar.gz)
    Source code(zip)
Owner
null
A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

Datafuse Labs 5k Jan 9, 2023
A Distributed SQL Database - Building the Database in the Public to Learn Database Internals

Table of Contents Overview Usage TODO MVCC in entangleDB SQL Query Execution in entangleDB entangleDB Raft Consensus Engine What I am trying to build

Sarthak Dalabehera 38 Jan 2, 2024
open source training courses about distributed database and distributed systemes

Welcome to learn Talent Plan Courses! Talent Plan is an open source training program initiated by PingCAP. It aims to create or combine some open sour

PingCAP 8.3k Dec 30, 2022
RisingWave is a cloud-native streaming database that uses SQL as the interface language.

RisingWave is a cloud-native streaming database that uses SQL as the interface language. It is designed to reduce the complexity and cost of building real-time applications. RisingWave consumes streaming data, performs continuous queries, and updates results dynamically. As a database system, RisingWave maintains results inside its own storage and allows users to access data efficiently.

Singularity Data 3.7k Jan 2, 2023
Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis

OnTimeDB Scalable and fast data store optimised for time series data such as financial data, events, metrics for real time analysis OnTimeDB is a time

Stuart 2 Apr 5, 2022
High performance and distributed KV store w/ REST API. 🦀

About Lucid KV High performance and distributed KV store w/ REST API. ?? Introduction Lucid is an high performance, secure and distributed key-value s

Lucid ᵏᵛ 306 Dec 28, 2022
Rust version of the Haskell ERD tool. Translates a plain text description of a relational database schema to dot files representing an entity relation diagram.

erd-rs Rust CLI tool for creating entity-relationship diagrams from plain text markup. Based on erd (uses the same input format and output rendering).

Dave Challis 32 Jul 25, 2022
Visualize your database schema

dbviz Visualize your database schema. The tool loads database schema and draws it as a graph. Usage $ dbviz -d database_name | dot -Tpng > schema.png

yunmikun2 2 Sep 4, 2022
A prototype of a high-performance KV database built with Rust.

async-redis A prototype of a high-performance KV database built with Rust. Author: 3andero 11/10/2021 Overview The project starts as a fork of mini-re

null 3 Nov 29, 2022
A dotfiles manager, with real time fle watching and 100% less sym-links!

Kubo A dotfile manager that watches files in real time. Usage Create a directory called .kubo in $HOME, then create a file called kubo.toml in .kubo.

StandingPad 5 Jul 24, 2023
Rust High Performance compile-time ORM(RBSON based)

WebSite | 简体中文 | Showcase | 案例 A highly Performant,Safe,Dynamic SQL(Compile time) ORM framework written in Rust, inspired by Mybatis and MybatisPlus.

rbatis 1.7k Jan 7, 2023
Distributed transactional key-value database, originally created to complement TiDB

Website | Documentation | Community Chat TiKV is an open-source, distributed, and transactional key-value database. Unlike other traditional NoSQL sys

TiKV Project 12.4k Jan 3, 2023
small distributed database protocol

clepsydra Overview This is a work-in-progress implementation of a core protocol for a minimalist distributed database. It strives to be as small and s

Graydon Hoare 19 Dec 2, 2021
A scalable, distributed, collaborative, document-graph database, for the realtime web

is the ultimate cloud database for tomorrow's applications Develop easier. Build faster. Scale quicker. What is SurrealDB? SurrealDB is an end-to-end

SurrealDB 16.9k Jan 8, 2023
Distributed SQL database in Rust, written as a learning project

toyDB Distributed SQL database in Rust, written as a learning project. Most components are built from scratch, including: Raft-based distributed conse

Erik Grinaker 4.6k Jan 8, 2023
Distributed, version controlled, SQL database with cryptographically verifiable storage, queries and results. Think git for postgres.

SDB - SignatureDB Distributed, version controlled, SQL database with cryptographically verifiable storage, queries and results. Think git for postgres

Fremantle Industries 5 Apr 26, 2022
Embedded Distributed Encrypted Database (Research).

EDED Embedded Distributed Encrypted Database. Research projects to support ESSE. WIP Distributed design features Adapt to personal distributed usecase

Sun 2 Jan 6, 2022
An easy-to-use, zero-downtime schema migration tool for Postgres

Reshape is an easy-to-use, zero-downtime schema migration tool for Postgres. It automatically handles complex migrations that would normally require downtime or manual multi-step changes.

Fabian Lindfors 1.4k Dec 25, 2022
Skybase is an extremely fast, secure and reliable real-time NoSQL database with automated snapshots and SSL

Skybase The next-generation NoSQL database What is Skybase? Skybase (or SkybaseDB/SDB) is an effort to provide the best of key/value stores, document

Skybase 1.4k Dec 29, 2022