A Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture, built to make the Data Cloud easy

Datafuse Labs

Last update: Jan 9, 2023

Overview

Datafuse

Modern Real-Time Data Processing & Analytics DBMS with Cloud-Native Architecture

Built to make the Data Cloud easy!

Principles

Fearless
- No data races, No unsafe, Minimize unhandled errors
High Performance
- Everything is Parallelism
High Scalability
- Everything is Distributed
High Reliability
- Datafuse primary design goal is reliability

Architecture

Performance

Memory SIMD-Vector processing performance only
Dataset: 100,000,000,000 (100 Billion)
Hardware: AMD Ryzen 7 PRO 4750U, 8 CPU Cores, 16 Threads
Rust: rustc 1.55.0-nightly (868c702d0 2021-06-30)
Build with Link-time Optimization and Using CPU Specific Instructions
ClickHouse server version 21.4.6 revision 54447

Query	FuseQuery (v0.4.48-nightly)	ClickHouse (v21.4.6)
SELECT avg(number) FROM numbers_mt(100000000000)	4.35 s. (22.97 billion rows/s., 183.91 GB/s.)	×1.4 slow, (6.04 s.) (16.57 billion rows/s., 132.52 GB/s.)
SELECT sum(number) FROM numbers_mt(100000000000)	4.20 s. (23.79 billion rows/s., 190.50 GB/s.)	×1.4 slow, (5.90 s.) (16.95 billion rows/s., 135.62 GB/s.)
SELECT min(number) FROM numbers_mt(100000000000)	4.92 s. (20.31 billion rows/s., 162.64 GB/s.)	×2.7 slow, (13.05 s.) (7.66 billion rows/s., 61.26 GB/s.)
SELECT max(number) FROM numbers_mt(100000000000)	4.77 s. (20.95 billion rows/s., 167.78 GB/s.)	×3.0 slow, (14.07 s.) (7.11 billion rows/s., 56.86 GB/s.)
SELECT count(number) FROM numbers_mt(100000000000)	2.91 s. (34.33 billion rows/s., 274.90 GB/s.)	×1.3 slow, (3.71 s.) (26.93 billion rows/s., 215.43 GB/s.)
SELECT sum(number+number+number) FROM numbers_mt(100000000000)	19.83 s. (5.04 billion rows/s., 40.37 GB/s.)	×12.1 slow, (233.71 s.) (427.87 million rows/s., 3.42 GB/s.)
SELECT sum(number) / count(number) FROM numbers_mt(100000000000)	3.90 s. (25.62 billion rows/s., 205.13 GB/s.)	×2.5 slow, (9.70 s.) (10.31 billion rows/s., 82.52 GB/s.)
SELECT sum(number) / count(number), max(number), min(number) FROM numbers_mt(100000000000)	8.28 s. (12.07 billion rows/s., 96.66 GB/s.)	×4.0 slow, (32.87 s.) (3.04 billion rows/s., 24.34 GB/s.)
SELECT number FROM numbers_mt(10000000000) ORDER BY number DESC LIMIT 100	4.80 s. (2.08 billion rows/s., 16.67 GB/s.)	×2.9 slow, (13.95 s.) (716.62 million rows/s., 5.73 GB/s.)
SELECT max(number), sum(number) FROM numbers_mt(1000000000) GROUP BY number % 3, number % 4, number % 5	6.31 s. (158.49 million rows/s., 1.27 GB/s.)	×1.02 fast, (6.18 s.) (161.84 million rows/s., 1.29 GB/s.)

Note:

ClickHouse system.numbers_mt is 16-way parallelism processing, gist
FuseQuery system.numbers_mt is 16-way parallelism processing, gist

Getting Started

Roadmap

Datafuse is currently in Alpha and is not ready to be used in production, Roadmap 2021

Contributing

License

Datafuse is licensed under Apache 2.0.

Comments

[store] refactor: rename store to dfs
I hereby agree to the terms of the CLA available at: https://databend.rs/policies/cla/

Summary

[store] refactor: rename store to dfs

Changelog

Improvement

Related Issues
opened by drmingdrmer 387
Rename trait type names from I$Name to $Name
I hereby agree to the terms of the CLA available at: https://datafuse.rs/policies/cla/

Summary

Rename trait type names from I$Name to $Name

Changelog

Renames :

ITable to Table

IDatabase to Database

Removes IDataSource, use struct DataSource directly

And relevant code

Related Issues

Fixes #727

Test Plan

No extra ut/stateless_test
opened by dantengsky 127
add toStartOfWeek
I hereby agree to the terms of the CLA available at: https://databend.rs/policies/cla/

Summary

Summary about this PR

Changelog

Improvement

Related Issues

#853

Test Plan

Unit Tests ok

Stateless Tests ok
opened by dust1 99
ISSUE-2039: rm param "database" from get_table_by_id
I hereby agree to the terms of the CLA available at: https://databend.rs/policies/cla/

Summary

signature of Catalog::get_table_by_id changed to

fn get_table_by_id( &self, table_id: MetaId, table_version: Option<MetaVersion>, ) -> Result<Arc<TableMeta>>;

the "database_name" parameter has been removed.

NOTE:

add database_name and table_name to struct common/meta/types/Table

added annotation #[allow(clippy::large_enum_variant)] to pub enum AppliedState to disable the warning variant is 400 bytes

Changelog

Improvement

Not for changelog (changelog entry is not required)

Related Issues

Fixes #2039

Test Plan

Unit Tests

Stateless Tests
opened by dantengsky 70
[query/server/http] Add /v1/query.
I hereby agree to the terms of the CLA available at: https://databend.rs/policies/cla/

Summary

Add new endpoint /v1/query to support sql.

handler usage

post query with JSON with format HttpQueryRequest, and return JSON with format QueryResponse

Pagination with long polling.

result data is embedded in QueryResponse.

return the following 4 URIs, the client should rely on the endpoint /v1/query and URIs returned in the response. the other endpoints may change without notices.

next: get the next page (together with progress) with long polling. page N is released when Page N+1 is requested.

state: return struct like QueryResponse, but the data field is empty. short polling.

kill: kill running query

delete: kill running query and delete it

test_async can be a demo for their usage.

pub struct HttpQueryRequest { pub sql: String, }

pub struct QueryResponse { pub id: Option<String>, pub columns: Option<DataSchemaRef>, pub data: JsonBlockRef, pub next_uri: Option<String>, pub state_uri: Option<String>, pub kill_uri: Option<String>, pub delete_uri: Option<String>, pub query_state: Option<String>, pub request_error: Option<String>, pub query_error: Option<String>, pub query_progress: Option<ProgressValues>, }

internal

The web layer (query_handlers.rs) is separated from a more structured internal implementation in mod v1/query.

/v1/statement is kept to post raw SQL, may reuse the internal implementation, this will be done some others changes in another PR.

hope it may event be used to support GRPC if need later?

make sure the internal tokio task stop fast when the query is killed by SQL command or HTTP request

TODO

soon( new PR in 1 week or so)：

classification and organization of Errors returned will be polished later.

more tests.

rewrite /v1/statements.

Long term:

Add client session support.

Adjust the handler interface to a stable version with a formal doc. only necessary fields are added currently. including 2 JSON formats and optional URL parameters and headers.

better control of memory.

Changelog

New Feature

Related Issues

Fixes #2604

Test Plan

Unit Tests
pr-feature community-take
opened by youngsofun 63
[ci] fix gcov install failed
Signed-off-by: Chojan Shang [email protected]

I hereby agree to the terms of the CLA available at: https://datafuse.rs/policies/cla/

Summary

remove ~/.cargo/bin/ from cache

Changelog

Build/Testing/CI

Not for changelog (changelog entry is not required)

Related Issues

Fixes #1012

Test Plan

No
pr-build
opened by PsiACE 63
Implements Feature 630
Summary

It's a baby step of integrating Store with Query, which implements

update metadata after appending data parts to the table

remote table read_plan

remote table read

Basic statements like insert into ... and select .. from ... could be executed now. (and lots of interesting things are left to do)

Changelog

Store: implementions for ITable read_plan and read a5c42b2e5d14d042f3c3d928a35c625ca32f4410

Query: implements RemoteTalbe's read_plan & read b55eacf912bc7985765b870ecd658d505eb75a56

Adds remote flag to ReadDataSourcePlan deaea8ea29b4a6d4afb1390cdd3e0d3540b2597c

Tweaks stateless test cases ed69c92fc37c01650f17478f4d6e446f828f74ad

The following issues might be worthy of your concern:

Remove trait bound Sync from SendableDataBlockStream ed69c92fc37c01650f17478f4d6e446f828f74ad

Turns out, at least for now, we do not need this trait bound, and without Sync constraint, SendableDataBlockStream is more stream-combinator friendly.

Keep ITable::read_plan as a non-async method

By using runtime of ctx (and channel). IMHO, change ITable::read_plan to async fn may be too harsh at this stage.

Add an extra flag to ReadDataSourcePlan and SourceTransform

So that we could be aware of operating a remote table(and fetch remote table accordingly). It is a temp workaround, let's postpone it until the Catalog API is ready. SourceTransform::execute and FuseQueryContext are tweaked accordingly. pls see deaea8ea29b4a6d4afb1390cdd3e0d3540b2597c

Related Issues

resolves #630

Test Plan

UT & Stateless Testes

Progress

[x] Update meta

[x] Flight Service

[x] Store Client

[x] Remote table - read_plan

[x] Remote table - read (read partition)

[x] Unit tests & integration tests

[x] Multi-Node integration tests

[x] Code GC

[x] Squash commits
opened by dantengsky 63
[query] refactor: Table::read_plan does not need DatabendQueryContext anymore.
I hereby agree to the terms of the CLA available at: https://databend.rs/policies/cla/

Summary

[query] refactor: Table::read_plan does not need DatabendQueryContext anymore.

Why: Table is a low level concept in databend code base and should be a dependency of some other crate such as common/planner.

This commit is one of the steps to remove the Table dependency on query.

Trait Table references DatabendQueryContext as an argument type. In this commit it is replaced with another smaller type TableIOContext so that in future Table can be moved out of crate query.

TableIOContext provides data-access support, exposes the runtime used by query itself and provides two resource value: max thread number and node list.

Add TableIOContext to provide everything a Table need to build a plan or else.

Table::read_plan() use TableIOContext as argument to replace DatabendQueryContext.

When calling read_plan(), a temporary TableIOContext is built out of DatabendQueryContext.

DatabendQueryContext provides two additional supporting methods: get_single_node_table_io_context() and get_cluster_table_io_context().

fix: #2072

Changelog

Improvement

Related Issues

#2046

#2059
opened by drmingdrmer 52
Cast functions between String and Date16/Date32/DateTime32
I hereby agree to the terms of the CLA available at: https://databend.rs/policies/cla/

Summary

Summary about this PR

Changelog

New Feature

Related Issues

Related #853

Test Plan

Unit Tests

Stateless Tests
pr-feature
opened by sundy-li 50
ISSUE-1639:Remove session_api.rs
I hereby agree to the terms of the CLA available at: https://datafuse.rs/policies/cla/

Summary

Remove common/store-api/session_api.rs

Changelog

Improvement

Related Issues

Fixes #1639

Test Plan

Unit Tests

Stateless Tests
opened by jyz0309 39
Consider renaming project. DataFuse is too similar to DataFusion.

This project appears to have similar goals to Apache Arrow DataFusion, contains code from DataFusion, and has a very similar name.

The names "DataFuse" and "DataFusion" only differ by a few characters and this could cause confusion about the relationship between these projects.

On behalf of the Apache Arrow DataFusion community, who have put a lot of work into building the DataFusion software and brand over the past three years, I respectfully ask that you consider renaming this project.

opened by andygrove 35
Alter table sql support

I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/

Summary

feat: It is not a pr for review, it just a craft pr for running unit tests of alter table modification.

opened by lichuang 1

bug: expression branch wrap_nullable

Search before asking

[X] I had searched in the issues and found no similar issues.

Version

expression branch

What's Wrong?

wrap_nullable will wrap the nullable type and get the inner value. If the inner value is null , will use default value.

How to Reproduce?

CREATE TABLE t1(s String NULL, pat String NULL, pos Int64 NULL, occu Int64 NULL, ro Int64 NULL, mt String NULL) Engine = Fuse;

INSERT INTO t1 (s, pat, pos, occu, ro, mt) VALUES ('dog cat dog', 'dog', NULL, 1, 1, 'c');

SELECT s FROM t1 WHERE REGEXP_INSTR(s, pat, pos, occu, ro, mt) = 4;
ERROR 1105 (HY000): result row write failed: Code: 1001, displayText = Incorrect arguments to regexp_instr: position must be positive, but got 0.

expect result : NULL.

Are you willing to submit PR?

[ ] Yes I am willing to submit a PR!

C-bug

opened by TCeason 0

feat: update support subquery

mysql> create table t1(id1 int, val1 varchar(255));
Query OK, 0 rows affected (0.01 sec)

mysql> create table t2(id2 int, val2 varchar(255));
Query OK, 0 rows affected (0.01 sec)

mysql> insert into t1 values (1,'1') ;
Query OK, 1 row affected (0.01 sec)

mysql> insert into t2 values (1,'2');
Query OK, 1 row affected (0.01 sec)

mysql> update t1 set val1 = (select val2 from t2 where id1 = id2) where id1 in (select id2 from t2);
ERROR 1105 (HY000): Code: 1001, displayText = Unsupported physical scalar: SubqueryExpr(SubqueryExpr { typ: Any, subquery: SExpr { plan: EvalScalar(EvalScalar { items: [ScalarItem { scalar: BoundColumnRef(BoundColumnRef { column: ColumnBinding { database_name: Some("default"), table_name: Some("t2"), column_name: "id2", index: 4, data_type: Int32(Int32), visibility: Visible } }), index: 4 }] }), children: [SExpr { plan: LogicalGet(LogicalGet { table_index: 2, columns: {5, 4}, push_down_predicates: None, limit: None, order_by:

opened by zhyass 0

Roadmap 2023
After a full year of research and development in 2022, the functionality and stability of Databend were significantly enhanced, and several users began using it in production. Databend has helped them greatly reduce costs and operational complexity issues.

This is Databend Roadmap in 2023 (discussion).

See also:

Roadmap 2022: #3706

Roadmap 2021: #746

Main tasks

Features

| Task | Status | Comments | |------------------------------------------------------------------------------------|-------------|----------| | Update | IN PROGRESS | | | Merge | PLAN | | | Alter table | IN PROGRESS | | | Window function | PLAN | | | Lambda function and high-order functions | PLAN | | | TimestampTz data type | PLAN | | | Decimal data type | PLAN | | | Materialized view | PLAN | | | Support SET_VAR hints#8833 | PLAN | | | Parquet reader | PLAN | | | Distributed COPY | PLAN | | | JSON indexing | PLAN | | | DataFrame | PLAN | | | Data Sharing(community version) | IN PROGRESS | | | Concurrent query enhance | PLAN | |

Improvements

| Task | Status | Comments | |---------------------------------------------------------------------------|-------------|----------| | New expression#9411 | IN PROGRESS | | | Error message | PLAN | |

Planner

| Task | Status | Comments | |----------------------------------------------------------------------------------------------|-------------|----------------------| | Scalar expression normalization | PLAN | | | Column constraint framework | PLAN | | | Functional dependency framework#7438 | PLAN | | | Join reorder | IN PROGRESS | | | CBO for distributed plan | PLAN | | | Support TPC-DS | PLAN | | | Support optimization tracing | PLAN | Easy to debug/study. |

Cache

| Task | Status | Comments | |---------------------|---------|----------| | Unified cache layer | IN PROGRESS | | | Meta data cache | IN PROGRESS | | | Index data cache | IN PROGRESS | | | Block data cache | PLAN | |

Data Storage

| Task | Status | Comments | |---------------------------------|--------|----------------------------------------| | Fuse engine re-clustering | PLAN | | | Fuse engine orphan data cleanup | PLAN | | | Fuse engine segment tree | PLAN | Support large dataset(PB) in one table |

LakeHouse

| Task | Status | Comments | |------------------------------------|-------------|----------| | Apache Hive | IN PROGRESS | | | Apache Iceberg | IN PROGRESS | | | Querying external storage(Parquet) | IN PROGRESS | |

Distributed Query Execution

| Task | Status | Comments | |----------------------|-------------|----------| | Visualized profiling | IN PROGRESS | | | Aggregation spilling | IN PROGRESS | |

Resource Quota

| Task | Status | Comments | |------------------------------------------|-------------|----------| | Session-level quota control (CPU/Memory) | IN PROGRESS | | | User-level quota control (CPU/Memory) | PLAN | |

Integrations

| Task | Status | Comments | |-------------------------------------------|-------------|----------| | Dbt integration | IN PROGRESS | | | Airbyte integration | IN PROGRESS | | | Datadog Vector integrate with Rust-driver | IN PROGRESS | | | Datax integrate with Java-driver | IN PROGRESS | | | CDC with Flink | PLAN | | | CDC with Kafka | PLAN | |

Meta

| Task | Status | Comments | |-------------|-------------|----------| | Jepsen test | IN PROGRESS | |

Testing

| Task | Status | Comments | |---------------|-------------|-----------------------------------| | SQLlogic Test | IN PROGRESS | Supports more test cases | | SQLancer Test | IN PROGRESS | Supports more type and more cases | | Fuzzer Test | PLAN | |

Releases

[x] v0.8 #4591

[x] v0.7 #3428

[x] v0.6 #2525

[x] v0.5 #2257
opened by BohuTANG 4