Refactor Aggregator for improve performance and prepare refactor transform_aggregator_final #1701

zhang2014 · 2021-09-02T03:29:01Z

I hereby agree to the terms of the CLA available at: https://datafuse.rs/policies/cla/

Summary

Refactoring the aggregator can better adapt to refactor transform_aggregator_final
Investigate and fix the causes of performance degradation
- Inline not work
- Declare local references
- While match optimize match while
- clippy::ptr_arg

Query:
  SELECT  sum(number) FROM numbers_mt(10000000000)  group by number % 3, number % 4, number % 5;

Before:
  60 rows in set. Elapsed: 12.008 sec. Processed 10.00 billion rows, 80.00 GB (832.77 million rows/s., 6.66 GB/s.)

After:
  60 rows in set. Elapsed: 11.258 sec. Processed 10.00 billion rows, 80.00 GB (888.24 million rows/s., 7.11 GB/s.)

ClickHouse (master):
  60 rows in set. Elapsed: 12.152 sec. Processed 10.00 billion rows, 80.00 GB (822.93 million rows/s., 6.58 GB/s.)

Changelog

Performance Improvement

databend-bot · 2021-09-02T03:29:08Z

Hello @zhang2014, 🎉 Thank you for opening the pull request! 🎉
Your pull request state is not in Draft, please add Reviewers or Re-request review again!
FuseQuery: @BohuTANG @sundy-li @zhang2014
FuseStore: @drmingdrmer @dantengsky
Or visit datafuse roadmap for some clues.

databend-bot · 2021-09-02T03:29:12Z

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

codecov-commenter · 2021-09-02T03:57:22Z

Codecov Report

Merging #1701 (0198752) into master (7b2afe7) will increase coverage by 0%.
The diff coverage is 74%.

@@           Coverage Diff           @@
##           master   #1701    +/-   ##
=======================================
  Coverage      71%     72%            
=======================================
  Files         568     577     +9     
  Lines       33130   33960   +830     
=======================================
+ Hits        23823   24544   +721     
- Misses       9307    9416   +109

Impacted Files	Coverage Δ
...nctions/src/aggregates/aggregate_function_state.rs	`70% <ø> (ø)`
.../pipelines/transforms/group_by/aggregator_state.rs	`20% <38%> (+2%)`	⬆️
...ry/src/pipelines/transforms/group_by/aggregator.rs	`86% <85%> (-4%)`	⬇️
...pipelines/transforms/group_by/aggregator_params.rs	`93% <100%> (+<1%)`	⬆️
...pipelines/transforms/transform_group_by_partial.rs	`75% <100%> (+<1%)`	⬆️
query/src/api/http/v1/cluster.rs	`70% <0%> (-17%)`	⬇️
query/src/catalogs/impls/database_catalog.rs	`78% <0%> (-6%)`	⬇️
...unctions/src/aggregates/aggregate_function_test.rs	`87% <0%> (-1%)`	⬇️
query/src/sql/sql_parser.rs	`76% <0%> (-1%)`	⬇️
common/datavalues/src/macros.rs	`31% <0%> (-1%)`	⬇️
... and 29 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 7b2afe7...0198752. Read the comment docs.

databend-bot · 2021-09-02T05:03:44Z