Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[function] compatible with mysql when aggregate function work on empty data #848

Merged
merged 1 commit into from Jun 19, 2021
Merged

[function] compatible with mysql when aggregate function work on empty data #848

merged 1 commit into from Jun 19, 2021

Conversation

zhaox1n
Copy link
Contributor

@zhaox1n zhaox1n commented Jun 17, 2021 •
edited

I hereby agree to the terms of the CLA available at: https://datafuse.rs/policies/cla/

Summary

Summary about this PR

Changelog

  • Bug Fix
  • Improvement

Related Issues

Fixes #771

Test Plan

Unit Tests

Stateless Tests

@databend-bot databend-bot added the pr-bugfix this PR patches a bug in codebase label Jun 17, 2021
@databend-bot
Copy link
Member

Thanks for the contribution!
I have applied any labels matching special text in your PR Changelog.

Please review the labels and make any necessary changes.

@databend-bot
Copy link
Member

Hello @zhaox1n, 🎉 Thank you for opening the pull request! 🎉
Your pull request state is not in Draft, please add Reviewers or Re-request review again!
FuseQuery: @BohuTANG @sundy-li @zhang2014
FuseStore: @drmingdrmer @dantengsky
Or visit datafuse roadmap for some clues.

2 similar comments
@databend-bot
Copy link
Member

Hello @zhaox1n, 🎉 Thank you for opening the pull request! 🎉
Your pull request state is not in Draft, please add Reviewers or Re-request review again!
FuseQuery: @BohuTANG @sundy-li @zhang2014
FuseStore: @drmingdrmer @dantengsky
Or visit datafuse roadmap for some clues.

@databend-bot
Copy link
Member

Hello @zhaox1n, 🎉 Thank you for opening the pull request! 🎉
Your pull request state is not in Draft, please add Reviewers or Re-request review again!
FuseQuery: @BohuTANG @sundy-li @zhang2014
FuseStore: @drmingdrmer @dantengsky
Or visit datafuse roadmap for some clues.

@zhaox1n zhaox1n changed the title [function] compatible with mysql when aggregate function work on empt… [function] compatible with mysql when aggregate function work on empty data Jun 17, 2021
@databend-bot
Copy link
Member

Hello @zhaox1n, 🎉 Thank you for opening the pull request! 🎉
Your pull request state is not in Draft, please add Reviewers or Re-request review again!
FuseQuery: @BohuTANG @sundy-li @zhang2014
FuseStore: @drmingdrmer @dantengsky
Or visit datafuse roadmap for some clues.

zhang2014
zhang2014 reviewed Jun 17, 2021
View reviewed changes
common/aggregate_functions/src/aggregate_function_test.rs
display: "uniq",
func_name: "uniq",
columns: vec![columns[0].clone()],
expect: DataValue::UInt64(Some(0)),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe uniq function should return empty rows?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The uniq function seems to behave the same as count distinct, so I return 0

@sundy-li
Copy link
Member

sundy-li commented Jun 18, 2021 •
edited

@ zhaox1n thanks for the contribution.

Some advice about this pr:

  1. The state of the AggregateFunction should be initialized as Null, such as DataValueUInt64(None) for count function.
  2. To make the SQL return null results on empty data, you just need to modify 3 line codes.
--- a/fusequery/query/src/pipelines/transforms/transform_aggregator_final.rs
+++ b/fusequery/query/src/pipelines/transforms/transform_aggregator_final.rs
@@ -89,9 +89,9 @@ impl Processor for AggregatorFinalTransform {
         for func in &funcs {
             let merge_result = func.merge_result()?;
             // Check merge result null.
-            if merge_result.is_null() {
-                break;
-            }
+            // if merge_result.is_null() {
+            //     break;
+            // }
  1. Modify the pub fn to_array_with_size(&self, size: usize) -> Result<DataArrayRef> method to build array with other None type values, then we don't need transform_none_to_null anymore.

such as

-            DataValue::UInt64(Some(v)) => {
-                Ok(Arc::new(UInt64Array::from(vec![*v; size])) as DataArrayRef)
-            }
+            DataValue::UInt64(v) => match v {
+                Some(v) => Ok(Arc::new(UInt64Array::from(vec![*v; size])) as DataArrayRef),
+                None => Ok(new_null_array(&DataType::UInt64, size)),
+            },

  1. if input_rows == 0 is useless, we don't want this check in every impl functions.

  2. create our own data_value_to_string method to display the null value as MySQL behaviors.

@zhaox1n zhaox1n force-pushed the datafuse_771 branch 2 times, most recently from 2aa7545 to 7feb64c Compare June 18, 2021 16:00
@zhaox1n
Copy link
Contributor Author

zhaox1n commented Jun 18, 2021

@ zhaox1n thanks for the contribution.

Some advice about this pr:

  1. The state of the AggregateFunction should be initialized as Null, such as DataValueUInt64(None) for count function.
  2. To make the SQL return null results on empty data, you just need to modify 3 line codes.
--- a/fusequery/query/src/pipelines/transforms/transform_aggregator_final.rs
+++ b/fusequery/query/src/pipelines/transforms/transform_aggregator_final.rs
@@ -89,9 +89,9 @@ impl Processor for AggregatorFinalTransform {
         for func in &funcs {
             let merge_result = func.merge_result()?;
             // Check merge result null.
-            if merge_result.is_null() {
-                break;
-            }
+            // if merge_result.is_null() {
+            //     break;
+            // }
  1. Modify the pub fn to_array_with_size(&self, size: usize) -> Result<DataArrayRef> method to build array with other None type values, then we don't need transform_none_to_null anymore.

such as

-            DataValue::UInt64(Some(v)) => {
-                Ok(Arc::new(UInt64Array::from(vec![*v; size])) as DataArrayRef)
-            }
+            DataValue::UInt64(v) => match v {
+                Some(v) => Ok(Arc::new(UInt64Array::from(vec![*v; size])) as DataArrayRef),
+                None => Ok(new_null_array(&DataType::UInt64, size)),
+            },
  1. if input_rows == 0 is useless, we don't want this check in every impl functions.
  2. create our own data_value_to_string method to display the null value as MySQL behaviors.

CC @sundy-li @zhang2014

sundy-li
sundy-li approved these changes Jun 19, 2021
View reviewed changes
Copy link
Member

@sundy-li sundy-li left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

6 similar comments
@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

13 similar comments
@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@databend-bot
Copy link
Member

CI Passed
Reviewer Approved
Let's Merge

@sundy-li sundy-li merged commit 17e7682 into datafuselabs:master Jun 19, 2021
@BohuTANG
Copy link
Member

github api rejected the bot again

@zhaox1n zhaox1n deleted the datafuse_771 branch June 19, 2021 06:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@zhang2014 zhang2014 zhang2014 left review comments

@sundy-li sundy-li sundy-li approved these changes

Assignees
No one assigned
Labels
pr-bugfix this PR patches a bug in codebase
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Return default value when aggregating empty data
5 participants
@zhaox1n @databend-bot @sundy-li @BohuTANG @zhang2014