Skip to content

refactor: reduce the memory of the HashIndex in the AggregateHashTable #19046

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
forsaken628 wants to merge 11 commits into
base: main
Choose a base branch
from
Open

refactor: reduce the memory of the HashIndex in the AggregateHashTable #19046

forsaken628 wants to merge 11 commits into from

Conversation

@forsaken628
Copy link
Collaborator

@forsaken628 forsaken628 commented Dec 2, 2025 •
edited
Loading

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Introduce RowID, using smaller types to represent rows
  • No longer need temp_vector for compare
  • Aggregate supports const columns

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-refactor this PR changes the code base without new features or bugfix label Dec 2, 2025
@forsaken628 forsaken628 added the ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits label Dec 2, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

Docker Image for PR

  • tag: pr-19046-485a16d-1764683080

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

ClickBench Report

@forsaken628
Revert "hash match"
ad5a4a8 
This reverts commit 1e27e0f.
@forsaken628 forsaken628 added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 2, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

Docker Image for PR

  • tag: pr-19046-ab10226-1764700210

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2025

ClickBench Report

@forsaken628 forsaken628 added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

Docker Image for PR

  • tag: pr-19046-39fb5b9-1764758008

note: this image tag is only available for internal use.

@everpcpc everpcpc added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

Docker Image for PR

  • tag: pr-19046-6697d22-1764767755

note: this image tag is only available for internal use.

@everpcpc everpcpc added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

Docker Image for PR

  • tag: pr-19046-21c318b-1764774146

note: this image tag is only available for internal use.

@everpcpc everpcpc added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 3, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2025

Docker Image for PR

  • tag: pr-19046-b6a4e76-1764803676

note: this image tag is only available for internal use.

@everpcpc everpcpc added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 4, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2025

Docker Image for PR

  • tag: pr-19046-d0ad800-1764820048

note: this image tag is only available for internal use.

@everpcpc
Copy link
Member

everpcpc commented Dec 4, 2025

https://benchmark.databend.com/clickbench/pr/19046/19913624838/hits.html

@everpcpc everpcpc removed the ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits label Dec 4, 2025
@everpcpc everpcpc added the ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits label Dec 4, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2025

Docker Image for PR

  • tag: pr-19046-ab64d59-1764856860

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 4, 2025

ClickBench Report

@forsaken628 forsaken628 force-pushed the aggr_table branch 2 times, most recently from 8542e9f to 1fe1b23 Compare December 5, 2025 16:44
@forsaken628 forsaken628 added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 6, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 6, 2025

Docker Image for PR

  • tag: pr-19046-77b5534-1765013470

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 6, 2025

ClickBench Report

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Dec 6, 2025

ClickBench Report

@forsaken628 forsaken628 changed the title refactor: reduce memory refactor: reduce the memory of the HashIndex in the AggregateHashTable Dec 8, 2025
@forsaken628 forsaken628 marked this pull request as ready for review December 8, 2025 05:48
@forsaken628 forsaken628 requested a review from sundy-li December 8, 2025 05:50
chatgpt-codex-connector[bot]
chatgpt-codex-connector bot reviewed Dec 8, 2025
View reviewed changes
Copy link

@chatgpt-codex-connector chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@BohuTANG
Copy link
Member

BohuTANG commented Dec 9, 2025

@codex review

@chatgpt-codex-connector
Copy link

chatgpt-codex-connector bot commented Dec 9, 2025

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, add credits to your account and enable them for code reviews in your settings.

@BohuTANG BohuTANG added ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits and removed ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits labels Dec 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2025

Docker Image for PR

  • tag: pr-19046-ca96af3-1765291122

note: this image tag is only available for internal use.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2025

ClickBench Report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@chatgpt-codex-connector chatgpt-codex-connector[bot] chatgpt-codex-connector[bot] left review comments

@sundy-li sundy-li sundy-li approved these changes

Assignees

No one assigned

Labels

ci-benchmark-cloud Benchmark: run only cloud tests for tpch/hits pr-refactor this PR changes the code base without new features or bugfix

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@forsaken628 @everpcpc @BohuTANG @sundy-li