Skip to content

[fix](spill) Memory leak in the arena used by the aggregation source … #60205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
yiguolei merged 1 commit into from
Jan 26, 2026
Merged

[fix](spill) Memory leak in the arena used by the aggregation source … #60205

yiguolei merged 1 commit into from
Jan 26, 2026

Conversation

@dh-cloud
Copy link
Contributor

@dh-cloud dh-cloud commented Jan 24, 2026

…operator

After switching spill partition in partitioned aggregation source operator, the agg_arena_pool also needs to be reset.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 24, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yiguolei
Copy link
Contributor

yiguolei commented Jan 26, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

TPC-H: Total hot run time: 31245 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c90958e3246300357ce683ed2bd9b9d78bf3ec31, data reload: false

------ Round 1 ----------------------------------
q1	17662	4748	4557	4557
q2	2022	307	191	191
q3	10242	1274	731	731
q4	10238	897	315	315
q5	7731	2076	1863	1863
q6	201	168	137	137
q7	866	700	578	578
q8	9262	1385	1115	1115
q9	5068	4651	4545	4545
q10	6829	1661	1285	1285
q11	516	308	285	285
q12	384	373	220	220
q13	17800	3875	3139	3139
q14	238	244	214	214
q15	602	526	523	523
q16	647	626	590	590
q17	660	840	451	451
q18	6691	6457	6732	6457
q19	1904	1053	633	633
q20	403	372	231	231
q21	2986	2200	2127	2127
q22	1122	1072	1058	1058
Total cold run time: 104074 ms
Total hot run time: 31245 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5177	4972	4877	4877
q2	344	398	323	323
q3	2363	2922	2531	2531
q4	1497	1882	1568	1568
q5	4534	4505	4505	4505
q6	213	170	131	131
q7	2092	2059	1907	1907
q8	2585	2416	2428	2416
q9	7029	7282	7029	7029
q10	2521	2702	2298	2298
q11	511	452	441	441
q12	684	728	565	565
q13	3376	3802	3171	3171
q14	264	298	257	257
q15	539	515	495	495
q16	634	675	622	622
q17	1116	1370	1355	1355
q18	7491	7385	7096	7096
q19	827	821	811	811
q20	1922	1962	1836	1836
q21	4598	4285	4096	4096
q22	1053	1007	993	993
Total cold run time: 51370 ms
Total hot run time: 49323 ms

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

TPC-DS: Total hot run time: 173365 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c90958e3246300357ce683ed2bd9b9d78bf3ec31, data reload: false

query5	4782	658	495	495
query6	331	220	202	202
query7	4221	455	267	267
query8	330	239	226	226
query9	8741	2889	2869	2869
query10	443	341	261	261
query11	15341	15243	14881	14881
query12	187	119	112	112
query13	1245	465	359	359
query14	6475	3108	2907	2907
query14_1	2852	2775	2840	2775
query15	207	197	171	171
query16	978	491	455	455
query17	1077	701	597	597
query18	2644	439	333	333
query19	193	177	158	158
query20	133	120	119	119
query21	221	152	127	127
query22	3836	4036	3960	3960
query23	15839	15493	15288	15288
query23_1	15400	15278	15408	15278
query24	7172	1557	1136	1136
query24_1	1159	1166	1198	1166
query25	524	447	381	381
query26	1235	281	152	152
query27	2736	456	283	283
query28	4527	2184	2168	2168
query29	791	548	436	436
query30	317	239	206	206
query31	838	646	552	552
query32	84	75	72	72
query33	532	373	314	314
query34	916	890	552	552
query35	734	736	678	678
query36	886	932	876	876
query37	142	102	89	89
query38	2776	2814	2703	2703
query39	765	751	747	747
query39_1	706	710	728	710
query40	224	140	119	119
query41	85	78	78	78
query42	98	96	96	96
query43	428	444	412	412
query44	1371	757	752	752
query45	192	186	179	179
query46	870	959	592	592
query47	1399	1447	1315	1315
query48	314	330	238	238
query49	597	433	346	346
query50	706	270	207	207
query51	3866	3871	3863	3863
query52	94	94	82	82
query53	210	230	173	173
query54	283	257	264	257
query55	85	81	81	81
query56	313	310	336	310
query57	1011	999	981	981
query58	268	280	267	267
query59	2026	2222	2059	2059
query60	338	324	318	318
query61	142	145	135	135
query62	382	352	302	302
query63	199	163	164	163
query64	4882	1171	827	827
query65	3840	3769	3771	3769
query66	1389	424	316	316
query67	15610	15505	15407	15407
query68	2369	1061	729	729
query69	415	309	291	291
query70	994	901	906	901
query71	304	311	278	278
query72	5256	3132	3208	3132
query73	570	703	322	322
query74	8723	8698	8470	8470
query75	2316	2347	1905	1905
query76	2292	1086	676	676
query77	358	388	301	301
query78	9566	9691	9119	9119
query79	1072	903	596	596
query80	1293	547	441	441
query81	535	271	238	238
query82	1021	152	122	122
query83	336	263	257	257
query84	255	118	99	99
query85	866	477	402	402
query86	407	302	297	297
query87	2970	2894	2863	2863
query88	3531	2602	2570	2570
query89	324	262	232	232
query90	1967	175	175	175
query91	170	155	132	132
query92	78	75	73	73
query93	1092	1021	637	637
query94	654	318	295	295
query95	596	399	315	315
query96	637	512	234	234
query97	2354	2379	2335	2335
query98	215	213	205	205
query99	584	574	519	519
Total cold run time: 246727 ms
Total hot run time: 173365 ms

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

ClickBench: Total hot run time: 26.82 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c90958e3246300357ce683ed2bd9b9d78bf3ec31, data reload: false

query1	0.05	0.04	0.05
query2	0.10	0.04	0.05
query3	0.25	0.09	0.09
query4	1.61	0.12	0.12
query5	0.27	0.24	0.24
query6	1.15	0.67	0.65
query7	0.04	0.02	0.03
query8	0.05	0.04	0.05
query9	0.57	0.50	0.51
query10	0.55	0.55	0.54
query11	0.15	0.10	0.10
query12	0.14	0.10	0.12
query13	0.60	0.59	0.58
query14	0.96	0.93	0.93
query15	0.80	0.78	0.78
query16	0.39	0.43	0.40
query17	0.97	1.07	1.06
query18	0.23	0.22	0.21
query19	1.94	1.84	1.78
query20	0.02	0.01	0.02
query21	15.44	0.27	0.15
query22	5.08	0.06	0.05
query23	15.77	0.27	0.10
query24	2.47	0.34	0.39
query25	0.12	0.05	0.07
query26	0.14	0.14	0.14
query27	0.09	0.05	0.06
query28	4.53	1.06	0.88
query29	12.51	3.89	3.16
query30	0.27	0.14	0.12
query31	2.81	0.59	0.39
query32	3.24	0.56	0.45
query33	2.98	3.00	3.03
query34	16.03	5.09	4.41
query35	4.52	4.44	4.48
query36	0.68	0.52	0.48
query37	0.12	0.07	0.07
query38	0.08	0.04	0.04
query39	0.04	0.03	0.03
query40	0.16	0.13	0.14
query41	0.09	0.04	0.04
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 98.1 s
Total hot run time: 26.82 s

jacktengg
jacktengg reviewed Jan 26, 2026
View reviewed changes
@dh-cloud
[fix](spill) Memory leak in the arena used by the aggregation source …
0245590 
…operator

After switching spill partition in partitioned aggregation source
operator, the agg_arena_pool also needs to be reset.
@dh-cloud dh-cloud force-pushed the fix_aggsource_spill_mem branch from c90958e to 0245590 Compare January 26, 2026 07:40
jacktengg
jacktengg approved these changes Jan 26, 2026
View reviewed changes
Copy link
Contributor

@jacktengg jacktengg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 26, 2026
@github-actions
Copy link
Contributor

github-actions bot commented Jan 26, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 26, 2026

PR approved by anyone and no changes requested.

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

BE UT Coverage Report

Increment line coverage 100.00% (9/9) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.50% (19152/36479)
Line Coverage 35.87% (177962/496149)
Region Coverage 32.29% (137359/425420)
Branch Coverage 33.25% (59512/178964)

@jacktengg
Copy link
Contributor

jacktengg commented Jan 26, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

TPC-H: Total hot run time: 32435 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0245590fe3b2986aef7ce1c899ea013a8bbc6c4d, data reload: false

------ Round 1 ----------------------------------
q1	17636	5342	5094	5094
q2	2000	305	194	194
q3	10225	1298	746	746
q4	10212	863	309	309
q5	7533	2162	1902	1902
q6	198	184	149	149
q7	892	734	588	588
q8	9276	1404	1147	1147
q9	5241	4746	4783	4746
q10	6824	1951	1574	1574
q11	520	282	274	274
q12	332	374	234	234
q13	17783	4022	3200	3200
q14	227	233	222	222
q15	904	830	810	810
q16	682	672	621	621
q17	628	754	509	509
q18	6766	6516	6279	6279
q19	1236	989	632	632
q20	402	339	236	236
q21	2672	2023	1945	1945
q22	1051	1049	1024	1024
Total cold run time: 103240 ms
Total hot run time: 32435 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5260	5188	5250	5188
q2	273	334	250	250
q3	2222	2695	2319	2319
q4	1372	1778	1326	1326
q5	4251	4236	4304	4236
q6	279	260	183	183
q7	2087	2146	1835	1835
q8	2554	2376	2470	2376
q9	7401	7502	7492	7492
q10	2754	2962	2729	2729
q11	570	473	472	472
q12	691	730	613	613
q13	3893	4415	3409	3409
q14	314	319	305	305
q15	851	811	811	811
q16	663	738	701	701
q17	1148	1379	1361	1361
q18	8390	7974	7762	7762
q19	909	814	855	814
q20	2031	2047	1896	1896
q21	4518	4281	4190	4190
q22	1076	1036	983	983
Total cold run time: 53507 ms
Total hot run time: 51251 ms

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

ClickBench: Total hot run time: 28.17 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0245590fe3b2986aef7ce1c899ea013a8bbc6c4d, data reload: false

query1	0.05	0.05	0.05
query2	0.09	0.04	0.04
query3	0.25	0.08	0.08
query4	1.60	0.11	0.11
query5	0.27	0.26	0.24
query6	1.17	0.68	0.66
query7	0.03	0.02	0.02
query8	0.05	0.03	0.05
query9	0.56	0.50	0.49
query10	0.54	0.54	0.54
query11	0.14	0.09	0.10
query12	0.14	0.10	0.10
query13	0.62	0.61	0.60
query14	1.05	1.07	1.05
query15	0.87	0.86	0.84
query16	0.39	0.39	0.40
query17	1.17	1.06	1.12
query18	0.23	0.20	0.20
query19	2.02	1.94	1.99
query20	0.02	0.02	0.01
query21	15.42	0.26	0.14
query22	5.17	0.05	0.04
query23	15.92	0.27	0.10
query24	1.53	0.36	0.29
query25	0.10	0.06	0.07
query26	0.15	0.13	0.13
query27	0.07	0.10	0.06
query28	3.35	1.17	0.96
query29	12.57	3.94	3.21
query30	0.27	0.12	0.11
query31	2.81	0.63	0.41
query32	3.23	0.60	0.49
query33	3.28	3.30	3.25
query34	16.06	5.40	4.75
query35	4.81	4.82	4.75
query36	0.65	0.49	0.49
query37	0.11	0.07	0.06
query38	0.08	0.04	0.04
query39	0.05	0.03	0.03
query40	0.18	0.16	0.16
query41	0.08	0.03	0.03
query42	0.05	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 97.25 s
Total hot run time: 28.17 s

@doris-robot
Copy link

doris-robot commented Jan 26, 2026

BE UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.47% (19139/36479)
Line Coverage 35.85% (177844/496147)
Region Coverage 32.29% (137361/425414)
Branch Coverage 33.23% (59473/178961)

@yiguolei
Copy link
Contributor

yiguolei commented Jan 26, 2026

skip check_coverage

@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 26, 2026

BE Regression && UT Coverage Report

Increment line coverage 100.00% (8/8) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.51% (25568/35756)
Line Coverage 54.11% (268172/495600)
Region Coverage 51.64% (222008/429919)
Branch Coverage 53.14% (95501/179729)

@yiguolei yiguolei merged commit 96a3a1a into apache:master Jan 26, 2026
29 of 32 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 26, 2026
@dh-cloud
[fix](spill) Memory leak in the arena used by the aggregation source … (
35d5944 
#60205)

…operator

After switching spill partition in partitioned aggregation source
operator, the agg_arena_pool also needs to be reset.

### What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

### Release note

None

### Check List (For Author)

- Test <!-- At least one of them must be included. -->
    - [ ] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No code files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:
    - [ ] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?
    - [ ] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
@github-actions github-actions bot mentioned this pull request Jan 26, 2026
Merged
yiguolei pushed a commit that referenced this pull request Jan 27, 2026
@github-actions @dh-cloud
branch-4.0: [fix](spill) Memory leak in the arena used by the aggrega…
2ae026d 
…tion source … #60205 (#60246)

Cherry-picked from #60205

Co-authored-by: dh-cloud <60729713+dh-cloud@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiguolei yiguolei yiguolei approved these changes

@jacktengg jacktengg jacktengg approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@dh-cloud @Thearas @yiguolei @doris-robot @jacktengg @hello-stephen