Skip to content

feat: add configurable memory-limit for DuckDB engine#2059

Merged
ferenc-csaky merged 2 commits into
mainfrom
feat/duckdb-memory-limit
Jun 2, 2026
Merged

feat: add configurable memory-limit for DuckDB engine#2059
ferenc-csaky merged 2 commits into
mainfrom
feat/duckdb-memory-limit

Conversation

@WellMafra

Copy link
Copy Markdown
Contributor

Summary

  • Added memory-limit field to JdbcConfig.DuckDbConfig (JSON key memory-limit)
  • DuckDbExtensions.buildInitSql() now emits SET memory_limit='<value>' before loading extensions when the field is set
  • Fixed log statement: was logging null extensionDir value; now logs the env var name DUCKDB_EXTENSIONS_DIR

Usage

Configure in package.json server section:

"duckdb": {
  "url": "jdbc:duckdb:",
  "memory-limit": "20GB"
}

Without this setting, DuckDB defaults to ~80% of available system RAM — causing OOM kills when running alongside Flink in a shared pod.

Test plan

  • Deploy workbench query executor with memory-limit: "20GB" configured
  • Verify SELECT current_setting('memory_limit') in DuckDB returns "20.0 GiB"
  • Confirm TM pods no longer OOM crash loop under workbench query load

@codecov

codecov Bot commented May 11, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 0% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 13.92%. Comparing base (7e50529) to head (3cdf7e4).
⚠️ Report is 2 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...main/java/com/datasqrl/util/DuckDbInitializer.java 0.00% 9 Missing ⚠️
...a/com/datasqrl/graphql/jdbc/JdbcClientsConfig.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #2059      +/-   ##
============================================
- Coverage     13.92%   13.92%   -0.01%     
  Complexity      854      854              
============================================
  Files           604      604              
  Lines         17246    17251       +5     
  Branches       2085     2086       +1     
============================================
  Hits           2402     2402              
- Misses        14612    14617       +5     
  Partials        232      232              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

WellMafra and others added 2 commits June 1, 2026 17:20
DuckDB had no memory cap — by default it uses 80% of available RAM.
Add `memory-limit` field to `DuckDbConfig` and emit
`SET memory_limit='...'` as the first init SQL statement when configured.
The limit is applied even when the extensions directory is absent so
development environments also benefit.

Configure via package.json:
  "duckdb": { "url": "jdbc:duckdb:", "memory-limit": "20GB" }
@ferenc-csaky ferenc-csaky force-pushed the feat/duckdb-memory-limit branch from 4d82989 to 3cdf7e4 Compare June 1, 2026 15:37
@ferenc-csaky ferenc-csaky added this to the 0.10.6 milestone Jun 1, 2026
@ferenc-csaky ferenc-csaky added the enhancement New feature or request label Jun 1, 2026
@ferenc-csaky ferenc-csaky enabled auto-merge (squash) June 1, 2026 15:38
@ferenc-csaky ferenc-csaky merged commit c0dc92c into main Jun 2, 2026
20 of 22 checks passed
@ferenc-csaky ferenc-csaky deleted the feat/duckdb-memory-limit branch June 2, 2026 11:33
WellMafra added a commit that referenced this pull request Jun 12, 2026
Co-authored-by: Ferenc Csaky <ferenc@datasqrl.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants