feat: Native Polars SQLContext data parser bypassing DuckDB and Pandas - #740
feat: Native Polars SQLContext data parser bypassing DuckDB and Pandas#740Sagar369r wants to merge 1 commit into
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 93c60684ea
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| def get_datas_by_sql(self, sql: str) -> List[Dict[str, Any]]: | ||
| """Execute SQL query using Polars native multi-threaded SQLContext.""" | ||
| ctx = pl.SQLContext(pygwalker_mid_table=self.df.lazy()) | ||
| result_df = ctx.execute(sql, eager=True) |
There was a problem hiding this comment.
For Polars inputs passed to get_metrics_datas, pygwalker_tools/metrics/api.py builds SQL with get_metrics_sql and then calls parser.get_datas_by_sql(sql) for all non-Connector datasets (api.py:110-119). Several built-in metrics emit DuckDB temporal calls such as datediff(...) (pygwalker_tools/metrics/core.py:86 and core.py:130), which used to work because this parser delegated to DuckDB; sending the same SQL directly to pl.SQLContext makes those metrics fail for Polars DataFrames. Please either keep a DuckDB fallback for unsupported SQL or translate these queries before execution.
Useful? React with 👍 / 👎.
|
|
||
| @property | ||
| def data_size(self) -> int: | ||
| return self.df.estimated_size() |
There was a problem hiding this comment.
DataBridge compares data_parser.data_size with JUPYTER_BYTE_LIMIT to decide whether to keep queries in the kernel or send to_records(None) to the browser (pygwalker/services/data_bridge.py:36-38). estimated_size() reports Polars' in-memory buffer footprint, which can be much smaller than the JSON/record payload sent to the frontend, so large Polars datasets can now stay in browser mode and overload the notebook instead of switching to kernel mode as the previous JSON-sampling estimate did.
Useful? React with 👍 / 👎.
1 participant
Summary of Changes
This PR optimizes
PolarsDataFrameDataParser(pygwalker/data_parsers/polars_parser.py) to bypass all DuckDB table registrations (duckdb.register) and Pandas (pd.DataFrame) intermediate conversions when exploringpolars.DataFramedatasets.Key Technical Upgrades
SQLContextExecution: Overridesget_datas_by_sql(),get_datas_by_payload(), and batch query handlers to registerself.df.lazy()directly intopl.SQLContextand execute queries using multi-threaded Rust Arrow pipelines (ctx.execute(sql, eager=True)).field_metasanddata_size(self.df.estimated_size()) to inspect native Polars data types (pl.Datetime,pl.Int64,pl.Float64,pl.Utf8) instantly in memory without running sample SQL queries.@cached_property): Uses@cached_propertyonfield_metasto guarantee zero memory leaks across multi-dataset explorations (aligned with PR Fix memory leak by replacing @lru_cache with instance-level caching in data parsers #724).Test & Verification Proof
All upstream test suites pass without modification, confirming 100% backward compatibility and exact contract adherence:
tests/test_data_parsers.py): Alltest_data_parser_on_polarsand tabular input assertions pass (15 passed).tests/test_pygwalker_core.py,tests/test_computation.py): All 114 environment routing and computation mode tests pass (114 passed).