Skip to content

fix: avoid memory-heavy Qlib preprocessing reshape - #361

Open
ElhamDevelopmentStudio wants to merge 1 commit into
shiyu-coder:masterfrom
ElhamDevelopmentStudio:fix/qlib-preprocess-memory
Open

fix: avoid memory-heavy Qlib preprocessing reshape#361
ElhamDevelopmentStudio wants to merge 1 commit into
shiyu-coder:masterfrom
ElhamDevelopmentStudio:fix/qlib-preprocess-memory

Conversation

@ElhamDevelopmentStudio

Copy link
Copy Markdown
Contributor

Summary

  • Remove the global data_df.stack().unstack() operation from Qlib preprocessing.
  • Process data directly by the named instrument index level.
  • Preserve existing feature names, amount calculation, missing-value filtering, and minimum-history checks.
  • Add regression coverage for sparse observations, missing values, and both supported MultiIndex orders.

Problem

Preprocessing large Qlib datasets could stall at:

data_df.stack().unstack(level=1)

This operation can materialize a large, dense datetime/field/instrument intermediate, causing significant memory amplification for sparse datasets.

The updated implementation groups the original dataframe by instrument and processes one instrument at a time without constructing the global intermediate.

Validation

  • python -m pytest tests -q — 6 passed
  • Focused preprocessing regression — 2 passed
  • Python compilation check — passed
  • git diff --check — passed
  • Legacy and updated outputs compared across 500 instruments with sparse rows and missing values — exactly equal
  • Synthetic sparse benchmark reduced the intermediate representation from 1,500,000 cells to the original 60,000-cell dataset shape

Issue

Fixes #161

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

qlib_data_preprocess卡在data_df.stack().unstack()

1 participant