Add a workflow to extract used tables from notebooks and queries - #4441

Open

pritishpai wants to merge 52 commits into

mainfrom

feat/scan_all_code_workspace_used_table

Commits on Oct 8, 2025

Add WorkspaceTablesLinter for extracting tables used in notebooks
pritishpai
committed
a55645c
Add function to get language from path
pritishpai
committed
780f309
Simple integration test
pritishpai
committed
84bb967
used_tables_for_workspace_crawler
pritishpai
committed
0fdfddb
Add used_tables_for_workspace crawler
pritishpai
committed
eb8332d
Add scan_workspace_for_tables entrypoint for WorkspaceTablesLinter and test
pritishpai
committed
5a6d857
Add _discover_workspace_objects and cleanup for integration tests
pritishpai
committed
f333e34
Add functions to extract tables from path, parallelize the process per path
pritishpai
committed
579d0fa
Logging
pritishpai
committed
bb02a5e
Add more logging for integration test
pritishpai
committed
e08e983
Fix the WorkspaceObjectInfo constructor call
pritishpai
committed
c702d74
Add funtions to extract tables from notebooks and files
pritishpai
committed
8451aa2
Remove unused imports
pritishpai
committed
774f57c
Upload notebook content properly
pritishpai
committed
7b73283
Add _extract_tables_from_notebook
pritishpai
committed
ff5f046
Succesfully crawl read tables
pritishpai
committed
8f28f97
Add verification of tables crawled are fetched from the inventory table
pritishpai
committed
8fbc9d5
Insert crawled tables in the inventory database
pritishpai
committed
53059a4
Comment
pritishpai
committed
df44d79
Use empty TableMigrationIndex
pritishpai
committed
2fde3b1
Add a function to detect if it is a dataframe
pritishpai
committed
165d6a4
Add some unit tests
pritishpai
committed
2ee7405
Fmt changes
pritishpai
committed
e0113aa
Fmt changes
pritishpai
committed
c304da9
Create common function _get_str_content_from_path
pritishpai
committed
20449fa
Remove generated unit tests for now
pritishpai
committed
a67ab22
Fmt
pritishpai
committed
302c743
Fmt changes and cleanup for integration test
pritishpai
committed
11892b2
Remove unused code
pritishpai
committed
0e07783
Fmt changes and refactoring
pritishpai
committed
bf2908b
Basic unit testing
pritishpai
committed
ae2442f
fix unit tests
pritishpai
committed
3334234
Add a new workflow
pritishpai
committed
468a132
Change workflow name
pritishpai
committed
a07e0ad
Add additional integration test
pritishpai
committed
af312ff
Change workflow name
pritishpai
committed
728584b
Add workflow to deploy list
pritishpai
committed
c6527e0
Strip '%sql' for proper linting
pritishpai
committed
3049d4d
Fmt
pritishpai
committed
f4c17d6
Fmt
pritishpai
committed
68694c1
Add a placeholder readme
pritishpai
committed
c8a30b5
Refactoring
pritishpai
committed
a4345ef
Refactor workflow name and code structure
pritishpai
committed
1bd17dd

Commits on Oct 9, 2025

Refactor class name
pritishpai
committed
9ae2e2d
Refactor test name
pritishpai
committed
763ec10
Add a cli command for workspace code scanner
pritishpai
committed
a95ccf0
Add cli command to labs.yml
pritishpai
committed
a5bb05a
fmt changes
pritishpai
committed
fd5266f

Commits on Oct 10, 2025

docs enhance
pritishpai
committed
0bc5cb0
refactoring and adding cli command to docs
pritishpai
committed
5e67645
fmt
pritishpai
committed
6dce482

Commits on Oct 13, 2025

Fix sample query for per workspace area results
pritishpai
committed
9a6dacb