Merge internal - #43
Merged
Merged
Conversation
* Update documentation * Restore docs navigation entries Co-authored-by: ghanse <163584195+ghanse@users.noreply.github.com> * Fix docs guide navigation links Co-authored-by: ghanse <163584195+ghanse@users.noreply.github.com> --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* Vendor workspace artifacts into prepared bundles Lets the prepare phase download notebook source (and existing JAR/Python support) from a Databricks workspace so the generated DAB is self-contained and deployable across environments without depending on the source workspace holding the original notebook tree. * Opt-in toggle in `workspace_downloader` (`enable_workspace_downloads()` / `workspace_downloads_enabled()`) so library callers preserve current behavior; the CLI flips it on by default. * Pre-flight auth gate: `auth_available()` checks `.databrickscfg`, `DATABRICKS_CONFIG_PROFILE`, and `DATABRICKS_HOST`+`DATABRICKS_TOKEN`. `prompt_for_auth_if_missing()` surfaces an interactive prompt with explicit `databricks auth login` instructions before the prepare pass runs. * `dab_writer` walks the report ahead of time, collects workspace-resident paths from nested control-flow activities, prompts (and aborts on `n`), then enables downloads. New CLI flags: `--profile`, `--no-vendor-workspace-files`. * `NotebookActivity` preparer downloads the workspace source on hit, vendors to `src/notebooks/<basename>.py`, rewrites `notebook_task.notebook_path` to the bundle-relative path, and binds `job_cluster_key=default_cluster` so the downloaded code keeps classic-compute parity (the post-process bind step skips `../src/` paths reserved for orchestra-generated serverless notebooks). * `workspace_notebook_filename()` preserves the workspace basename verbatim (case, underscores, digits); falls back to `notebook_filename()` only when the path yields no usable segment. Strips `.py`, folds other extensions into the stem to avoid collisions between e.g. `runner.sql` and `runner.py`. * `write_notebooks` now coalesces identical writes, disambiguates content collisions with a `__N` suffix, and logs a warning so users know two workspace paths share a basename rather than silently overwriting one. * SKILL.md documents the new behavior, the auth prompt, and the two flags. * Tests: new `test_naming.py`, `test_notebook_writer.py`; updated `test_preparers.py` and `test_workspace_downloader.py` to cover the toggle, auth helpers, vendor-success branch, in-place fallback, basename preservation, and the verbatim test_notebook_001 case. Co-authored-by: Isaac * Format
* Add user-interface for specifying options * Fix docs rendering
This reverts commit 3c9cb71.
Improves conversion of control flow, expression-based parameters, and schedule triggers. Co-authored-by: Isaac
# Conflicts: # .gitignore # README.md
Flowx release
* Lakeflow connect ingestion pipeline handling * Add required connection parameters
* Revert "Update GitHub actions (#10)" This reverts commit 3c9cb71. * Improve expression resolution, control flow parsing, and setup (#11) * Refactor expression parsing (#1) * Improve control flow conversion (#2) * Update repo structure * Format modules * Initial commit * Initial release * Improve preparer coverage for managed ingestion pipelines (#2) * Lakeflow connect ingestion pipeline handling * Add required connection parameters * Add marketplace file (#3) --------- Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com> Co-authored-by: service-jira-pub-repo-auto <service.jira-pub-repo-auto@databricks.com>
* Refactor expression parsing (#1) * Improve control flow conversion (#2) Improves conversion of control flow, expression-based parameters, and schedule triggers. Co-authored-by: Isaac * docs: update README install and usage for marketplace + setup Updates README.md to align with new installation and usage patterns. Co-authored-by: Isaac * Remove legacy files
* Refactor expression parsing (#1) * Improve control flow conversion (#2) Improves conversion of control flow, expression-based parameters, and schedule triggers. Co-authored-by: Isaac * Merge external (#3) * Revert "Update GitHub actions (#10)" This reverts commit 3c9cb71. * Improve expression resolution, control flow parsing, and setup (#11) * Refactor expression parsing (#1) * Improve control flow conversion (#2) * Update repo structure * Format modules * Initial commit * Initial release * Improve preparer coverage for managed ingestion pipelines (#2) * Lakeflow connect ingestion pipeline handling * Add required connection parameters * Add marketplace file (#3) --------- Co-authored-by: Greg Hansen <163584195+ghanse@users.noreply.github.com> Co-authored-by: service-jira-pub-repo-auto <service.jira-pub-repo-auto@databricks.com> * Add SDK-based installation --------- Co-authored-by: service-jira-pub-repo-auto <service.jira-pub-repo-auto@databricks.com>
## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> This PR fixes issue templates for flowx. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> N/A ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [ ] added integration tests
## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> This PR adds dependabot and codecov configuration and updates CI to improve security. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> N/A ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [ ] added integration tests
CI ran unit tests only, so integration regressions reached reviewers. Run make integration in the build job, deselect the live-Azure suite that needs az login plus factory access, and xfail the six pre-existing ADF translation gaps non-strictly so they report XPASS once fixed.
Detect Airflow 3 (airflow.sdk, providers.standard) and strong 1.10 imports statically. Lower Asset/Dataset schedules to ANY_UPDATED/ALL_UPDATED table triggers when each asset declares a Databricks table, and fail closed on AssetOrTimeSchedule, mixed boolean expressions, unmapped assets, and ambiguous 1.10 schedule defaults. Route native async @task callables to linked leaf gaps. Reject dynamic Airflow imports with literal module names or executed source, and reserve flowx and Databricks task-schema notebook parameter keys.
…e-spike # Conflicts: # docs/content/docs/guide.mdx
## Changes Add a `"pipelines"` branch to `_load_report`, mirroring the adapter's `_load_pipelines`: filter each entry with `isinstance(p, dict) and "tasks" in p and "name" in p` and route it through the existing `_pipeline_dict_to_workflow` (the same machinery the single-pipeline branch uses). Strictly additive — the single-pipeline and `translations` branches are untouched, so existing reports behave exactly as before. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> Resolves #5 ### Tests Reproduced the abort with a multi-pipeline `{"pipelines":[...]}` report through the real `package` CLI; after the fix, `package` writes one bundle per pipeline (exit 0). Added `test_load_report_handles_pipelines_format` (`tests/unit/test_bundler.py`): a 2-pipeline report yields 2 workflows. - [x] manually tested - [x] added unit tests - [ ] added integration tests --------- Co-authored-by: Greg Hansen <gregory.hansen@databricks.com>
## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> This PR updates the flowx documentation to improve readability and update the installation, user guide, and configuration pages. It adds typed code blocks and modifies the site theme. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> N/A ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [ ] added integration tests
## Changes <!-- Summary of your changes that are easy to understand. Add screenshots when necessary --> This PR fixes broken documentation hyperlinks and makes some minor changes to the docs. ### Linked issues <!-- DOC: Link issue with a keyword: close, closes, closed, fix, fixes, fixed, resolve, resolves, resolved. See https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword --> N/A ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [x] manually tested - [ ] added unit tests - [ ] added integration tests
#31) ## Changes The output (including the packaged DAB) goes to the `flowx_output` directory. This dir is in .gitignore. DAB deploys will omit directories that are listed in .gitignore, so deploying ends up not uploading any Notebooks, etc. The change explicitly specifies the set of files that should be included in the deploy. ### Tests <!-- How is this tested? Please see the checklist below and also describe any other relevant tests --> - [ ] manually tested - [x] added unit tests - [ ] added integration tests
Reconcile the package pre-flight/reconciliation gate with main's multi-pipeline report fix (#6) and sync.include emission (#3): union the new prereq categories, thread skipped_pipelines through the real bundle writes, and keep _load_report's tuple return while folding in excluded-DAG filtering. Malformed report entries remain fail-closed.
…35) ## Changes Don't use YAML anchors but build a fresh `depends_on` dict per branch root. The IfCondition/Switch branch preparer (`inject_outcome_dependency`) built one `{"task_key", "outcome"}` dict and prepended that same object onto every branch-root task's `depends_on`. When a branch fans out to ≥2 root tasks, the shared object makes PyYAML serialize a YAML anchor/alias (`&id001 … *id001`) into the generated bundle YAML. The anchor is benign today — valid YAML that round-trips to identical content, passes `databricks bundle validate`, and deploys. But it's undesirable in generated config, and a stricter package pre-flight (e.g. the reconciliation-gating work in #13) rejects a `yaml_anchor` as an invariant violation. This builds a fresh dict per task, so no object is shared and no anchor is emitted. The edit is in `if_condition.py` because Switch routes its branch gating through the same `inject_outcome_dependency` helper — so one change covers both control-flow types. Semantically identical: both branch roots still gate on the condition outcome. ### Linked issues Resolves #34 ### Tests Rendered a test corpus of 363 real ADF pipelines with 0 YAML anchors after the change (was ~30 before). Behavior-preserving: a before/after render of the corpus differs only in the anchor lines — a canonical `yaml.safe_load` comparison of all 76 resource files shows 0 semantic differences. - [x] manually tested - [x] added unit tests - [ ] added integration tests
## Summary
Adds Apache Airflow as a production-oriented migration source alongside
ADF. Airflow DAG files are parsed statically with Python ASTs—flowx
never imports Airflow or executes customer DAG code—and are translated
into the shared `Pipeline` IR and packaged as one Declarative Automation
Bundle containing one Databricks Job per included DAG.
The Airflow path is fail-closed: every construct inside the supported
static DAG surface must be captured and translated, represented by a
linked failing placeholder, explicitly excluded, or reported as a
reconciliation failure before bundle files are written.
## Airflow source
- Supports assigned, context-manager, decorated, factory-generated, and
multiple DAGs per file; canonical and aliased imports; bounded literal
loops; narrow statically bindable helper factories; collision-safe task
identities; TaskGroups; TaskFlow; `chain()`/`cross_downstream()`; and
classic dependency syntax.
- Recognizes modern Airflow 2, Airflow 3
`airflow.sdk`/`airflow.providers.standard`, and strong Airflow 1.10
authoring syntax. Airflow 3 Asset/Dataset schedules lower to
`ANY_UPDATED`/`ALL_UPDATED` table triggers only when the Unity Catalog
table mapping is explicit; ambiguous schedules, mixed Asset expressions,
`AssetOrTimeSchedule`, and ambiguous Airflow 1.10 scheduling defaults
become source-semantic gaps.
- Translates Python, Bash/SSH, Spark submit, Databricks provider, SQL,
dbt/Cosmos, sensor, cross-DAG, TaskFlow, dynamic mapping, retry/timeout,
parameter, template, and trigger-rule semantics across the supported
static subset.
- Emits one shared bundle for all included DAGs so cross-DAG job
references remain resolvable. `--exclude-dag` keeps excluded DAGs
visible in audit and coverage while preventing Job emission.
- Supports static and PyDABs dbt factory modes, with dbt chain ordering
and package-time bundle invariants preserved.
## Reconciliation and packaging safety
- Uses stable DAG/task/edge capture identities, claim-based DAG-body
classification, source audit candidates, and a transformation ledger to
prevent silent task, edge, setting, argument, or mapping loss.
- Reports `verified`, `verified_with_gaps`, `failed`, and `excluded`;
`failed` exits nonzero and package preflight rejects the report before
creating, pruning, or overwriting destination bundle files.
- Unsupported leaf semantics become linked placeholder notebooks that
raise `NotImplementedError`; DAG-level schedule and graph semantics
become explicit `AirflowSourceSemantics` gaps.
- Coverage is based on audited candidates rather than emitted tasks and
separately reports deterministic coverage, translation-path coverage,
failed/excluded counts, reconciliation status, unresolved agentic
outcomes, and mechanically validated code-attached coverage.
- Package preflight also validates duplicate/dangling task keys,
dependency cycles, cross-DAG references, and the complete generated
bundle in a temporary directory before destination writes.
## Reviewed agentic gap workflow
- Replaces the unsafe Airflow `merge_agentic` path with
fingerprint-bound `resolve-agentic prepare`, `stage`, and `apply`
operations. The deterministic translation report remains immutable;
every apply rebuilds from that baseline.
- Flowx owns parsing, task identity, graph structure, schedules, task
policy, IR, and packaging. The provider can return only a V1
one-gap/one-leaf payload and cannot express task keys, dependencies,
retries, timeouts, clusters, schedules, or graph mutations.
- Staged candidates are schema-, source-hash-, provider-version-,
argument-disposition-, import-hygiene-, parameter-namespace-, and
task-invariance-validated. Apply rechecks the full graph/policy
invariants and writes a separate agentic report plus durable review
evidence.
- Supports selective acceptance, hash-bound `--accept-all`, explicit
accept-none review via `--review-complete`, and `--reset`;
`needs_input`, `deferred`, declined, and unreviewed gaps remain failing
placeholders.
- Vendors and pins the `airflow-to-dabs` Flowx provider profile and
fixtures from upstream release v0.2.1. Provider absence degrades safely
to unresolved placeholders; incompatible or tampered provider content
fails closed.
- The existing ADF agentic workflow and ADF migration semantics are not
changed by the Airflow resolver contract.
## Source routing, reporting, and CI
- Makes `--source {adf,airflow}` explicit for discover, convert, and
migrate across the CLI, skills, adapter, MCP server, bootstrap examples,
and application documentation; package remains source-independent.
- Extends inventory, Unity Catalog result rows, and the AI/BI dashboard
with audited, deterministic, agentic, failed, excluded, reconciliation,
review-outcome, and code-attached coverage fields.
- Runs the non-live integration suite in CI. Live Azure integration
remains a separate target, and known pre-existing ADF translation gaps
are non-strict xfails so an eventual fix surfaces as XPASS.
## Validation
- Airflow-focused unit suites: 301 passed.
- Broader offline unit validation: 1,101 passed; 3 live-Databricks
preparer tests were deselected because they require workspace
DNS/authentication.
- `mypy src/flowx/`: clean across 101 source files.
- Ruff checks and formatting pass for all files changed by the final
Airflow compatibility and agentic-contract work.
## Deliberate V1 limits
- Static analysis only: dynamic Python that cannot be proven safe is
rejected or represented as a gap rather than executed.
- Agentic resolution is leaf-only. GraphPatch, branching rewrites,
`@task_group` expansion, and multi-task replacement remain explicit
placeholders until a future graph-aware contract can preserve
reconciliation guarantees.
- Mechanically validated code attachment is not semantic certification;
accepted provider output remains visibly agentic and requires review.
Adopt reconciled behavior from the internal branch: gate the global-parameter-resolution option to ADF, point IR serialization at flowx.ir_serde, and trim CI to the public unit-test suite. The lockfile-normalization step rewrites any index URL to the public package index without naming an internal host. Co-authored-by: Matthew Moorcroft <matthew.moorcroft@databricks.com>
filter-repo rewrote all SHAs during sanitization, detaching this branch from main's history. This merge re-establishes main as an ancestor so the sanitized reconciliation can open as a PR. The branch tree is authoritative (-s ours); main contributes no content changes.
There was a problem hiding this comment.
🟡 Changes recommended
The review identified a correctness issue in the new dependency-cycle invariant logic and a report-serialization pattern that can mask data-shape bugs.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR merges a large set of internal updates into flowx, primarily adding an Airflow migration source alongside ADF, expanding IR/reporting coverage (including schema evolution + dashboard updates), and tightening bundle/package validation and documentation to reflect the new multi-source architecture.
Changes:
- Add a
flowx.sourcesregistry and implement Airflow discover/convert routing, fixtures, and golden/integration tests for Airflow conversion. - Extend IR + preparers to support generated notebook/script sources, SQL warehouse tasks, dbt-factory activities, and propagate job-level metadata (timeouts/notifications/tags) where applicable.
- Expand coverage/results reporting (new metrics + dashboard queries) and strengthen bundle invariants + SETUP.md prereqs rendering; update CI, docs, and skills to match the new
--sourcerequirement.
File summaries
| File | Description |
|---|---|
| tests/unit/test_web_body_and_param_defaults.py | Update imports to new ADF source module structure |
| tests/unit/test_until_agentic_handler.py | Update imports to new ADF source module structure |
| tests/unit/test_sql_task_and_table_trigger.py | Add unit tests for SqlActivity and table_update triggers |
| tests/unit/test_source_router.py | Add unit tests for adapter --source routing and registry behavior |
| tests/unit/test_resolve_field.py | Update imports to new ADF translator module path |
| tests/unit/test_reporting_results.py | Add tests for results schema evolution + new coverage columns |
| tests/unit/test_reporting_dashboard.py | Add dashboard validation using sqlglot + new metrics assertions |
| tests/unit/test_reporting_coverage.py | Add tests for audited counts + agentic outcomes/fingerprints |
| tests/unit/test_query_analysis.py | Update imports to new ADF query analysis module path |
| tests/unit/test_profile_report.py | Update imports to new ADF loader module path |
| tests/unit/test_prereqs_writer.py | Add tests for skipped pipelines surfacing in SETUP.md |
| tests/unit/test_preparers.py | Update IR serde usage + _load_report return shape |
| tests/unit/test_param_dedup.py | Add Airflow policy round-trip tests + malformed report rejection |
| tests/unit/test_merge_agentic.py | Update merge entrypoint import to IR serde |
| tests/unit/test_mcp_migrate.py | Require source in migrate MCP command payloads |
| tests/unit/test_ir_rewriter.py | Update import to new ADF IR rewriter module path |
| tests/unit/test_code_generator.py | Replace internal example hostname with neutral example |
| tests/unit/test_bundle_invariants.py | Add unit tests for new invariants (empty job, cycles, run_job refs) |
| tests/unit/test_airflow_provider_sync.py | Add tests for vendored provider pin/sync + runtime pin validation |
| tests/unit/test_airflow_adapter_reporting.py | Add tests for source-aware prompts + Airflow coverage columns |
| tests/unit/test_adf_loader.py | Update imports + rename “orchestra” wording to “flowx” |
| tests/resources/json/pipelines/pl_test_webactivity_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_wait_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_switch_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_sparkpython_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_sparkjar_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_setvariable_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_notebook_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_lookup_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_ifcondition_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_foreach_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_filter_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_executepipeline_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_delete_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_copy_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/json/pipelines/pl_test_appendvariable_coverage.json | Scrub internal subscription/resource identifiers in fixture |
| tests/resources/airflow/review_repros/t9_triggerrule.py | Add Airflow repro fixture DAG (trigger_rule) |
| tests/resources/airflow/review_repros/t8_helperfn.py | Add Airflow repro fixture DAG (helper fn operator creation) |
| tests/resources/airflow/review_repros/t7_subclass.py | Add Airflow repro fixture DAG (operator subclass) |
| tests/resources/airflow/review_repros/t6_chain.py | Add Airflow repro fixture DAG (chain/cross_downstream) |
| tests/resources/airflow/review_repros/t5_alias.py | Add Airflow repro fixture DAG (import alias) |
| tests/resources/airflow/review_repros/t4_bashjinja.py | Add Airflow repro fixture DAG (bash jinja templates) |
| tests/resources/airflow/review_repros/t32_multiassigned.py | Add Airflow repro fixture DAG (multiple DAGs in file) |
| tests/resources/airflow/review_repros/t31_inject.py | Add Airflow repro fixture DAG (command injection marker) |
| tests/resources/airflow/review_repros/t30_dagvar2.py | Add Airflow repro fixture DAG (dag assigned var + args) |
| tests/resources/airflow/review_repros/t3_collide.py | Add Airflow repro fixture DAG (task_id collisions) |
| tests/resources/airflow/review_repros/t29_dagsem.py | Add Airflow repro fixture DAG (dag semantics fields) |
| tests/resources/airflow/review_repros/t28_nodash.py | Add Airflow repro fixture DAG (ds_nodash templating) |
| tests/resources/airflow/review_repros/t27_ss.py | Add Airflow repro fixture DAG (SparkSubmitOperator) |
| tests/resources/airflow/review_repros/t26_loopedge.py | Add Airflow repro fixture DAG (loop edges) |
| tests/resources/airflow/review_repros/t25_tr3.py | Add Airflow repro fixture DAG (trigger rule strings) |
| tests/resources/airflow/review_repros/t24_sensorscope.py | Add Airflow repro fixture DAG (sensor scoping) |
| tests/resources/airflow/review_repros/t23_tr2.py | Add Airflow repro fixture DAG (TriggerRule enum) |
| tests/resources/airflow/review_repros/t22_expandbash.py | Add Airflow repro fixture DAG (partial/expand mapping) |
| tests/resources/airflow/review_repros/t21_partialexpand.py | Add Airflow repro fixture DAG (partial expand) |
| tests/resources/airflow/review_repros/t20_sqlesc.py | Add Airflow repro fixture DAG (SQL operator quoting/templates) |
| tests/resources/airflow/review_repros/t2_sparksubmit.py | Add Airflow repro fixture DAG (spark-submit bash patterns) |
| tests/resources/airflow/review_repros/t19_fncollide.py | Add Airflow repro fixture DAG (callable name collision) |
| tests/resources/airflow/review_repros/t18_xcompush.py | Add Airflow repro fixture DAG (callable deps/constants) |
| tests/resources/airflow/review_repros/t17_taskflow.py | Add Airflow repro fixture DAG (TaskFlow API) |
| tests/resources/airflow/review_repros/t16_sensor.py | Add Airflow repro fixture DAG (sensor timeout/poke) |
| tests/resources/airflow/review_repros/t15_magic.py | Add Airflow repro fixture DAG (magic blocks in bash) |
| tests/resources/airflow/review_repros/t14_retries.py | Add Airflow repro fixture DAG (retries/retry_delay) |
| tests/resources/airflow/review_repros/t13_sqlescape.py | Add Airflow repro fixture DAG (SQL escaping/templates) |
| tests/resources/airflow/review_repros/t12_globals.py | Add Airflow repro fixture DAG (globals registration) |
| tests/resources/airflow/review_repros/t11_dagvar.py | Add Airflow repro fixture DAG (dag var assigned) |
| tests/resources/airflow/review_repros/t10_loopliteral.py | Add Airflow repro fixture DAG (list append loop) |
| tests/resources/airflow/review_repros/t1_loop.py | Add Airflow repro fixture DAG (sequential loop deps) |
| tests/resources/airflow/review_repros/a8_classic_mapping.py | Add Airflow audit fixture DAG (mapping) |
| tests/resources/airflow/review_repros/a2_task_key_collision.py | Add Airflow audit fixture DAG (collision resolution) |
| tests/resources/airflow/review_repros/a1_assigned_dag.py | Add Airflow audit fixture DAG (classic DAG config) |
| tests/resources/airflow/orders_analytics_dag.py | Add sample Airflow DAG fixture used by router tests |
| tests/resources/airflow/golden_pipeline_dag.py | Add representative Airflow DAG for golden-bundle integration test |
| tests/integration/test_path_equivalence.py | Update integration to new ADF loader/translate + IR serde |
| tests/integration/test_golden_output.py | Mark pre-existing ADF gaps as xfail (non-strict) |
| tests/integration/test_end_to_end.py | Mark pre-existing ADF gaps as xfail (non-strict) |
| tests/integration/test_airflow_golden_bundle.py | Add end-to-end Airflow golden bundle test |
| tests/integration/test_adf_live.py | Update imports + scrub internal live-test subscription identifiers |
| tests/conftest.py | Update test fixture imports to new ADF loader path |
| src/flowx/validate/bundle_invariants.py | Add dependency cycle check + dangling run_job reference validation |
| src/flowx/sources/airflow/convert.py | Implement Airflow convert phase writing shared translation report |
| src/flowx/sources/airflow/init.py | Add Airflow source package marker |
| src/flowx/sources/adf/translators/web_activity.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/wait.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/switch.py | Repoint loader/resolver imports under sources/adf |
| src/flowx/sources/adf/translators/spark_python.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/spark_jar.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/set_variable.py | Add SetVariable translator with return-value unwrapping logic |
| src/flowx/sources/adf/translators/resolve.py | Add shared resolver helpers (bridge lowering, dict/int helpers) |
| src/flowx/sources/adf/translators/notebook.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/lookup.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/if_condition.py | Repoint bridge helpers under sources/adf |
| src/flowx/sources/adf/translators/for_each.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/filter.py | Add Filter translator with safe lowering of predicate code |
| src/flowx/sources/adf/translators/execute_pipeline.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/delete.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/databricks_job.py | Repoint shared resolver imports under sources/adf |
| src/flowx/sources/adf/translators/copy.py | Repoint query analysis imports under sources/adf |
| src/flowx/sources/adf/translators/append_variable.py | Add AppendVariable translator with context threading |
| src/flowx/sources/adf/translators/init.py | Add ADF translators package marker |
| src/flowx/sources/adf/init.py | Add ADF source package marker |
| src/flowx/sources/init.py | Add source registry + phase module routing definitions |
| src/flowx/reporting/results.py | Add new result columns + schema evolution via SHOW COLUMNS/ALTER TABLE |
| src/flowx/preparer/workflow_preparer.py | Add Sql/dbt preparers + propagate Airflow job metadata + run_if passthrough |
| src/flowx/preparer/activity_preparers/sql.py | Add SqlActivity preparer writing extracted .sql file for sql_task |
| src/flowx/preparer/activity_preparers/spark_python.py | Prefer generated source over workspace download when present |
| src/flowx/preparer/activity_preparers/notebook.py | Prefer generated notebook source over workspace download/placeholder |
| src/flowx/preparer/activity_preparers/if_condition.py | Avoid YAML anchors by not reusing a shared depends_on dict |
| src/flowx/motifs/collapser.py | Preserve new pipeline metadata fields when collapsing motifs |
| src/flowx/models/ir.py | Add new IR types/fields (generated_source, SqlActivity, DbtFactoryActivity, audit metadata) |
| src/flowx/dbt/init.py | Add dbt support package marker |
| src/flowx/bundler/prereqs_writer.py | Extend SETUP.md prereqs for dbt hooks, Airflow backfills, skipped pipelines |
| src/flowx/adapter/constants.py | Add Airflow source-path input constant |
| skills/flowx-setup/SKILL.md | Document --source usage in examples |
| skills/flowx-resolve-airflow-gaps/SKILL.md | Add skill doc for fingerprint-bound Airflow gap resolution workflow |
| skills/flowx-resolve-airflow-gaps/references/contract-v1.md | Add Airflow agentic gap contract reference |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/references/lakeflow-connect.md | Add vendored provider knowledge reference |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/provider.json | Add vendored provider manifest + pin metadata |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/PROFILE.md | Add vendored provider profile |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/resolution-sql.json | Add vendored contract fixture (resolved SQL) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/resolution-spark-python.json | Add vendored contract fixture (resolved spark python) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/resolution-notebook.json | Add vendored contract fixture (resolved notebook) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/resolution-needs-input.json | Add vendored contract fixture (needs_input) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/resolution-deferred.json | Add vendored contract fixture (deferred) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/gap-sql.json | Add vendored contract fixture (gap SQL envelope) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/gap-spark-python.json | Add vendored contract fixture (gap spark python envelope) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/gap-notebook.json | Add vendored contract fixture (gap notebook envelope) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/gap-needs-input.json | Add vendored contract fixture (gap needs_input envelope) |
| skills/flowx-resolve-airflow-gaps/references/airflow-to-dabs/providers/flowx-gap-resolver/fixtures/gap-deferred.json | Add vendored contract fixture (gap deferred envelope) |
| skills/flowx-package/SKILL.md | Update examples to include source where required |
| skills/flowx-migrate/SKILL.md | Update migrate workflow docs for multi-source (--source) |
| skills/flowx-migrate/references/workflow.md | Update architecture text for new source module structure |
| skills/flowx-discover/sources/airflow.md | Add Airflow discover source guide |
| skills/flowx-discover/sources/adf.md | Add ADF discover source guide |
| skills/flowx-convert/sources/airflow.md | Add Airflow convert source guide |
| scripts/bootstrap.sh | Update examples to include --source |
| pyproject.toml | Add ruff per-file ignores for Airflow fixtures + exclude review repros |
| Makefile | Split integration target into filtered vs live-friendly variants |
| docs/content/docs/installation.mdx | Update troubleshooting guidance + --source example |
| docs/content/docs/guide.mdx | Document results recording/dashboard + Airflow coverage semantics |
| docs/content/docs/configuration.mdx | Document expanded results metrics + dashboard widgets |
| docs/content/docs/architecture.mdx | Update architecture docs for resolve_agentic + docs link fix |
| app/README.md | Update MCP command docs for multi-source + resolve_agentic |
| AGENTS.md | Update local invocation examples to include --source |
| .github/workflows/push.yml | Normalize uv.lock public index URLs more generally |
Review details
- Files reviewed: 148/189 changed files
- Comments generated: 2
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+83
to
+87
| pipeline_dicts = [pipeline_to_dict(pipeline) for pipeline in pipelines] | ||
| payload = pipeline_dicts[0] if len(pipeline_dicts) == 1 else {"pipelines": pipeline_dicts} | ||
| report_file = work_dir / "translation_report.json" | ||
| report_file.write_text(json.dumps(payload, indent=2, default=str), encoding="utf-8") | ||
|
|
Comment on lines
+179
to
+183
| keys: list[str] = [ | ||
| task["task_key"] for task in tasks if isinstance(task, dict) and isinstance(task.get("task_key"), str) | ||
| ] | ||
| key_set = set(keys) | ||
| in_degree: dict[str, int] = {key: 0 for key in keys} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
This PR is based on #22 and merges internal changes into the flowx repo:
sync.includein generateddatabricks.yml(@alexnastetsky )Linked issues
Resolves #18
Tests