feat(up): pipeline release version constrains#187
Open
project-defiant wants to merge 10 commits intodevfrom
Open
feat(up): pipeline release version constrains#187project-defiant wants to merge 10 commits intodevfrom
project-defiant wants to merge 10 commits intodevfrom
Conversation
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
project-defiant
changed the title
Contributor
There was a problem hiding this comment.
Pull request overview
Introduces a single, validated run_name-driven versioning/output-location model for Unified Pipeline runs, replacing the prior dev_uri or release_uri pattern and removing the separate release_name field from the YAML config.
Changes:
- Add
PipelineRunConfig(Pydantic) to validaterun_nameand deriverelease_uri/release_name. - Update Unified Pipeline config + step URI builders to rely on a single
release_uri. - Add tests and expand documentation describing
run_name,is_dev, and derived labels.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_run_config.py | Adds unit tests for PipelineRunConfig validation and URI/name derivation. |
| src/orchestration/models/run_config.py | New Pydantic model implementing run_name validation and release_uri / release_name derivation. |
| src/orchestration/dags/config/unified_pipeline.py | Switches to PipelineRunConfig and consistently passes release_uri / derived release_name into templates. |
| src/orchestration/dags/config/unified_pipeline.yaml | Replaces release_name with documented run_name + is_dev behavior. |
| src/orchestration/utils/common.py | Centralizes GCS bucket root constants used for URI construction. |
| src/orchestration/models/step.py | Removes dev_uri or release_uri fallback; uses release_uri only. |
| src/orchestration/operators/differs/manifest_artifact_differ.py | Docstring update reflecting release_uri usage. |
| docs/unified_pipeline/config.md | Documents run_name format, is_dev behavior, and derived release_name. |
| docs/unified_pipeline/README.md | Updates release-run guidelines to include setting run_name. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+10
to
+20
| _CURRENT_YYMM = datetime.now().strftime("%y%m") | ||
|
|
||
|
|
||
| def _future_yymm() -> str: | ||
| now = datetime.now() | ||
| if now.month == 12: | ||
| return f"{(now.year + 1) % 100:02d}01" | ||
| return f"{now.year % 100:02d}{now.month + 1:02d}" | ||
|
|
||
|
|
||
| _FUTURE_YYMM = _future_yymm() |
| # N — revision number, starting from 1 (increment if re-running the same release) | ||
| # | ||
| # Examples: 'sz/platform-2605-1', 'abc/ppp-2606-2' | ||
| run_name: 'sz/platform-2605-1' |
Comment on lines
+41
to
+49
| yymm = int(match.group(3)) | ||
| current_yymm = int(datetime.now().strftime("%y%m")) | ||
| if yymm < current_yymm: | ||
| raise ValueError( | ||
| f"run_name date '{match.group(3)}' is in the past " | ||
| f"(current: {current_yymm:04d}). " | ||
| "Update run_name to the current or a future YYMM to avoid " | ||
| "overwriting an existing release." | ||
| ) |
Comment on lines
+18
to
+21
| | `flavor` | `platform` for a public Platform release, `ppp` for a Partner Preview release | `platform` | | ||
| | `YYMM` | Two-digit year + two-digit month of the release. Must be the current month or later — past dates are rejected to prevent overwriting existing releases. | `2605` | | ||
| | `N` | Revision number, starting from 1. Increment if re-running the same release. | `1` | | ||
|
|
Comment on lines
+5
to
+8
| # YYMM — two-digit year + two-digit month of the release (e.g. '2605' for May 2026) | ||
| # must be the current month or later, to prevent overwriting existing releases | ||
| # N — revision number, starting from 1 (increment if re-running the same release) | ||
| # |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Context
Adds
PipelineRunConfigPydantic model (src/orchestration/models/run_config.py) that validatesrun_nameagainst the pattern<prefix>/(platform|ppp)-YYMM-N(e.g.sz/platform-2605-1),rejects past dates, and rejects invalid month numbers (outside 01–12)
release_namefromunified_pipeline.yaml— all URI derivation and downstream version labelling is now driven by a singlerun_namefield; GCS bucket constants moved toutils/common.pydev_uri or release_uriconditional pattern with a singlerelease_uriproperty on bothPipelineRunConfigandUnifiedPipelineConfig;is_devflag remains explicit in the YAMLwith a clear description
release_uribehaviouris_dev: true(default) →gs://open-targets-pipeline-runs/<run_name>is_dev: false→gs://open-targets-pre-data-releases/<flavor>-<YYMM>A secondary label
release_name(<flavor>-<YYMM>) is derived fromrun_nameand passed to PTS asot_releaseand to Gentropy asl2g_training_version.Test Plan
uv run pytestpasses (72 tests, 1 expected xfail)run_nameto a past date (e.g.sz/platform-2603-1) —UnifiedPipelineConfiginstantiation raisesValidationErrorrun_nameto an invalid format (e.g.sz/platform-2605) — raisesValidationErrorrun_nameto an invalid month (e.g.sz/platform-2699-1) — raisesValidationErrorrun_namewithis_dev: true—release_uriresolves togs://open-targets-pipeline-runs/<run_name>is_dev: false—release_uriresolves togs://open-targets-pre-data-releases/<flavor>-<YYMM>Closes Enforce pipeline run version in orchestration issues#4326