feat(deps): unify pip deps via PEP 503 indexes + thin requirements#962
Open
feat(deps): unify pip deps via PEP 503 indexes + thin requirements#962
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
Unifies Python dependency resolution by switching from direct wheel URLs to package/version pins intended to be served via PEP 503 “simple” indexes, and wires Bazel pip parsing to use a shared index list across platforms.
Changes:
- Replaced many direct wheel URLs in per-platform
deps/requirements_*.txtwith pinned package names/versions. - Expanded
deps/pip.bzlPIP_EXTRA_ARGSto include the unified set of extra PEP 503 indexes (rtp-opensource simple + PyTorch per-ABI indexes + mirror). - Added a stable
WORKSPACErepository alias (rtp_opensource_deps) intended to keep a handle to opensourcedeps/whenrtp_depsis overridden internally.
Reviewed changes
Copilot reviewed 9 out of 9 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| deps/requirements_torch_gpu_cuda12_9.txt | Switches CUDA 12.9 GPU deps from direct URLs to pinned names/versions. |
| deps/requirements_torch_gpu_cuda12.txt | Switches CUDA 12.6 GPU deps from direct URLs to pinned names/versions. |
| deps/requirements_torch_cpu.txt | Updates CPU Torch pin and removes direct wheel URLs. |
| deps/requirements_rocm.txt | Switches ROCm deps to pinned names/versions and adds amd-smi. |
| deps/requirements_cuda12_arm.txt | Switches CUDA12 ARM deps from direct URLs to pinned names/versions. |
| deps/requirements_cpu_arm.txt | Replaces direct ARM CPU Torch wheel URL with a version pin. |
| deps/pip.bzl | Centralizes pip index configuration via a shared extra-index list. |
| deps/http.bzl | Adds TODO commentary about consolidating duplicated http_archive entries. |
| WORKSPACE | Adds rtp_opensource_deps local_repository alias for internal override scenarios. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| # Consolidate here as multi-URL lists (artlab mirror first, public URL fallback); | ||
| # requires per-config `bazel build` verification before landing to catch URL | ||
| # availability regressions. | ||
| # See /home/liukan.lk/.claude/plans/serialized-wibbling-snail.md Phase 5. |
Collaborator
Author
AI Code Review - PR #962Status: BLOCKING Summary: P0/0 · P1/3 · P2/4 · P3/0 Blocking IssuesP1
Non-blocking SuggestionsP2
Checklist Violations (5 fail / 23 total)General Principles Checklist
Strengths
|
Collaborator
Author
AI Code Review - PR #962Status: BLOCKING Summary: P0/0 · P1/1 · P2/0 · P3/1 Blocking IssuesP1
Non-blocking SuggestionsP3
Checklist Violations (1 fail / 23 total)General Principles Checklist
Strengths
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Bazel Python dependency flow to resolve wheels consistently across platforms by switching to a unified set of PEP 503 indexes (Aliyun PyPI mirror + download.pytorch.org per-accelerator + an OSS “simple/” index for custom wheels), and regenerates lockfiles accordingly.
Changes:
- Reworked
deps/pip.bzlto use a consolidated index list and simplified (thin) requirements inputs, with lockfiles regenerated byuv. - Removed CPU-only and ARM-CPU pip/lock flows and Bazel configs, and adjusted default dependency selection to fall through to CUDA 12.9.
- Regenerated CUDA12.9 / ROCm / CUDA12 ARM lockfiles to match the unified index configuration and new “thin requirements” approach.
Reviewed changes
Copilot reviewed 16 out of 19 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| deps/requirements_torch_gpu_cuda12_9.txt | Converts CUDA12.9 requirements from direct wheel URLs to pinned packages + PEP 503/simple URLs where needed. |
| deps/requirements_rocm.txt | Updates ROCm requirements to use unified indexes and thin inputs. |
| deps/requirements_cuda12_arm.txt | Updates CUDA12 ARM requirements to the new thin input style and unified indexes. |
| deps/requirements_lock_torch_gpu_cuda12_9.txt | Regenerated CUDA12.9 lockfile via uv, embedding unified index configuration. |
| deps/requirements_lock_rocm.txt | Regenerated ROCm lockfile via uv, embedding unified index configuration. |
| deps/requirements_lock_cuda12_arm.txt | Regenerated CUDA12 ARM lockfile via uv, embedding unified index configuration. |
| deps/pip.bzl | Centralizes pip index config and updates pip_parse repos to the new lockfiles. |
| deps/BUILD | Removes deprecated compile targets and keeps only CUDA12.9 / ROCm / CUDA12 ARM compile targets. |
| deps/http.bzl | Removes CPU torch archives and adds a TODO about consolidating duplicate http_archives across open/internal. |
| arch_config/arch_select.bzl | Drops CPU/ARM-CPU branches and changes default selection to CUDA12.9. |
| WORKSPACE | Adds rtp_opensource_deps and removes CPU/ARM-CPU pip repo installs. |
| BUILD | Notes that using_arm / using_cpu config_settings are now label-only and never match. |
| .bazelrc | Removes --config=cpu and --config=arm build configs. |
Comments suppressed due to low confidence (2)
deps/pip.bzl:46
pip_deps()no longer declares apip_parserepo for CUDA12 (cu126), butarch_config/arch_select.bzlstill references@pip_gpu_cuda12_torchforcuda_pre_12_9. This will break--config=cuda12(and any other path that needs the cu126 Python deps). Restore apip_gpu_cuda12_torchpip_parse with a cu126 lockfile, or remove/retarget thecuda_pre_12_9dependency path so it doesn’t reference a missing repo.
def pip_deps():
pip_parse(
name = "pip_ppu_torch",
requirements_lock = "@rtp_deps//:requirements_lock_torch_gpu_cuda12_9.txt",
python_interpreter = "/opt/conda310/bin/python3",
extra_pip_args = PIP_EXTRA_ARGS,
timeout = 3600,
)
pip_parse(
name = "pip_gpu_cuda12_9_torch",
requirements_lock = "@rtp_deps//:requirements_lock_torch_gpu_cuda12_9.txt",
python_interpreter = "/opt/conda310/bin/python3",
extra_pip_args = PIP_EXTRA_ARGS,
timeout = 3600,
quiet = False,
)
WORKSPACE:56
- WORKSPACE no longer loads/calls
pip_gpu_cuda12_torch_install_deps(), but the repo is still referenced fromarch_config/arch_select.bzl(@pip_gpu_cuda12_torch). If CUDA12 (cu126) remains supported,pip_deps()and WORKSPACE both need to create/install that repo; if it’s being dropped, please also remove the remainingpip_gpu_cuda12_torchreferences and associated select branches/configs so workspace loading can’t fail.
load("@rtp_deps//:pip.bzl", "pip_deps")
pip_deps()
load("@pip_ppu_torch//:requirements.bzl", pip_ppu_torch_install_deps = "install_deps")
pip_ppu_torch_install_deps()
load("@pip_gpu_cuda12_9_torch//:requirements.bzl", pip_gpu_cuda12_9_torch_install_deps = "install_deps")
pip_gpu_cuda12_9_torch_install_deps()
load("@pip_cuda12_arm_torch//:requirements.bzl", pip_cuda12_arm_torch_install_deps = "install_deps")
pip_cuda12_arm_torch_install_deps()
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+10
to
+11
| # availability regressions. | ||
| # See /home/liukan.lk/.claude/plans/serialized-wibbling-snail.md Phase 5. |
| # to wrapper target relate with different system config | ||
| load("@pip_cpu_torch//:requirements.bzl", requirement_cpu="requirement") | ||
| load("@pip_arm_torch//:requirements.bzl", requirement_arm="requirement") | ||
| load("@pip_gpu_cuda12_torch//:requirements.bzl", requirement_gpu_cuda12="requirement") |
Collaborator
Author
AI Code Review - PR #962Status: BLOCKING Summary: P0/0 · P1/2 · P2/2 · P3/0 Blocking IssuesP1
Non-blocking SuggestionsP2
Checklist Violations (4 fail / 23 total)General Principles Checklist
Strengths
|
LLLLKKKK
force-pushed
the
feature/pip_unify_v2
branch
from
ef772e7 to
e4f6072
Compare
Collaborator
Author
AI Code Review - PR #962Status: BLOCKING Summary: P0/1 · P1/0 · P2/1 · P3/1 Blocking IssuesP0
Non-blocking SuggestionsP2
P3
Checklist Violations (2 fail / 23 total)General Principles Checklist
Strengths
|
LLLLKKKK
force-pushed
the
feature/pip_unify_v2
branch
from
e4f6072 to
735509e
Compare
Collaborator
Author
AI Code Review - PR #962Status: BLOCKING Summary: P0/1 · P1/2 · P2/2 · P3/0 Blocking IssuesP0
P1
Non-blocking SuggestionsP2
Checklist Violations (6 fail / 23 total)General Principles Checklist
Strengths
|
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Bazel/Python dependency workflow to resolve wheels from a unified set of PEP 503 indexes, switching to “thin” per-platform requirement inputs (shared requirements_base.txt) and regenerated lockfiles so the same filenames/versions resolve consistently across environments.
Changes:
- Standardize pip index configuration via
deps/pip.bzl(--index-url+ PEP 503--extra-index-urllist). - Replace direct wheel URLs in requirement inputs with pinned package specs and PEP 503/simple-based references; regenerate lockfiles accordingly.
- Remove legacy CPU/CUDA12(cu126) requirement sources/targets and adjust Bazel selects and WORKSPACE wiring.
Reviewed changes
Copilot reviewed 16 out of 19 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| deps/pip.bzl | Defines unified pip index configuration and the remaining pip_parse repos. |
| deps/BUILD | Exports per-config requirements files and removes compile targets for dropped configs. |
| deps/http.bzl | Removes CPU torch http_archives; adds TODOs for future consolidation with internal overlay. |
| deps/requirements_torch_gpu_cuda12_9.txt | Converts to pinned “thin” input requirements for cu129 x86_64. |
| deps/requirements_lock_torch_gpu_cuda12_9.txt | Regenerated lockfile using unified index set. |
| deps/requirements_rocm.txt | Updates ROCm requirements to use pinned specs and unified indexes. |
| deps/requirements_lock_rocm.txt | Regenerated ROCm lockfile using unified index set. |
| deps/requirements_cuda12_arm.txt | Adds pinned CUDA12-arm requirements aligned with unified indexes. |
| deps/requirements_lock_cuda12_arm.txt | Regenerated CUDA12-arm lockfile using unified index set. |
| arch_config/arch_select.bzl | Removes CPU/ARM-CPU pip repos, changes default Python deps selection, and adjusts torch deps selection. |
| WORKSPACE | Adds rtp_opensource_deps and removes some pip repo installs; still loads @pip_ppu_torch. |
| BUILD | Keeps using_arm/using_cpu config_settings as “label-only” placeholders. |
| .bazelrc | Removes --config=cpu and --config=arm sections. |
| deps/requirements_torch_gpu_cuda12.txt (deleted) | Removes legacy cu126 GPU requirements input. |
| deps/requirements_torch_cpu.txt (deleted) | Removes legacy CPU torch requirements input. |
| deps/requirements_cpu_arm.txt (deleted) | Removes legacy ARM CPU requirements input. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
49
to
51
| load("@pip_ppu_torch//:requirements.bzl", pip_ppu_torch_install_deps = "install_deps") | ||
| pip_ppu_torch_install_deps() | ||
|
|
Comment on lines
+2
to
+9
| load("@pip_gpu_cuda12_9_torch//:requirements.bzl", requirement_gpu_cuda12_9="requirement") | ||
| load("@pip_gpu_rocm_torch//:requirements.bzl", requirement_gpu_rocm="requirement") | ||
| load("@rtp_llm//bazel:defs.bzl", "copy_so") | ||
|
|
||
| # cuda12 (cu126/torch2.6) was dropped — pip_gpu_cuda12_torch no longer registered | ||
| # in deps/pip.bzl. Remaining `cuda_pre_12_9` select branches resolve via cuda12_9. | ||
| requirement_gpu_cuda12 = requirement_gpu_cuda12_9 | ||
|
|
| @@ -77,26 +80,16 @@ def torch_deps(): | |||
| "@torch_rocm//:torch", | |||
| "@torch_rocm//:torch_libs", | |||
| ], | |||
Collaborator
Author
AI Code Review - PR #962Status: LGTM Summary: P0/0 · P1/0 · P2/1 · P3/1 lgtm ready to ci Non-blocking SuggestionsP2
P3
Checklist Violations (3 fail / 23 total)General Principles Checklist
Strengths
|
LLLLKKKK
force-pushed
the
feature/pip_unify_v2
branch
from
0a1800a to
642ed26
Compare
Collaborator
|
internal source has been updated, please review the changes! |
Contributor
There was a problem hiding this comment.
Pull request overview
Updates the Bazel/Python dependency plumbing to use a unified set of PEP 503 indexes (Aliyun PyPI mirror + PyTorch per-platform indexes + RTP OSS “simple” bucket) and shifts the repository’s default CUDA build/test config from cuda12_6 to cuda12_9.
Changes:
- Replace per-platform “full” requirements with thin inputs and regenerated lockfiles (now generated by
uv). - Simplify Bazel pip parsing to focus on
cuda12_9+cuda12_arm+ ROCm and remove older CPU/CUDA12.6-era branches/configs. - Update scripts/docs/test configs to use
--config=cuda12_9.
Reviewed changes
Copilot reviewed 30 out of 33 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| rtp_llm/test/perf_test/multi_node/multi_runner.sh | Updates default Bazel build args to --config=cuda12_9. |
| rtp_llm/test/perf_test/multi_node/multi_benchmark_config.yaml | Updates benchmark build args to --config=cuda12_9. |
| rtp_llm/models_py/standalone/BUILD | Removes cuda_pre_12_9 select branches. |
| rtp_llm/models_py/modules/hybrid/test/BUILD | Removes cuda_pre_12_9 select branches for test deps. |
| rtp_llm/models_py/modules/factory/attention/cuda_impl/test/BUILD | Removes cuda_pre_12_9 select branches for test deps. |
| rtp_llm/models_py/modules/factory/attention/cuda_cp_impl/test/BUILD | Removes cuda_pre_12_9 select branches for test deps. |
| rtp_llm/models_py/kernels/cuda/test/BUILD | Removes commented cuda_pre_12_9 branch in a disabled test stanza. |
| rtp_llm/BUILD | Removes ARM/CPU-specific select branches and cuda_pre_12_9 branches in Python deps lists. |
| docs/start/install.md | Updates example build command/config to cuda12_9. |
| docs/references/profiling.md | Updates profiling command to cuda12_9. |
| docs/references/debug.md | Updates debug test command to cuda12_9. |
| docs/benchmark/benchmark.md | Updates benchmark doc snippet to cuda12_9. |
| docs/backend/3fs.md | Updates build example to cuda12_9. |
| deps/requirements_torch_gpu_cuda12_9.txt | Converts to normalized names + PEP503 style references/pins for unified resolution. |
| deps/requirements_torch_gpu_cuda12.txt | Removes legacy CUDA12.6 requirements input. |
| deps/requirements_torch_cpu.txt | Removes legacy CPU requirements input. |
| deps/requirements_rocm.txt | Switches ROCm requirements to unified indexes and pinned names/versions. |
| deps/requirements_lock_torch_gpu_cuda12_9.txt | Regenerated lockfile (uv) with unified index configuration and hashes. |
| deps/requirements_lock_rocm.txt | Regenerated lockfile (uv) with unified index configuration and hashes. |
| deps/requirements_lock_cuda12_arm.txt | Regenerated lockfile (uv) for the cuda12_arm config. |
| deps/requirements_cuda12_arm.txt | Defines thin cuda12_arm requirements input (torch/torchvision + custom wheels). |
| deps/requirements_cpu_arm.txt | Removes legacy ARM-CPU requirements input. |
| deps/pip.bzl | Defines unified PIP_EXTRA_ARGS; drops older pip repos and adds pip_cuda12_arm_torch. |
| deps/http.bzl | Removes older torch http_archives; keeps torch 2.8 CUDA wheel and ROCm torch wheel. |
| deps/BUILD | Exports per-config requirement sources and drops legacy compile targets. |
| arch_config/arch_select.bzl | Removes CPU/ARM-CPU requirement routing and cuda_pre_12_9 routing; simplifies torch deps selection. |
| WORKSPACE | Adds rtp_opensource_deps repo and removes legacy pip installs, while keeping CUDA12.9/ROCm/ARM installs. |
| BUILD.pytorch | Removes cuda_pre_12_9 linkopts branch. |
| BUILD | Removes cuda_pre_12_9, using_arm, using_cpu config settings; updates compdb refresh target to cuda12_9. |
| .bazelrc | Removes cuda12_2, cuda12_6, cpu, arm configs; keeps cuda12_9 configs. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
49
to
51
| load("@pip_ppu_torch//:requirements.bzl", pip_ppu_torch_install_deps = "install_deps") | ||
| pip_ppu_torch_install_deps() | ||
|
|
Collaborator
Author
AI Code Review - PR #962Status: LGTM Summary: P0/0 · P1/0 · P2/1 · P3/3 lgtm ready to ci Non-blocking SuggestionsP2
P3
Checklist Violations (3 fail / 25 total)General Principles Checklist
Strengths
|
Collaborator
|
internal source has been updated, please review the changes! |
LLLLKKKK
added a commit
that referenced
this pull request
May 1, 2026
utils/util.py imports aiohttp but the py_library target never declared the pip dep. Surfaced on ut-sm8x in PR #962 (util_test, duplicated_kv_test → ModuleNotFoundError: No module named 'aiohttp'). Likely latent for a while — only visible when cuda12_9_x86 with a strict py_test runfiles sandbox runs these targets. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
Unifies Bazel/RULES_PYTHON dependency resolution around a single set of PEP 503 indexes (Aliyun PyPI mirror + download.pytorch.org per backend + RTP OSS “simple” index for custom wheels), while also standardizing the repo’s default CUDA build path onto cuda12_9 and regenerating lockfiles accordingly.
Changes:
- Reworked
deps/pip.bzl+ requirements inputs/lockfiles to resolve from PEP 503 indexes (and removed legacy CPU / older CUDA12 inputs). - Updated Bazel config surface to remove
cuda12_6/ CPU / ARM-CPU settings and align scripts/docs/tests oncuda12_9. - Added an opensource deps repository handle (
rtp_opensource_deps) for internal overlay reuse.
Reviewed changes
Copilot reviewed 30 out of 33 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| rtp_llm/test/perf_test/multi_node/multi_runner.sh | Switch default multi-node build/test config from cuda12_6 → cuda12_9. |
| rtp_llm/test/perf_test/multi_node/multi_benchmark_config.yaml | Updates benchmark build args to cuda12_9. |
| rtp_llm/models_py/standalone/BUILD | Drops cuda_pre_12_9 select branches in standalone deps. |
| rtp_llm/models_py/modules/hybrid/test/BUILD | Removes cuda_pre_12_9 flashinfer deps branch. |
| rtp_llm/models_py/modules/factory/attention/cuda_impl/test/BUILD | Removes cuda_pre_12_9 flashinfer deps branches for tests. |
| rtp_llm/models_py/modules/factory/attention/cuda_cp_impl/test/BUILD | Removes cuda_pre_12_9 flashinfer deps branches for CP tests. |
| rtp_llm/models_py/kernels/cuda/test/BUILD | Removes commented cuda_pre_12_9 select branch. |
| rtp_llm/BUILD | Removes legacy ARM/CPU selects, adds aiohttp to deps list, and keeps CUDA12.9/ARM/ROCm selects. |
| docs/start/install.md | Updates example Bazel build command to cuda12_9. |
| docs/references/profiling.md | Updates profiling example to cuda12_9. |
| docs/references/debug.md | Updates debug/test example to cuda12_9. |
| docs/benchmark/benchmark.md | Updates benchmark doc snippet to cuda12_9. |
| docs/backend/3fs.md | Updates 3FS build example to cuda12_9. |
| deps/requirements_torch_gpu_cuda12_9.txt | Converts to “thin” named requirements resolving via unified indexes. |
| deps/requirements_torch_gpu_cuda12.txt | Removes legacy CUDA12 (cu126) input requirements file. |
| deps/requirements_torch_cpu.txt | Removes legacy CPU torch requirements file. |
| deps/requirements_rocm.txt | Converts ROCm requirements to named deps aligned with unified indexes. |
| deps/requirements_lock_torch_gpu_cuda12_9.txt | Regenerated lockfile using the new index set and inputs. |
| deps/requirements_lock_rocm.txt | Regenerated ROCm lockfile using the new index set and inputs. |
| deps/requirements_lock_cuda12_arm.txt | Regenerated CUDA12 ARM lockfile using the new index set and inputs. |
| deps/requirements_cuda12_arm.txt | New thin CUDA12 ARM requirements source. |
| deps/requirements_cpu_arm.txt | Removes legacy ARM CPU requirements file. |
| deps/pip.bzl | Centralizes index config (explicit --index-url + extra indexes) and streamlines pip_parse repos to CUDA12.9/ARM/ROCm (+ PPU stub). |
| deps/http.bzl | Removes legacy torch http_archives (cpu + cu126), keeps torch 2.8 CUDA archive and other deps. |
| deps/BUILD | Exports per-config requirements sources and removes compile rules for deleted CPU/cu126 variants. |
| arch_config/arch_select.bzl | Simplifies requirement selection toward CUDA12.9/ROCm and drops legacy CPU/cu126 selects. |
| WORKSPACE | Adds rtp_opensource_deps repo handle and updates pip install_deps set (CUDA12.9/ARM/ROCm). |
| BUILD.pytorch | Removes pre-12.9 CUDA linkopts branch. |
| BUILD | Removes cuda_pre_12_9, using_arm, using_cpu config_settings; updates compdb config. |
| .bazelrc | Removes cuda12_2, cuda12_6, cpu, and arm configs; keeps cuda12_9_* and ROCm. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
15
to
22
| deps = select({ | ||
| "@rtp_llm//:cuda_pre_12_9": [requirement_gpu_cuda12(name)], | ||
| "@rtp_llm//:using_cuda12_9_x86": [requirement_gpu_cuda12_9(name)], | ||
| "@rtp_llm//:using_rocm": [requirement_gpu_rocm(name)], | ||
| "@rtp_llm//:using_arm": [requirement_arm(name)], | ||
| "//conditions:default": [requirement_cpu(name)], | ||
| # Default falls through to cuda12_9 (the canonical x86 GPU build). | ||
| # CPU-only and ARM-CPU configs were removed; if you need them, | ||
| # restore the corresponding pip_parse + lockfile + select branch. | ||
| "//conditions:default": [requirement_gpu_cuda12_9(name)], | ||
| }), |
| "@rtp_llm//:using_cuda12": ["torch==2.6.0+cu126"], | ||
| "@rtp_llm//:using_rocm": ["pyrsmi==0.2.0", "amdsmi@https://sinian-metrics-platform.oss-cn-hangzhou.aliyuncs.com/kis%2FAMD%2Famd_smi%2Fali%2Famd_smi.tar", "aiter@https://sinian-metrics-platform.oss-cn-hangzhou.aliyuncs.com/kis/AMD/RTP/aiter-0.1.13.dev14%2Bgfa35072d0.d20260402-cp310-cp310-linux_x86_64.whl"], | ||
| "//conditions:default": ["torch==2.1.2"], | ||
| # Default covers cuda12_9_x86, cuda12_9_arm (both use torch 2.8+cu129). |
Collaborator
Author
AI Code Review - PR #962Status: LGTM Summary: P0/0 · P1/0 · P2/0 · P3/0 lgtm ready to ci Checklist ✅ (25 items passed)Strengths
|
Collaborator
|
internal source has been updated, please review the changes! |
…CE/utils fixes Squashed roll-up of the pip-unification work for the opensource side. Strict separation invariant (Phase 2): - Opensource builds may only consume mirrors.aliyun.com/pypi/simple/ + rtp-opensource OSS + download.pytorch.org/whl/<cfg>/. - Internal builds may only consume artlab.alibaba-inc.com + rtp-opensource OSS. - The rtp-opensource OSS bucket is the SINGLE shared mirror for RTP-LLM custom wheels (flash_attn, deep_ep, deep_gemm, flashinfer_*, rtp_kernel, etc.) — same URL serves both sides. Build cfg cleanup: - Drop cuda12 (cu126/torch2.6) build path entirely; cuda12_9 (cu129/torch2.8) is now the canonical x86 GPU build. cuda_pre_12_9 config_setting + every select branch that used it removed (latent ABI mismatch: cu126 native libs with torch 2.8 Python deps from the cu129 pip_parse). - Drop using_arm / using_cpu config_settings + their select branches — --config=arm / --config=cpu bazelrc entries were removed with the lockfiles so the [] branches were unreachable and default branches (decord, xfastertransformer_devel) silently applied. - Drop dead pip_ppu_torch lockfile registration from opensource pip.bzl (no using_ppu select branch in opensource arch_select.bzl). - Drop unused cu126 + cpu --extra-index-url from opensource pip.bzl — halves per-package query surface against download.pytorch.org and reduces regen timeout flakiness. - Drop CPU-only and ARM-CPU build paths (requirements_torch_cpu.txt, requirements_torch_arm.txt, requirements_cpu_arm.txt) — no bazel cfg wires them up anymore. - Drop build:cuda12_2 / build:cuda12_6 from .bazelrc; retarget multi-node perf test scripts/yaml + 5 docs (install/benchmark/debug/profiling/3fs) to --config=cuda12_9. - Delete torch_2.6_py310_cuda http_archive + cuda_pre_12_9 branch in BUILD.pytorch linkopts. Shared-wheel hygiene: - aarch64 wheels retagged manylinux_2_28_aarch64 (uv refuses bare linux_aarch64 under aarch64-manylinux python-platform) and uploaded to OSS for direct version-pin. - URL-pin patched wheels to OUR OSS so resolver can't silently swap to an upstream copy with different bytes (flash_attn 2.8.3+cu12torch2.8cxx11abiTRUE, flashinfer-python 0.6.6, flashinfer-cubin 0.6.6 — we host patched variants). - nvidia-cutlass-dsl pinned 4.4.1 (latest on artlab; hash matches across artlab + aliyun mirror); pulls libs-base==4.4.1 transitive. Whl_deps tightening: - whl_deps() select key changed from :using_cuda12 → :using_cuda12_9_x86 + default. The :using_cuda12 key over-matched every CUDA variant (cuda12_9 inherits build:cuda12 which sets using_cuda12=true), so cu129 wheels were baking torch==2.6.0+cu126 into install_requires. WORKSPACE/pip stub: - WORKSPACE unconditionally calls pip_ppu_torch_install_deps(). Internal builds satisfy that via --override_repository=rtp_deps but opensource builds don't — so opensource-only builds errored 'Failed to load Starlark extension @pip_ppu_torch//:requirements.bzl'. Register pip_ppu_torch in opensource pip.bzl as a lazy alias of the cuda12_9 lockfile (pip_parse is lazy, no opensource select depends on it, so no PPU wheel is ever fetched). py_library deps: - //rtp_llm:utils now declares :aiohttp. utils/util.py imports aiohttp but the target had no pip dep — surfaced as ModuleNotFoundError on ut-sm8x's util_test + duplicated_kv_test once the strict py_test runfiles sandbox started enforcing it. Verified end-to-end on Aone CI pipeline 1346, run 39069908: all 28 jobs SUCCESS (cuda12_9 / cuda12_9_arm / amd / ppu / frontend builds + ut + smoke + perf + open_source variants). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
LLLLKKKK
force-pushed
the
feature/pip_unify_v2
branch
from
fe60498 to
d0a6fd8
Compare
Collaborator
Author
AI Code Review - PR #962Status: LGTM Summary: P0/0 · P1/0 · P2/1 · P3/2 lgtm ready to ci Non-blocking SuggestionsP2
P3
Checklist Violations (1 fail / 36 total)RTP-LLM Checklist
Strengths
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Switches the build to a single set of PEP 503 indexes so the same wheel filenames and versions resolve cleanly across all platforms.
Test plan