Skip to content

fix(seed): make 189 CAS rapid upload carry the right file, hashes and slice size - #3341

Merged
xrgzs merged 9 commits into
mainfrom
fix/189-seed-cas-rapid
Nov 6, 2026
Merged

fix(seed): make 189 CAS rapid upload carry the right file, hashes and slice size#3341
xrgzs merged 9 commits into
mainfrom
fix/189-seed-cas-rapid

Conversation

@PIKACHUIM

@PIKACHUIM PIKACHUIM commented Nov 4, 2026

Copy link
Copy Markdown
Member

Summary / 摘要

修复 189pc（天翼云）CAS 秒传的四个独立缺陷，以及一个会覆盖用户种子的设计问题。

用户可感知的行为变化：

  • .cas 秒传不再必然报 target driver cannot reuse the available hashes，请求会真正发到云端。
  • 秒传创建的对象名严格取种子内文件名（视频名），不会再出现「秒传成功但只多了一个 .cas」。
  • 失败文案携带真实原因，例如 秒传失败：云端不存在该文件（fileMD5=…, sliceMD5=…, size=…）。
  • 账号当日流量用尽时明确提示 天翼云盘账号当日上传流量已用尽…，同批其余文件不再逐个请求。
  • 对 .cas 文件再执行「生成种子」会被拒绝，且不会覆盖原 .cas。
  • 秒传一个「描述种子文件本身」的种子会被拒绝并说明原因，而不是在目标目录再造一个同名 .cas。

重要实现变化：

  • 秒传请求直接由种子文件构造（不再经 EncodeSeed(seed, "torrent") → Decode 中转），
    避免 torrent 容器强制要求 SHA-1 pieces 导致 CAS 种子必然失败。
  • 189pc：有分片信息时走 initMultiUpload → checkTransSecond → commitMultiUploadFile，
    否则退回旧的 createUploadFile.action；CAS 报错更具体时优先上报。
  • CAS 哈希语义修正：sliceMd5 是最终分片摘要（云端判重唯一比较值），显式值优先，
    逐片列表只在缺失时用于推算。
  • 分片大小统一由 torrent.CASSliceSize(文件大小) 决定（10MiB / 20MiB / 更大文件按 1999 片
    上限向上取 10MiB 的整数倍），piece_size 对 CAS 不再生效；189pc.partSize 改为委托同一实现。
  • .cas 新增 partsMd5（逐片 MD5），仅在切法与天翼规则一致时写出；sliceMd5 恒写最终摘要。
  • 生成 / 重算 / 转换补算 / 上传 sidecar 四条链路都按每个文件的 CASSliceSize 计算哈希；
    多文件跨分片档位时种子级只保留一种切法，其余文件只留最终摘要。
  • 新增 errs.DailyFlowLimitExceeded 终止态错误。

配置、存储、API 或兼容性变化：

  • .cas 载荷新增可选字段 partsMd5；不再写出旧的 slice_md5s（仍可读取）。

  • 兼容性：新代码可读旧 .cas（含只有 md5 + sliceMd5 的五字段格式）；旧代码读新 .cas
    时可只凭 sliceMd5 + slice_size 完成秒传。

  • 无配置项变更；piece_size 对 CAS 的语义变更为「忽略」。

  • This PR has breaking changes.
    / 此 PR 包含破坏性变更。

  • This PR changes public API, config, storage format, or migration behavior.
    / 此 PR 修改了公开 API、配置、存储格式或迁移行为。

  • This PR requires corresponding changes in related repositories.
    / 此 PR 需要关联仓库同步修改。

Related repository PRs / 关联仓库 PR:

  • OpenList-Frontend: 无必需改动。可选后续：种子生成/上传对话框在选中 CAS 时把「分片大小」改为
    只读提示（后端已忽略该值，当前 UI 对大文件仍显示 10 MiB，属提示性问题）。
  • OpenList-Docs:

Related Issues / 关联 Issue

无关联 Issue（问题由实际使用中复现并定位），本节可删除。

Testing / 测试

  • go test ./...
  • Manual test / 手动测试:

执行过的命令：

go build ./...
go vet ./server/handles/... ./internal/fs/... ./pkg/torrent/... ./drivers/189pc/... ./internal/driver/...
go test -p 1 ./server/handles/... ./internal/fs/... ./pkg/torrent/... ./drivers/189pc/... -count=1

结果：以上全部通过（含新增回归测试）。未执行全量 go test ./...（仓库含大量驱动用例，本轮只
覆盖受影响包）。并行跑多包时曾出现一次 server/handles 中 media review 用例偶发失败，连续重跑
（含 -count=1 与 -p 1）均通过，与本次改动无共享代码或全局状态。

手动验证：在真实 189 实例上复现并定位了三个阶段的报错（哈希不可用 / 账号限流 / 生成 .cas），
对照实例日志确认请求已按预期到达云端；成功路径由单测覆盖（五字段 .cas → 秒传请求名、size、
分片大小、sliceMd5 均取原值）。未在真实账号上完成一次成功的大文件秒传（受账号当日流量配额限制）。

新增测试：

  • TestLegacyCASToRapidRequestKeepsVideoName：五字段 .cas 解析后，秒传请求名必须是视频名，
    size / 分片大小 / sliceMd5 取原值，且不臆造逐片哈希
  • TestSeedRapidUploadBlockerRejectsSeedOfSeed：描述种子容器的种子禁止秒传
  • TestBuildSeedFileRapidUploadRequestKeepsCASHashMatrix
  • TestCASSliceSizeMatches189Rule / TestCASUniformSliceSize
  • TestEncodeCASStoresFinalSliceMD5AndParts / TestDecodeCASWithFinalSliceMD5Only /
    TestDecodeCASAcceptsLegacySliceMD5s / TestCASSliceDigestPrefersFinalDigest
  • TestIsSeedContainerName / TestIsDayFlowOverLimited

手工验收步骤

  1. 用只带 md5 + sliceMd5 的 .cas（例如 3GB 视频）秒传到 189：请求名应为视频名，命中后
    生成视频文件；云端没有该数据时返回带 fileMD5/sliceMD5/size 的具体错误。
  2. 对 .cas 文件执行「生成种子」：应被拒绝，且原 .cas 不被覆盖。
  3. 秒传一个「描述 .cas 自身」的种子：应报错说明，而不是生成新的 .cas。
  4. 账号当日流量用尽时：首个文件报明确原因，同批其余文件一次性标记为不可用。

Checklist / 检查清单

  • I have read CONTRIBUTING.
    / 我已阅读 CONTRIBUTING。
  • I confirm this contribution follows the repository license, contribution policy, and code of conduct.
    / 我确认此贡献符合仓库许可证、贡献规范和行为准则。
  • I have formatted the changed code with gofmt, go fmt, or prettier where applicable.
    / 我已按适用情况使用 gofmt、go fmt 或 prettier 格式化变更代码。
  • I have requested review from relevant maintainers or code owners where applicable.
    / 我已在适用情况下请求相关维护者或代码所有者审查。

AI Disclosure / AI 使用声明

  • This PR includes AI-assisted content.
    / 此 PR 包含 AI 辅助内容。

Tools used / 使用工具:

  • ChatGPT
  • Codex
  • GitHub Copilot
  • Claude
  • Gemini
  • Other (please specify) / 其他（请注明）: CodeBuddy (DeepSeek-V4.1-Flash)

Usage scope / 使用范围:

  • Code generation / 代码生成

  • Refactoring / 重构

  • Documentation / 文档

  • Tests / 测试

  • Translation / 翻译

  • Review assistance / 审查辅助

  • I have reviewed and validated all AI-assisted content included in this PR.
    / 我已审核并验证此 PR 中的所有 AI 辅助内容。

  • I have ensured that all AI-assisted commits include Co-Authored-By attribution.
    / 我已确保所有 AI 辅助提交都包含 Co-Authored-By 归属信息。

  • I can reproduce all AI-assisted content included in this PR without any AI tools.
    / 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。

pikachuren added 9 commits November 4, 2026 22:43
pikachuren
fix(seed): keep CAS hash matrix for 189 rapid upload
58aab1d 
CAS seeds only carry MD5 + slice MD5s, but the quick-save path rebuilt the rapid-upload request through a torrent container, which requires SHA-1 piece hashes. Every 189 秒传 therefore failed before any request was sent and reported 'target driver cannot reuse the available hashes'.

- build the rapid-upload request directly from the seed file, keeping the whole MD5, slice MD5s, the drive's slice size and the legacy slice digest
- let 189pc run initMultiUpload/checkTransSecond/commitMultiUploadFile when slice hashes are available, falling back to the legacy whole-MD5 upload
- surface the real rapid-upload error instead of the generic message
pikachuren
fix(189pc): surface daily flow limit instead of generic rapid-upload …
2cb4f2a 
…error

The CAS path now reaches 189's upload API, and the first real rejection is UserDayFlowOverLimited (account daily upload flow exhausted). Treat it as a terminal account state instead of retrying:

- add errs.DailyFlowLimitExceeded and map UserDayFlowOverLimited / 'flow is out' onto it (189 returns it as a business code inside HTTP 200)
- skip the legacy whole-MD5 fallback once the account is limited
- report 'rapid upload failed: ...' with the real reason and mark the rest of the batch unavailable instead of one request per file
pikachuren
fix(seed): refuse to generate a seed from a seed container
51cace9 
Generating a .cas seed from a .cas file produced an artifact with exactly the same name in the same folder, so PutDirectly overwrote the original seed with a seed-of-seed: the payload then described the few-hundred-byte metadata file instead of the video. Rapid upload of that seed 'succeeds' and only creates another .cas, which looks like CAS 秒传 doing nothing.

- skip .oss/.torrent/.cas/.cas.torrent when expanding a directory
- reject an explicit selection of a seed container with an actionable message
- refuse to write an artifact over one of its own source files
pikachuren
refactor(cas): derive slice size from file size and store the final d…
26e64fa 
…igest

CAS 判重只比较最终分片摘要（逐片 MD5 汇总后再取 MD5），而切法由文件大小固定：10MiB / 20MiB / 更大文件按 1999 片上限向上取 10MiB 的整数倍。之前把 sliceMd5 当成「可变分片的哈希」、总是从逐片列表现算，切法一变就对不上云端。

- add torrent.CASSliceSize / CASPartCount / CASUniformSliceSize as the single source of truth; 189pc.partSize delegates to it
- .cas writes the final digest in sliceMd5 and the per-slice list in partsMd5; slice_md5s stays readable for old files
- rapid upload prefers the explicit sliceMd5 and only derives it from parts when absent
- generation, recalculate and inline conversion hash CAS files at CASSliceSize per file and drop parts that do not match that slicing
- piece_size no longer influences CAS hashing
pikachuren
fix(seed): refuse to rapid-upload a seed that describes itself
9debbef 
A .cas generated from another .cas documents the metadata file itself (name ends with .cas, a few hundred bytes). Rapid-uploading it just creates another same-named .cas in the target folder, which looks like CAS 秒传 doing nothing. Reject that case with an actionable message instead.
pikachuren
test(seed): lock the rapid-upload name and the seed-of-seed guard
cd47048 
- extract seedRapidUploadBlocker so the guard is testable
- add an end-to-end test proving a legacy 5-field .cas resolves to the video name, size, slice size and final slice MD5
- add a test that seed containers are refused for rapid upload
pikachuren
refactor(seed): output seed and torrent messages in English
382bac4 
Turn the error strings, task statuses and progress labels of the transfer-seed / torrent feature into English, including the 189 CAS rapid-upload path and the front-end facing guards. Comments and unrelated log lines are untouched.
pikachuren
refactor(cas): make the CAS slice rule a verbatim port of the 189 upl…
f6a548c 
…oad rule

CAS slice size must be exactly what 189 uses when uploading, otherwise the sliceMd5 submitted for rapid upload can never match the digest the drive stored.

- CASSliceSize now copies 189pc partSize expression by expression (same DEFAULT constant, same float expression, same 5x lower bound)
- expectedCASPartCount mirrors fastUpload's count: one slice when size <= sliceSize, otherwise ceil(size/sliceSize)
- add cross-package tests comparing CASSliceSize/CASPartCount against a verbatim copy of the 189 rule, and partSize against CASSliceSize
pikachuren
fix(seed): address CAS sidecar and digest review findings
a44056f 
P1 FsStream hashed uploads with seedPieceSize() while the sidecar declared the CAS slice size; unknown-size CAS uploads now fail before writing the file instead of after. P1 buildCASFileEntry only compared piece counts, so 8MiB slices could be reused as 10MiB CAS slices; it now requires the slicing to match. P2 multi-slice payloads without sliceMd5 are rejected instead of falling back to the whole-file MD5, and oversized CAS slice sizes are clamped for the seed container only.
@xrgzs
xrgzs merged commit b8ab760 into main Nov 6, 2026
13 checks passed
@xrgzs
xrgzs deleted the fix/189-seed-cas-rapid branch November 6, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@xrgzs xrgzs xrgzs approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@PIKACHUIM @xrgzs