Summary
Three optional reliability features that harden the build pipeline for enterprise and self-hosted runner environments. These are inspired by battle-tested patterns from production Unity CI pipelines running on self-hosted runners with massive monorepos.
All three features are opt-in — they do nothing unless explicitly enabled, and they never block the build on failure (warnings only).
Features
1. Git Corruption Detection & Recovery
Self-hosted runners accumulate git state across builds. Corrupted .git directories, stale lock files, and broken submodule backing stores cause mysterious build failures that waste developer time.
What it does:
- Pre-checkout integrity check — runs
git fsck --no-dangling to detect object corruption before the build starts
- Stale lock file cleanup — removes
index.lock, shallow.lock, config.lock, and other lock files left by crashed git processes
- Submodule
.git file validation — verifies that submodule .git files point to existing backing stores in .git/modules/
- System git config corruption bypass — sets
GIT_CONFIG_NOSYSTEM=1 when system-level config is corrupted
- Automatic recovery — if corruption is detected, removes the corrupted
.git directory and triggers a fresh clone
Inputs:
gitIntegrityCheck — Enable pre-checkout git integrity verification (default: false)
gitAutoRecover — Automatically recover from detected corruption (default: true when integrity check enabled)
2. Reserved Filename Cleanup
Windows has reserved device names (nul, con, prn, aux, com1–com9, lpt1–lpt9) that cannot be used as filenames. When these appear in Unity project assets (often from cross-platform development or bad imports), they cause the Unity asset importer to enter infinite loops, hanging the build indefinitely.
What it does:
- Scans the project's
Assets/ directory for files or directories matching Windows reserved names
- Removes or renames them before Unity opens the project
- Logs warnings about each removed file so the source can be investigated
Inputs:
cleanReservedFilenames — Enable reserved filename cleanup before build (default: false)
3. Build Output Archival
For self-hosted runners with persistent storage, manage build output lifecycle with retention policies. Prevents disk space exhaustion from accumulated builds while keeping recent outputs accessible.
What it does:
- Moves completed build outputs to a configurable archive location after the build
- Maintains a retention count per location (e.g., keep 3 most recent builds)
- Supports tiered storage (fast local → bulk archive)
- Cleans up old archives beyond the retention limit
Inputs:
buildArchiveEnabled — Enable post-build output archival (default: false)
buildArchivePath — Path to archive location (default: ../builds-archive)
buildArchiveRetention — Number of builds to retain per platform (default: 3)
Implementation
New Service: BuildReliabilityService
Location: src/model/orchestrator/services/reliability/build-reliability-service.ts
export class BuildReliabilityService {
// Git integrity
static async checkGitIntegrity(repoPath: string): Promise<boolean>;
static async cleanStaleLockFiles(repoPath: string): Promise<number>;
static async validateSubmoduleBackingStores(repoPath: string): Promise<string[]>;
static async recoverCorruptedRepo(repoPath: string): Promise<void>;
// Reserved filenames
static async cleanReservedFilenames(projectPath: string): Promise<string[]>;
// Build archival
static async archiveBuildOutput(outputPath: string, archivePath: string, retention: number): Promise<void>;
static async enforceRetention(archivePath: string, retention: number): Promise<number>;
}
Integration Points
- Git integrity — runs in
src/index.ts before checkout/submodule init, and in remote-client/index.ts before bootstrapRepository()
- Reserved filenames — runs after checkout, before Unity build starts
- Build archival — runs after build completes, in the cleanup phase
Testing
Unit tests at src/model/orchestrator/services/reliability/build-reliability-service.test.ts covering:
- Lock file detection and cleanup (mock filesystem)
- Reserved filename pattern matching
- Retention policy enforcement
- Recovery flow (mock git commands)
Related
| PR |
Description |
Status |
| #808 |
feat(orchestrator): build reliability features |
Draft |
Summary
Three optional reliability features that harden the build pipeline for enterprise and self-hosted runner environments. These are inspired by battle-tested patterns from production Unity CI pipelines running on self-hosted runners with massive monorepos.
All three features are opt-in — they do nothing unless explicitly enabled, and they never block the build on failure (warnings only).
Features
1. Git Corruption Detection & Recovery
Self-hosted runners accumulate git state across builds. Corrupted
.gitdirectories, stale lock files, and broken submodule backing stores cause mysterious build failures that waste developer time.What it does:
git fsck --no-danglingto detect object corruption before the build startsindex.lock,shallow.lock,config.lock, and other lock files left by crashed git processes.gitfile validation — verifies that submodule.gitfiles point to existing backing stores in.git/modules/GIT_CONFIG_NOSYSTEM=1when system-level config is corrupted.gitdirectory and triggers a fresh cloneInputs:
gitIntegrityCheck— Enable pre-checkout git integrity verification (default: false)gitAutoRecover— Automatically recover from detected corruption (default: true when integrity check enabled)2. Reserved Filename Cleanup
Windows has reserved device names (
nul,con,prn,aux,com1–com9,lpt1–lpt9) that cannot be used as filenames. When these appear in Unity project assets (often from cross-platform development or bad imports), they cause the Unity asset importer to enter infinite loops, hanging the build indefinitely.What it does:
Assets/directory for files or directories matching Windows reserved namesInputs:
cleanReservedFilenames— Enable reserved filename cleanup before build (default: false)3. Build Output Archival
For self-hosted runners with persistent storage, manage build output lifecycle with retention policies. Prevents disk space exhaustion from accumulated builds while keeping recent outputs accessible.
What it does:
Inputs:
buildArchiveEnabled— Enable post-build output archival (default: false)buildArchivePath— Path to archive location (default:../builds-archive)buildArchiveRetention— Number of builds to retain per platform (default: 3)Implementation
New Service:
BuildReliabilityServiceLocation:
src/model/orchestrator/services/reliability/build-reliability-service.tsIntegration Points
src/index.tsbefore checkout/submodule init, and inremote-client/index.tsbeforebootstrapRepository()Testing
Unit tests at
src/model/orchestrator/services/reliability/build-reliability-service.test.tscovering:Related