Fix/phase1 leak killers - #4
Open
fjkiani wants to merge 10 commits into
Open
Conversation
added 10 commits
May 29, 2026 21:43
- Add railway.toml: healthcheck /health 300s, restart ON_FAILURE - Add requirements-deploy.txt: strip streamlit/redis/alpaca-py/discord (~140MB) - Remove Render guard that forced API_LIGHT_MODE=0 (was overriding our fix) - API_LIGHT_MODE=1 on Railway skips UnifiedAlphaMonitor (~300-400MB startup) while keeping all API endpoints functional via compute_kill_chain() Root cause of 512MB OOM: monitor_initializer.initialize_all() loads 8+ heavy objects simultaneously at boot. This fix bypasses that entirely. Phase 1 leak fixes (d11a26f) remain in place for slow-burn protection.
Railway's Railpack detected pyproject.toml and ran poetry install, then couldn't find requirements-deploy.txt in the build context. Fix: use NIXPACKS builder with explicit nixpacks.toml that: - Uses python311 + gcc - Runs pip install -r requirements.txt (tracked in git) - Starts uvicorn with --workers 1 (single process, 512MB limit) API_LIGHT_MODE=1 env var (set via Railway API) skips the UnifiedAlphaMonitor startup bomb (~300-400MB).
- Railpack builder with buildCommand = pip install -r requirements.txt - requirements.txt is tracked in git and present in railway up upload - Remove nixpacks.toml (only works with GitHub-connected deploys, not railway up) - requirements-deploy.txt was gitignored so wasn't in the upload context
Railway's Railpack auto-detected pyproject.toml and ran poetry install, then couldn't find requirements-deploy.txt in the build context. Fix: switch to DOCKERFILE builder for full control: - Dockerfile: python:3.11-slim + pip install requirements-deploy.txt - requirements-deploy.txt: strips streamlit/redis/alpaca-py/discord (~140MB) - API_LIGHT_MODE=1 baked into Dockerfile ENV (skips 300-400MB monitor startup) - .dockerignore: excludes tests/, docs/, backtests/, cache/ from build context - railway.toml: builder=DOCKERFILE, healthcheck /health 300s, ON_FAILURE restart This is the definitive fix for the 512MB OOM: 1. Dockerfile controls exactly what gets installed (no poetry surprise) 2. API_LIGHT_MODE=1 skips UnifiedAlphaMonitor at startup 3. Phase 1 leak fixes (d11a26f) still in place for slow-burn protection
Wildcard patterns (*.md, *.json, *.png, etc.) in .dockerignore were causing railway up to exclude requirements-deploy.txt from the build context, making Docker COPY fail with 'not found'. Removed all extension wildcards. Only exclude specific dirs and known-safe patterns. requirements-deploy.txt is now always included.
railway up uploads a local snapshot and .dockerignore was filtering requirements-deploy.txt out of the build context. Fix: inline all production deps directly in the RUN pip install layer. No separate requirements file to COPY — self-contained Dockerfile. CMD uses JSON array form (fixes JSONArgsRecommended lint warning). PORT hardcoded to 8000 in CMD (Railway overrides via env var anyway).
Hard 'import redis' at module level crashed uvicorn startup when redis package is not installed (Railway deploy without Redis service). Fix: move 'import redis' inside get_redis() behind the REDIS_URL check. Redis is only needed when REDIS_URL env var is configured — if not set, get_redis() returns None and all callers handle None gracefully. No behavior change when REDIS_URL is set.
Previously API_LIGHT_MODE=1 only skipped UnifiedAlphaMonitor but still launched 4 staggered threads (DP recorder, AXLFI differ, volume spike, premarket scheduler) + brain polling loop + alpha graph loop + auto-snapshot. This caused RSS to grow from 110MB → 402MB in 35 minutes on Railway, projected to exceed 512MB OOM threshold within 1 hour. Fix: API_LIGHT_MODE=1 now returns immediately after marking all thread statuses as disabled. ONLY FastAPI request handlers run. True idle baseline. Thread footprint to be profiled in Phase 2 before re-enabling selectively.
Add _rss_burn_logger() async loop that reads RSS every 60s using psutil (or /proc/self/status fallback) and appends to _rss_history deque (maxlen=200, ~3.3h of history). Runs in both API_LIGHT_MODE=1 and full mode. Also emits structured RSS_BURN log lines captured by Railway log stream. New endpoints: GET /debug/memory — current rss_mb, vms_mb snapshot GET /debug/memory/history — full burn curve from startup to now No external API dependency. No auth. Ground truth from inside the container. Fixes the blind-polling problem where Railway metrics API had auth failures.
…itecture
Establishes the MDC (Master Data Control) protocol for multi-agent
parallel development. Replaces ad-hoc solo-developer approach with
compartmentalized squad ownership and typed inter-agent contracts.
Files created:
docs/architecture/mdc/00-master-index.mdc — squad directory, capability
map, 5 inter-agent contracts, phase status, hard rules
docs/architecture/mdc/01-core-infra.mdc — Squad 1: memory, SQLite,
async I/O, WebSockets, SnapshotState singleton
docs/architecture/mdc/02-execution-risk.mdc — Squad 2: gate logic,
RegimeResult typed, AAPL desync, DTE wiring, limit orders
docs/architecture/mdc/03-macro-data.mdc — Squad 3: DTECache singleton,
silent exception fixes, VIX fallback, narrative divergence typed
docs/architecture/mdc/04-training-ml.mdc — Squad 4: win rate analytics,
quality gates, standardized snapshot schema, outcome auto-resolution
docs/architecture/mdc/05-command-center.mdc — Squad 5: gate reason
transparency, DTE display, signal lineage tree, WS reconnection
Gap inventory: 15 INF + 12 EX + 15 MAC + 12 ML + 14 UI = 68 tracked gaps
Phase 1: 9 gaps CLOSED (commits d11a26f, 99c4d48, 1a194f4, bc9924f)
Phase 2: 59 gaps OPEN — distributed across 5 squads with acceptance criteria
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.