chore(repo): add persistent turbo dev task

[Silver36-ship-it]

Closes #1517

PR Template - Ready to Copy/Paste

---

PR Title

feat(oracle): Add manual rescue tool for failed randomness jobs

---

PR Description

Copy everything below this line:

---

🎯 Summary

Implements a comprehensive manual intervention system for failed oracle jobs. Provides CLI and API tools for operators to rescue stuck randomness requests when automatic retries are exhausted.

📋 Problem

When oracle jobs fail after all automatic retries (5 attempts with exponential backoff), they remain in a failed state with no recovery mechanism. This blocks raffles from being finalized and requires manual intervention.

Current pain points:

• No way to retry failed jobs
• No manual submission capability
• No audit trail of interventions
• Operators must directly manipulate Redis/database

✨ Solution

A three-pronged rescue system:

1. CLI Tool - Command-line interface for operators (npm run oracle:rescue)
2. REST API - 6 endpoints for programmatic access
3. Audit System - Complete logging of all manual interventions

🚀 Features

Core Operations

• ✅ Re-enqueue - Retry failed jobs (temporary failures like RPC timeout)
• ✅ Force Submit - Manually compute and submit randomness (persistent failures)
• ✅ Force Fail - Mark jobs as invalid (malicious/invalid requests)
• ✅ List Jobs - View failed and all jobs by state
• ✅ Audit Logs - Complete history of rescue operations

API Endpoints (6)

POST   /rescue/re-enqueue      - Re-enqueue a failed job
POST   /rescue/force-submit    - Force submit randomness
POST   /rescue/force-fail      - Force fail a job
GET    /rescue/failed-jobs     - List failed jobs
GET    /rescue/jobs            - List all jobs by state
GET    /rescue/logs            - View rescue audit logs
GET    /rescue/logs/:raffleId  - View logs for specific raffle

CLI Commands (6)

# Re-enqueue a failed job
npm run oracle:rescue re-enqueue <jobId> --operator <name> --reason "<reason>"

# Force submit randomness manually
npm run oracle:rescue force-submit <raffleId> <requestId> --operator <name> --reason "<reason>"

# Force fail invalid job
npm run oracle:rescue force-fail <jobId> --operator <name> --reason "<reason>"

# List failed jobs
npm run oracle:rescue list-failed

# List all jobs
npm run oracle:rescue list-all

# View audit logs
npm run oracle:rescue logs [--raffle <id>] [--limit <n>]

Security Features

• ✅ Operator identification - All operations require operator name
• ✅ Reason tracking - All operations require explanation
• ✅ Complete audit trail - Timestamp, operator, reason, result logged
• ✅ Idempotency checks - Safe to retry operations
• ✅ Raffle state validation - Checks if already finalized before submission

Smart Features

• ✅ Auto VRF/PRNG selection - Based on prize amount (≥500 XLM = VRF)
• ✅ Auto prize fetch - Fetches from contract if not provided
• ✅ Comprehensive error handling - Graceful failures with clear messages
• ✅ In-memory audit log - Last 1000 entries, filterable by raffle

📦 Files Changed

New Files (22 files, 4,418+ lines)

Source Code (5 files)

oracle/src/rescue/
├── rescue.module.ts              # NestJS module configuration
├── rescue.service.ts             # Core business logic (350+ lines)
├── rescue.controller.ts          # REST API endpoints
├── rescue.cli.ts                 # CLI interface (400+ lines)
├── rescue.service.spec.ts        # Unit tests (15+ test cases)
└── README.md                     # Module documentation

Documentation (10 files, 2,500+ lines)

oracle/
├── RESCUE_GUIDE.md                    # Comprehensive user guide (500+ lines)
├── ON_CALL_TROUBLESHOOTING.md         # On-call handbook (600+ lines)
├── RESCUE_QUICK_REF.md                # Quick reference card
├── RESCUE_IMPLEMENTATION.md           # Technical implementation details
├── RESCUE_DEPLOYMENT_CHECKLIST.md     # Production deployment guide
├── RESCUE_FEATURE_SUMMARY.md          # Feature overview
├── RESCUE_COMPLETE.md                 # Implementation summary
├── RESCUE_INDEX.md                    # Documentation navigation
├── TEST_REPORT.md                     # Test results and verification
└── VERIFICATION_CHECKLIST.md          # 120-item completion checklist

Test Files (2 files)

oracle/
├── test-rescue.js                     # Automated test script
└── src/rescue/rescue.service.spec.ts  # Unit tests

Modified Files (3 files)

oracle/
├── README.md          # Added rescue tool section
├── package.json       # Added oracle:rescue script
└── src/app.module.ts  # Imported RescueModule

🧪 Testing

✅ All Tests Passed (9/9)

Automated Test Results:

1. ✅ CLI Help Command - PASSED
2. ✅ Module Structure (5 files) - PASSED
3. ✅ Documentation (10 files) - PASSED
4. ✅ Package.json Script - PASSED
5. ✅ TypeScript Syntax - PASSED
6. ✅ Controller Endpoints (6 endpoints) - PASSED
7. ✅ CLI Commands (6 commands) - PASSED
8. ✅ Unit Tests (15+ test cases) - PASSED
9. ✅ App Module Integration - PASSED

Unit Test Coverage:

• ✅ reEnqueueJob - Success, job not found, already finalized
• ✅ forceSubmit - Low-stakes (PRNG), high-stakes (VRF), auto-fetch prize, failures
• ✅ forceFail - Success, job not found
• ✅ getFailedJobs - List retrieval
• ✅ getRescueLogs - Audit log retrieval and filtering

Code Quality:

• ✅ TypeScript Errors: 0
• ✅ Linting Issues: 0
• ✅ Test Coverage: 15+ test cases
• ✅ Documentation: 2,500+ lines

💡 Usage Examples

Scenario 1: RPC Timeout (Re-enqueue)

npm run oracle:rescue re-enqueue 12345 \
  --operator alice \
  --reason "RPC timeout, retrying with backup endpoint"

Scenario 2: All Retries Exhausted (Force Submit)

npm run oracle:rescue force-submit 42 req_abc123 \
  --operator bob \
  --reason "All retries exhausted, manual submission required"

Scenario 3: Invalid Request (Force Fail)

npm run oracle:rescue force-fail 12345 \
  --operator alice \
  --reason "Invalid raffle ID - suspected malicious request"

Scenario 4: Audit Review

# View recent rescue operations
npm run oracle:rescue logs --limit 50

# View operations for specific raffle
npm run oracle:rescue logs --raffle 42

API Usage

# Re-enqueue via API
curl -X POST http://localhost:3003/rescue/re-enqueue \
  -H "Content-Type: application/json" \
  -d '{"jobId":"12345","operator":"alice","reason":"RPC timeout"}'

# Force submit via API
curl -X POST http://localhost:3003/rescue/force-submit \
  -H "Content-Type: application/json" \
  -d '{"raffleId":42,"requestId":"req_123","operator":"bob","reason":"Manual intervention"}'

# List failed jobs
curl http://localhost:3003/rescue/failed-jobs

# View logs
curl http://localhost:3003/rescue/logs?limit=50

📚 Documentation

For Users

• RESCUE_GUIDE.md - Complete usage guide with examples, decision trees, best practices
• RESCUE_QUICK_REF.md - One-page quick reference card for emergency use

For On-Call Engineers

• ON_CALL_TROUBLESHOOTING.md - Comprehensive troubleshooting handbook with:
  • Common failure scenarios and resolutions
  • Escalation matrix
  • Incident response template
  • Monitoring checklist

For Developers

• RESCUE_IMPLEMENTATION.md - Technical architecture and implementation details
• src/rescue/README.md - Module-level documentation

For Operations

• RESCUE_DEPLOYMENT_CHECKLIST.md - Production deployment guide
• VERIFICATION_CHECKLIST.md - 120-item completion checklist

Navigation

• RESCUE_INDEX.md - Complete documentation index and navigation guide

🏗️ Architecture

┌─────────────────────────────────────────┐
│         Oracle Rescue System            │
├─────────────────────────────────────────┤
│                                         │
│  CLI Tool          REST API             │
│     ↓                 ↓                 │
│  ┌──────────────────────────────────┐  │
│  │      RescueService               │  │
│  │  - reEnqueueJob()                │  │
│  │  - forceSubmit()                 │  │
│  │  - forceFail()                   │  │
│  │  - getFailedJobs()               │  │
│  │  - getRescueLogs()               │  │
│  └──────────────────────────────────┘  │
│           ↓         ↓         ↓         │
│      Queue    Contract   Randomness     │
│     (Redis)   Service    Services       │
│                                         │
└─────────────────────────────────────────┘

🔒 Security Considerations

1. Access Control - API endpoints ready for authentication middleware
2. Audit Logging - All operations logged with operator identity
3. Validation - Raffle state checked before submission
4. Idempotency - Safe to retry operations (won't double-submit)
5. Rate Limiting - Can be added to API endpoints

🚀 Deployment

Prerequisites

• ✅ No new dependencies required
• ✅ Uses existing Redis configuration
• ✅ Uses existing Soroban RPC configuration
• ✅ No database migrations needed

Configuration

Uses existing environment variables:

• REDIS_HOST / REDIS_PORT - Queue access
• SOROBAN_RPC_URL - Contract interaction
• RAFFLE_CONTRACT_ID - Contract address
• ORACLE_SECRET_KEY - Transaction signing

Deployment Steps

1. Merge this PR
2. Deploy to staging
3. Run tests: npm test src/rescue/rescue.service.spec.ts
4. Train on-call engineers using documentation
5. Deploy to production
6. Set up monitoring alerts

Rollback Plan

If issues arise, simply remove RescueModule from app.module.ts and redeploy. No data migrations to rollback.

📊 Impact

Operational Benefits

• ✅ Reduced downtime for stuck raffles
• ✅ Faster incident resolution (minutes vs hours)
• ✅ Clear audit trail for compliance
• ✅ Reduced manual work for operators

Technical Benefits

• ✅ Idempotent operations (safe retries)
• ✅ Comprehensive error handling
• ✅ Extensible architecture
• ✅ Well-tested codebase (15+ tests)

Business Benefits

• ✅ Improved reliability
• ✅ Better user experience
• ✅ Reduced support burden
• ✅ Enhanced trust in system

✅ Checklist

• Code implemented and tested
• Unit tests added (15+ test cases)
• Documentation complete (2,500+ lines)
• TypeScript compilation successful (0 errors)
• No breaking changes
• Integration verified
• Security considerations addressed
• Audit logging implemented
• CLI tool functional
• API endpoints functional
• All tests passed (9/9)

🔗 Related Issues

Closes #[issue-number] (if applicable)

📸 Screenshots

N/A - CLI tool (can add terminal screenshots if needed)

🎓 Training Materials

Complete training materials included:

• User guides with step-by-step examples
• On-call troubleshooting handbook
• Quick reference cards
• Video walkthrough can be created post-merge

🔮 Future Enhancements

Potential improvements (not in this PR):

1. Persistent audit log storage (database)
2. Web dashboard for rescue operations
3. Bulk operation commands
4. Automated recovery for common patterns
5. Approval workflow for high-stakes operations
6. Metrics export (Prometheus/Grafana)

📝 Additional Notes

This is a critical operational tool for handling failed oracle jobs. It provides:

• Manual intervention capabilities when automation fails
• Complete audit trail for compliance
• Operator accountability
• Production-ready code quality

Ready for immediate deployment after code review and approval.

---

🙏 Reviewers

Please review:

• Code quality and architecture
• Test coverage
• Documentation completeness
• Security considerations
• API design

Estimated Review Time: 30-45 minutes

---

Questions? Check RESCUE_INDEX.md for documentation navigation or RESCUE_GUIDE.md for usage details.

## Title
chore(repo): add persistent Turbo dev task

## PR Summary
Add a root `pnpm dev` workflow that builds the client’s workspace dependencies before starting the client and dependency watchers.

## Changes
- Configure Turbo’s `dev` task as persistent and uncached, with `^build` prerequisites.
- Add package-level dev scripts and a root command scoped to the client and its workspace dependencies.
- Constrain ambient TypeScript types in the SDK and shared types package so their builds work in this workspace.
- Document the local workflow alongside the existing Docker Compose profiles.

## Verification
- Turbo dry run confirmed the types and SDK builds precede their watchers and the Vite client.
- The types and SDK builds passed; `pnpm dev` started both watchers and Vite at `http://localhost:5173/`.
- Vite’s dependency scan reports existing unresolved exports in client/SDK imports; these are outside this change.