Skip to content

README.md

Latest commit

 

History

History
182 lines (146 loc) · 7.02 KB

README.md

File metadata and controls

182 lines (146 loc) · 7.02 KB

Services Control Plane

Control plane for managing hosted StreamingFast services on Kubernetes.

Project Structure

├── k8s/api/v1alpha1/      # CRD type definitions (HostedServiceDeployment)
├── cli/                   # TypeScript CLI for interacting with control plane
├── cmd/control-freak/     # Go gRPC server & CLI commands
├── store/deployment/      # Deployment data persistence (PostgreSQL)
├── dsn/                   # Database connection string builder
├── store/event/           # Deployment event tracking (PostgreSQL)
├── grpc/                  # gRPC service implementations
├── k8s/
│   ├── deployer/          # StatefulSet builders (sink-sql, foundational-store)
│   ├── operator/          # CRD operator/controller (reconcile loop)
│   ├── setup/             # Local development environment (START HERE!)
│   └── tests/             # Integration tests
├── pb/                    # Generated protobuf code
├── proto-internal/        # Internal proto definitions
├── registry/              # Network & foundation store registries (PostgreSQL)
└── vault/                 # Hashicorp Vault integration (secrets)

Quick Start (Local Development)

Want to test without touching production? Start here!

# One-time setup (creates local K8s cluster with all dependencies)
go run ./cmd/control-freak local setup

# Start port forwards (runs in foreground, Ctrl+C to stop)
go run ./cmd/control-freak local port-forward

# Run CLI (in another terminal)
cd cli
NODE_ENV=local bun run index.ts

You now have a complete local environment with:

  • Local Kubernetes cluster (k3d/k3s)
  • gRPC server running with CRD operator
  • PostgreSQL for deployment data, events, and registries
  • Hashicorp Vault for secret management

Full guide: k8s/setup/QUICKSTART.md Detailed docs: k8s/setup/README.md

Features

  • Deploy: Create service deployments on Kubernetes via CRDs
  • Undeploy: Remove deployments and clean up resources
  • SetReplica: Scale deployments up or down
  • UpdateDeploymentConfig: Update spkg, execution config, output config
  • ResetDeployment: Drop schema/truncate and restart from scratch
  • Logs: Stream logs from deployment pods
  • Deployment State: Real-time monitoring via CRD operator status
  • Deployment Events: Lifecycle event tracking in PostgreSQL
  • Deployment Persistence: All deployment data stored in PostgreSQL deployments table
  • Secret Management: Database passwords stored in Hashicorp Vault
  • Organization Filtering: Multi-tenant access control via organization_id
  • Admin Service: Separate gRPC service for CLI/admin portal operations
  • Network Registry: Manage substreams network endpoints
  • Foundation Store Registry: Manage foundational store deployments
  • TUI: Terminal interface for deployment management

Architecture

┌─────────────┐      gRPC        ┌───────────────────┐
│     CLI     │◄────────────────►│  control-freak    │
│ (TypeScript)│                  │  (Go gRPC server) │
└─────────────┘                  └────────┬──────────┘
                                          │
                          ┌───────────────┼───────────────┐
                          │               │               │
                          ▼               ▼               ▼
                   ┌────────────┐  ┌────────────┐  ┌──────────┐
                   │ Kubernetes │  │ PostgreSQL │  │  Vault   │
                   │   CRDs     │  │            │  │          │
                   └─────┬──────┘  └────────────┘  └──────────┘
                         │           deployments     secrets
                         │           events
                    ┌────┴──────┐    registries
                    │ Operator  │
                    │(reconcile)│
                    └────┬──────┘
                         │
                         ▼
                  ┌──────────────┐
                  │ StatefulSets │
                  │    + Pods    │
                  └──────────────┘

Data flow:

  1. CLI sends gRPC request to control-freak
  2. control-freak creates/updates HostedServiceDeployment CRD
  3. control-freak persists deployment data to PostgreSQL deployments table (fire-and-forget)
  4. CRD operator reconciles CRD → StatefulSet + Service
  5. Operator updates CRD status with live pod state
  6. Secrets stored in Vault, injected via Vault Agent sidecar

Development

Make server changes

# Edit Go code, then rebuild and restart in cluster:
go run ./cmd/control-freak local restart

Make CLI changes

# Edit TypeScript code, then:
cd cli
NODE_ENV=local bun run index.ts

Run tests

# Unit tests only
go test $(go list ./... | grep -v k8s/tests) -v -count 1

# Integration tests (requires k3d cluster + SUBSTREAMS_API_KEY)
KEEP_CLUSTER=true SUBSTREAMS_API_KEY=<key> go test ./k8s/tests/... -v -count 1

# All tests
KEEP_CLUSTER=true SUBSTREAMS_API_KEY=<key> go test ./... -v -count 1

# Specific test
go test ./k8s/tests/... -run TestDeploymentStore -v

Teardown local environment

go run ./cmd/control-freak local teardown

Production Deployment

# Build and run gRPC server
go build -o control-freak ./cmd/control-freak
./control-freak serve --postgres-dsn "postgres://..." --vault-addr "http://vault:8200"

# Run CLI
cd cli
bun run index.ts

Environment Variables

Variable Default Description
NODE_ENV production local for dev cluster, production for prod
GRPC_HOST Varies by env gRPC server address
POSTGRES_DSN Required PostgreSQL connection string
VAULT_ADDR Optional Hashicorp Vault address
VAULT_TOKEN Optional Vault authentication token
NAMESPACE default Kubernetes namespace for deployments
ORGANIZATION_ID Required (CLI) Organization ID for multi-tenant filtering
SUBSTREAMS_API_KEY Required (tests) API key for substreams integration tests

PostgreSQL Tables

Table Purpose
deployments All deployment configuration and metadata
deployment_events Lifecycle events (created, deleted, scaled, etc.)
foundation_store_registry Foundational store deployment registry
network_endpoints Network endpoint registry
network_aliases Network name → endpoint mappings

License

See LICENSE file.