Skip to content
BranchesTags
 
 

Repository files navigation

Substrate AI

A production-ready AI agent framework with streaming, memory, tools, and MCP integration.

Built on modern LLM infrastructure with OpenRouter support, PostgreSQL persistence, and extensible tool architecture.

🚀 Quick Start (One-Click Setup!)

Option 1: Automatic Setup (Recommended)

# Clone the repository
git clone https://github.com/your-username/substrate-ai.git
cd substrate-ai

# Run the setup wizard - it does EVERYTHING for you!
python setup.py

The setup script will:

  • ✅ Create Python virtual environment
  • ✅ Install all backend dependencies
  • ✅ Create configuration files
  • ✅ Install frontend dependencies
  • ✅ Validate your setup

After setup, just add your API key:

# Edit backend/.env and add your OpenRouter API key
# Get one at: https://openrouter.ai/keys

Option 2: Manual Setup

# Backend
cd backend
python3 -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate
pip install -r requirements.txt

# Frontend
cd ../frontend
npm install

# Configure
cp backend/.env.example backend/.env
# Edit backend/.env and add OPENROUTER_API_KEY=sk-or-v1-your-key

Start the Application

# Terminal 1: Backend
cd backend
source venv/bin/activate
python api/server.py

# Terminal 2: Frontend
cd frontend
npm run dev

# Open http://localhost:5173 🎉

📖 Full guide: See QUICK_START.md

✨ New users: The repository includes ALEX - a pre-configured example agent. Run python setup_alex.py after configuring your API key to get started immediately!

✨ Features

Core Capabilities

  • 🤖 Multi-Model Support - OpenRouter integration with 100+ LLMs
  • 💬 Streaming Responses - Real-time token streaming with SSE
  • 🧠 Memory System - Short-term (PostgreSQL) + Long-term (ChromaDB embeddings)
  • 🛠️ Tool Execution - Extensible tool architecture with built-in tools
  • 🔄 Session Management - Multi-session support with conversation history
  • 💰 Cost Tracking - Real-time token usage and cost monitoring

Advanced Features

  • 🧩 MCP Integration - Model Context Protocol for code execution & browser automation
  • 📊 PostgreSQL Backend - Scalable conversation & memory persistence
  • 🕸️ Graph RAG - Knowledge graph retrieval (works without Neo4j - uses local DB fallback!)
  • 🎯 Vision Support - Gemini Flash integration for image analysis
  • 🔐 Security Hardened - Sandboxed code execution, rate limiting, domain whitelisting
  • 📈 Token Efficiency - 98.7% context window savings via MCP code execution
  • 🎨 Modern UI - React + TypeScript + Tailwind CSS

🧠 Miras Memory Architecture (NEW!)

Based on Google Research Titans/Miras papers:

  • 🔄 Retention Gates - Dynamic memory decay/boost based on access patterns
  • 👁️ Attentional Bias - Multi-factor scoring (semantic + temporal + importance + access)
  • 🏛️ Hierarchical Memory - 3-tier system (Working → Episodic → Semantic)
  • 📈 Online Learning - Hebbian associations + feedback learning during runtime

📚 Documentation

Getting Started

Advanced Topics

Testing & Security

🏗️ Architecture

┌─────────────────────────────────────────────────┐
│              Frontend (React)                    │
│  • Real-time streaming UI                       │
│  • Session management                           │
│  • Memory blocks editor                         │
│  • Cost & token tracking                        │
└─────────────────┬───────────────────────────────┘
                  │
                  │ HTTP/SSE
                  │
┌─────────────────▼───────────────────────────────┐
│           Backend (Python)                       │
│                                                  │
│  ┌────────────────────────────────────────┐    │
│  │     Consciousness Loop                 │    │
│  │  • Model routing (OpenRouter)          │    │
│  │  • Stream management                   │    │
│  │  • Tool execution                      │    │
│  │  • Memory integration                  │    │
│  └────────────────────────────────────────┘    │
│                                                  │
│  ┌─────────────┐  ┌─────────────┐             │
│  │  Memory     │  │   Tools     │             │
│  │  System     │  │  Registry   │             │
│  │  + MIRAS    │  │             │             │
│  │ • Core      │  │ • Web       │             │
│  │ • Archival  │  │ • Search    │             │
│  │ • Embedding │  │ • Discord   │             │
│  │ • Retention │  │ • ArXiv     │             │
│  │ • Hebbian   │  │ • Jina      │             │
│  └─────────────┘  └─────────────┘             │
│                                                  │
│  ┌────────────────────────────────────────┐    │
│  │        Graph RAG System               │    │
│  │  • Knowledge graph retrieval          │    │
│  │  • Neo4j (optional) or local DB       │    │
│  │  • Relationship extraction            │    │
│  └────────────────────────────────────────┘    │
│                                                  │
│  ┌────────────────────────────────────────┐    │
│  │        MCP Integration                 │    │
│  │  • Code execution sandbox              │    │
│  │  • Browser automation (Playwright)     │    │
│  │  • Skills learning system              │    │
│  │  • Vision analysis (Gemini)            │    │
│  └────────────────────────────────────────┘    │
└──────────────────┬──────────────────────────────┘
                   │
      ┌────────────┼────────────┬────────────┐
      │            │            │            │
┌─────▼─────┐ ┌───▼────┐ ┌────▼─────┐ ┌────▼─────┐
│PostgreSQL │ │ChromaDB│ │MCP Servers│ │  Neo4j   │
│Persistence│ │Vectors │ │(External) │ │(Optional)│
└───────────┘ └────────┘ └──────────┘ └──────────┘

🔧 Tech Stack

Backend

  • Python 3.11+ - Core runtime
  • Flask - API server with SSE streaming
  • PostgreSQL - Primary database (conversation history, memory)
  • ChromaDB - Vector embeddings for semantic search
  • Neo4j - Graph database for Graph RAG (optional, local DB fallback)
  • OpenRouter - Multi-model LLM gateway
  • RestrictedPython - Sandboxed code execution

Frontend

  • React 18 - UI framework
  • TypeScript - Type safety
  • Tailwind CSS - Styling
  • Vite - Build tool & dev server

MCP Integration

  • Playwright - Browser automation (Chromium)
  • Gemini 2.0 Flash - Vision analysis (free tier)
  • fastmcp - MCP protocol implementation
  • MCP Servers - Stdio-based external tools

🛠️ Built-in Tools

Memory Management

  • core_memory_append - Add to agent's core memory
  • core_memory_replace - Modify core memory
  • archival_memory_insert - Store in long-term memory
  • archival_memory_search - Semantic search across memories

Miras Memory Architecture

Advanced memory features based on Google Research:

  • retention_gate.compute_retention() - Calculate memory retention score
  • attentional_bias.compute_attention_score() - Multi-factor relevance scoring
  • hierarchical_memory.store() - Store in tiered memory system
  • memory_learner.on_memories_accessed() - Record Hebbian associations
  • memory_learner.record_feedback() - Learn from user feedback

Web & Research

  • fetch_webpage - Retrieve and parse web pages
  • web_search - DuckDuckGo search
  • arxiv_search - Academic paper search
  • jina_reader - Advanced web content extraction

Integration

  • discord_send_message - Discord bot integration
  • spotify_control - Spotify playback control
  • execute_code - Sandboxed Python execution (MCP)

Graph RAG

  • /api/graph/nodes - Get graph nodes
  • /api/graph/edges - Get graph relationships
  • /api/graph/stats - Graph statistics
  • /api/graph/rag - Retrieve context from knowledge graph

Browser Automation (MCP)

  • navigate - Browser navigation
  • screenshot - Capture with vision analysis
  • extract_text - DOM text extraction
  • click / fill_form - Page interaction
  • search_google - Google search automation

📦 Installation

Prerequisites

  • Python 3.11+
  • Node.js 18+
  • PostgreSQL 14+ (optional, SQLite fallback available)
  • OpenRouter API key

Backend Setup

cd backend

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # Mac/Linux
# venv\Scripts\activate  # Windows

# Install dependencies
pip install -r requirements.txt

# Configure environment
cp config/.env.example .env
# Edit .env with your API keys

# Optional: Install Playwright for MCP browser automation
playwright install chromium

Frontend Setup

cd frontend

# Install dependencies
npm install

# Start dev server
npm run dev

🔐 Security Features

Code Execution Sandbox

  • ✅ RestrictedPython compilation (no unsafe operations)
  • ✅ 30-second timeout enforcement
  • ✅ 512MB memory limit per execution
  • ✅ Isolated workspace per session
  • ✅ No file system access outside sandbox
  • ✅ No network access except via MCP tools

Browser Automation Security

  • ✅ Domain whitelist (Wikipedia, GitHub, ArXiv, etc.)
  • ✅ Domain blacklist (banking, payments blocked)
  • ✅ Rate limiting (10 nav/min, 5 screenshots/min)
  • ✅ Headless mode only (no GUI)
  • ✅ HTTPS enforcement on sensitive operations

API Security

  • ✅ Rate limiting on all endpoints
  • ✅ CORS configuration
  • ✅ Input sanitization
  • ✅ API key validation

📋 Full audit: See FINAL_SECURITY_CHECK.md

🚦 Usage Examples

Basic Chat

# The agent maintains context across messages
User: "My name is Alex"
Agent: "Nice to meet you, Alex! I've stored that in my memory."

User: "What's my name?"
Agent: "Your name is Alex!"

Tool Usage

# Memory tools
User: "Remember that I'm learning Python"
Agent: *uses core_memory_append*
Agent: "I've added that to my memory about you!"

# Web search
User: "What's the latest on quantum computing?"
Agent: *uses web_search*
Agent: "Here's what I found about quantum computing..."

MCP Code Execution

# Browser automation with vision
User: "Take a screenshot of Wikipedia's homepage and describe it"
Agent: *writes code*
url = "https://en.wikipedia.org"
result = await mcp.browser.screenshot(url, analyze=True)
print(result['analysis'])

Result: Vision analysis returned, 98.7% token savings vs manual browsing

📖 Full MCP guide: See MCP_SYSTEM_OVERVIEW.md

📊 Performance

Token Efficiency

  • Without MCP: ~100,000 tokens for complex web tasks
  • With MCP: ~2,000 tokens (98.7% reduction)
  • Streaming: <50ms first token latency

Execution Speed

  • Code compilation: <50ms (RestrictedPython)
  • Typical execution: 200-500ms
  • Max timeout: 30s (enforced)

Memory Performance

  • Core memory: O(1) access (PostgreSQL indexed)
  • Archival search: <200ms (ChromaDB vector similarity)
  • Skills lookup: O(log n) with semantic indexing

🧪 Testing

# Backend tests
cd backend
python test_startup.py

# Integration tests
python test_mcp_integration.py

# Full test results
cat ../TESTING_RESULTS.md

🗺️ Roadmap

Completed ✅

  • Multi-model OpenRouter integration
  • Streaming SSE responses
  • PostgreSQL persistence
  • Memory system (core + archival)
  • Tool execution framework
  • MCP code execution sandbox
  • Browser automation (Playwright)
  • Vision analysis (Gemini)
  • Skills learning system
  • Cost tracking
  • Miras Memory Architecture (December 2025)
    • Retention Gates (dynamic memory decay/boost)
    • Attentional Bias (multi-factor retrieval scoring)
    • Hierarchical Memory (Working → Episodic → Semantic)
    • Online Learning (Hebbian associations + feedback)

In Progress 🚧

  • Additional MCP servers (filesystem, database)
  • Collaborative skill libraries
  • Advanced prompt engineering UI
  • Multi-agent orchestration

Planned 🎯

  • Voice interface
  • Mobile app
  • Cloud deployment templates
  • Plugin marketplace

🤝 Contributing

This is an open-source project. Contributions welcome!

Development Setup

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Test thoroughly
  5. Submit a pull request

Code Standards

  • Python: PEP 8, type hints, docstrings
  • TypeScript: ESLint, Prettier, strict mode
  • Tests: Add tests for new features
  • Docs: Update relevant documentation

📄 License

See LICENSE for details.

🙏 Acknowledgments

Technologies

  • OpenRouter - Multi-model API gateway
  • Anthropic MCP - Model Context Protocol architecture
  • Playwright - Browser automation framework
  • Gemini - Vision analysis (Google)
  • PostgreSQL - Database engine
  • ChromaDB - Vector embeddings

Research

  • Google Titans/Miras - Advanced memory architecture (Retention Gates, Attentional Bias, Online Learning)
  • "It's All Connected" - Test-time memorization and retention research

Community

Built with inspiration from:

  • Letta (formerly MemGPT) - Memory architecture patterns
  • LangChain - Tool execution concepts
  • AutoGPT - Agent autonomy ideas

📧 Support

  • 🐛 Bug Reports: GitHub Issues
  • 💬 Questions: GitHub Discussions
  • 📖 Documentation: See /docs folder
  • 🔧 Troubleshooting: See QUICK_START.md

Built for developers who need production-ready AI agents.

Version 1.1.0 | Last Updated: December 2025

🧠 Miras Memory Architecture Details

Based on Google Research Titans & Miras papers, this framework implements a 4-phase advanced memory system:

Phase 1: Retention Gates (~490 lines)

File: backend/core/retention_gate.py

Dynamic memory decay/boost based on:

  • Importance (35% weight)
  • Access count (30% weight)
  • Temporal recency (25% weight)
  • Base retention (10% weight)
from core.retention_gate import RetentionGate

gate = RetentionGate()
score = gate.compute_retention(memory)  # 0.0 - 1.0
action = gate.get_action(score)  # BOOST, KEEP, CONSOLIDATE, DECAY, ARCHIVE

Phase 2: Attentional Bias (~610 lines)

File: backend/core/attentional_bias.py

5 attention modes with automatic query analysis:

  • STANDARD - Balanced retrieval
  • SEMANTIC_HEAVY - Meaning-focused
  • TEMPORAL_HEAVY - Time-sensitive ("when did we...")
  • IMPORTANCE_HEAVY - Critical information
  • EMOTIONAL - Relationship/feeling queries
from core.attentional_bias import QueryAnalyzer, AttentionalBias

analyzer = QueryAnalyzer()
mode = analyzer.analyze("When did we last meet?")  # → TEMPORAL

bias = AttentionalBias()
score = bias.compute_attention_score(memory, query, mode)

Phase 3: Hierarchical Memory (~720 lines)

File: backend/core/hierarchical_memory.py

3-tier memory architecture:

  • Working Memory - Fast, volatile, LRU eviction (current session)
  • Episodic Memory - Medium-term, retention gates (recent history)
  • Semantic Memory - Long-term, Graph DB integration (permanent knowledge)
from core.hierarchical_memory import HierarchicalMemory, MemoryItem

hier = HierarchicalMemory()
hier.store(memory_item)  # Auto-routes to appropriate tier
hier.consolidate()  # Move memories between tiers

Phase 4: Online Learning (~500 lines)

File: backend/core/memory_learner.py

Hebbian learning: "Neurons that fire together, wire together"

  • Memories accessed together form associations
  • User feedback adjusts importance
  • Association decay for unused connections
from core.memory_learner import MemoryLearner, FeedbackType

learner = MemoryLearner()
learner.on_memories_accessed(['mem1', 'mem2'], query="...")  # Forms associations
learner.record_feedback('mem1', FeedbackType.HELPFUL)  # +0.5 importance
learner.record_feedback('mem2', FeedbackType.NOT_HELPFUL)  # -0.2 importance

Total: ~2,320 lines of research-backed memory architecture!

📖 Full documentation: See docs/MIRAS_TITANS_INTEGRATION.md

About

Open-source framework for building stateful AI agents with memory, tools, and streaming. Works with any OpenAI-compatible API.

Resources

Readme
License
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages