🤖 SwarAI

Multi-Agent AI Task Automation Assistant

A sophisticated multi-agent AI system powered by CrewAI, LangChain, and Groq LLM

Features • Quick Start • Installation • Usage • API • What's New • Contributing

📖 Table of Contents

Overview
Features
Architecture
Quick Start
Installation
Configuration
Usage
AI Agents
API Reference
Development
Troubleshooting
Contributing
Recent Improvements
Roadmap
License

🌟 Overview

SwarAI is an advanced multi-agent AI task automation assistant that combines natural language processing, voice recognition, file management, and cross-platform communication into a unified, intelligent system.

Key Highlights

🤖 13 Specialized AI Agents for different tasks
🎤 Voice Recognition with multiple engines (Google Speech, Whisper AI)
🗣️ Text-to-Speech with multiple TTS engines (Edge TTS, gTTS, Coqui)
📱 WhatsApp Integration for automated messaging
📁 Intelligent File Search with fuzzy matching
🔄 Multi-Agent Orchestration using CrewAI
🌐 Modern Web Interface built with Next.js
🚀 FastAPI Backend with WebSocket support
💾 Conversation Memory with MongoDB (optional)

✨ Features

🎯 Core Capabilities

1. Conversational AI

Natural language understanding with context awareness
Personality-driven responses
Multi-turn conversation support
Intent classification and routing
Emotional intelligence

2. Voice Recognition & TTS

Speech-to-Text: Google Speech Recognition, Whisper AI
Text-to-Speech: Microsoft Edge TTS, Google TTS, Coqui TTS, pyttsx3
Multi-language support (English variants)
Noise reduction and ambient adjustment
Real-time voice processing

3. WhatsApp Automation

Send messages via voice or text commands
Contact management with fuzzy search
WhatsApp URL generation (wa.me format)
Natural language command parsing
Multiple command pattern support

4. File Management

Cross-platform file search (Windows, macOS, Linux)
Fuzzy matching algorithm
Recursive directory searching
File operations: search, open, share
Performance-optimized scanning

5. System Control

Volume control (Windows with pycaw)
Brightness adjustment
Battery status monitoring
System information retrieval
Application launching

6. Multi-Agent Orchestration

CrewAI-powered agent coordination
Complex workflow execution
Task delegation and routing
Error recovery and fallbacks
Parallel task processing

🏗️ Architecture

System Architecture

┌─────────────────────────────────────────────────────────────┐
│                     Frontend (Next.js)                       │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────┐      │
│  │ Voice UI     │  │ Agent Cards  │  │ Results      │      │
│  └──────────────┘  └──────────────┘  └──────────────┘      │
└────────────────────────┬────────────────────────────────────┘
                         │ HTTP/WebSocket
┌────────────────────────┴────────────────────────────────────┐
│                   Backend (FastAPI)                          │
│  ┌──────────────────────────────────────────────────────┐   │
│  │              Agent Manager (MCP)                      │   │
│  │  - Intent Detection  - Agent Routing  - Workflows    │   │
│  └──────────────────────────────────────────────────────┘   │
│                         │                                    │
│  ┌──────────────────────┴──────────────────────────────┐   │
│  │              Specialized Agents                      │   │
│  │  WhatsApp │ FileSearch │ Conversation │ System      │   │
│  │  Email │ Calendar │ Payment │ WebSearch │ ...       │   │
│  └──────────────────────────────────────────────────────┘   │
│                         │                                    │
│  ┌──────────────────────┴──────────────────────────────┐   │
│  │           CrewAI Orchestration Layer                 │   │
│  │  - Multi-Agent Coordination                          │   │
│  │  - Task Delegation                                   │   │
│  │  - Workflow Management                               │   │
│  └──────────────────────────────────────────────────────┘   │
│                         │                                    │
│  ┌──────────────────────┴──────────────────────────────┐   │
│  │              LLM Integration (Groq)                  │   │
│  │  LangChain │ LangGraph │ Groq LLM │ Context Memory  │   │
│  └──────────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────────┘

Technology Stack

Backend

Framework: FastAPI 0.115+
AI/ML:
- CrewAI 0.86+ (Multi-agent orchestration)
- LangChain 1.2+ (AI framework)
- LangGraph 1.0+ (Stateful workflows)
- Groq LLM (Language model)
Speech:
- SpeechRecognition 3.10+
- gTTS 2.5+
- pydub 0.25+
- pygame 2.5+
Database:
- MongoDB (via pymongo/motor)
Server: Uvicorn (ASGI)

Frontend

Framework: Next.js 15.5
UI: React 19, TailwindCSS 4
State: Zustand
Queries: TanStack Query
Components: Radix UI, Framer Motion
Icons: Lucide React

🚀 Quick Start

Prerequisites

Python 3.10 or higher
Node.js 18+ and npm
Groq API Key (Get one free)
MongoDB (optional, for conversation memory)

1. Clone the Repository

git clone https://github.com/shashankpc7746/SwarAI.git
cd SwarAI

2. Backend Setup

# Create virtual environment
python -m venv venv

# Activate virtual environment
# Windows:
venv\Scripts\activate
# Linux/Mac:
source venv/bin/activate

# Install dependencies
cd backend
pip install -r requirements.txt

# Configure environment
cp .env.example .env
# Edit .env and add your GROQ_API_KEY

3. Frontend Setup

cd frontend
npm install --legacy-peer-deps

4. Run the Application

Terminal 1 - Backend:

cd backend
python main.py

Terminal 2 - Frontend:

cd frontend
npm run dev

5. Access the Application

Frontend UI: http://localhost:3000
API Documentation: http://localhost:8000/docs
Health Check: http://localhost:8000/health

📦 Installation

Detailed Backend Installation

Create and activate virtual environment:

python -m venv venv
venv\Scripts\activate  # Windows
source venv/bin/activate  # Linux/Mac

Install Python dependencies:

cd backend
pip install -r requirements.txt

Set up environment variables:
```
cp .env.example .env
```

Edit .env file:

# Required
GROQ_API_KEY=your_groq_api_key_here
GROQ_MODEL=llama-3.1-70b-versatile

# Optional
MONGODB_URL=mongodb://localhost:27017
MONGODB_DATABASE=swarai_assistant
TTS_ENGINE=edge  # edge, gtts, coqui, pyttsx3
ENABLE_VOICE_FEEDBACK=true

Detailed Frontend Installation

Install Node.js dependencies:

cd frontend
npm install --legacy-peer-deps

Configure environment (optional):

# Create .env.local if needed
echo "NEXT_PUBLIC_API_URL=http://localhost:8000" > .env.local

Optional Dependencies

For full system control features:

# Windows volume control
pip install pycaw comtypes

# System monitoring
pip install psutil

# Brightness control
pip install screen-brightness-control

⚙️ Configuration

Environment Variables

Core Configuration

Variable	Description	Default	Required
`GROQ_API_KEY`	Groq API key for LLM	-	✅ Yes
`GROQ_MODEL`	Groq model to use	`llama-3.1-70b-versatile`	No
`FASTAPI_HOST`	Backend host	`0.0.0.0`	No
`FASTAPI_PORT`	Backend port	`8000`	No

Voice & Speech

Variable	Description	Default
`TTS_ENGINE`	TTS engine (edge/gtts/coqui/pyttsx3)	`edge`
`SWARAI_VOICE`	Voice for TTS	`en-US-AriaNeural`
`ENABLE_VOICE_FEEDBACK`	Enable voice responses	`true`
`SPEECH_TIMEOUT`	Speech recognition timeout (seconds)	`7`
`SPEECH_PHRASE_TIME_LIMIT`	Max phrase duration (seconds)	`15`

Database

Variable	Description	Default
`MONGODB_URL`	MongoDB connection string	`mongodb://localhost:27017`
`MONGODB_DATABASE`	Database name	`swarai_assistant`
`CONVERSATION_MEMORY_LIMIT`	Max conversation history	`50`

Agent Configuration

Variable	Description	Default
`AGENT_TEMPERATURE`	LLM temperature	`0.1`
`MAX_RESPONSE_TOKENS`	Max tokens in response	`1000`

💻 Usage

Voice Commands

WhatsApp

"Send WhatsApp to Jay: Hello, how are you?"
"Message Mom: I'll be late for dinner"
"WhatsApp Vijay: Can we reschedule the meeting?"

File Search

"Find my presentation"
"Search for report.pdf"
"Open the latest invoice"
"Find photos from last week"

System Control

"Set volume to 50%"
"Increase brightness"
"Check battery status"
"What's my system info?"

Conversation

"Hello SwarAI!"
"What can you do?"
"Help me with my tasks"
"Tell me a joke"

API Usage

Process Command

curl -X POST http://localhost:8000/process-command \
  -H "Content-Type: application/json" \
  -d '{"command": "Send WhatsApp to Jay: Hello!"}'

Text-to-Speech

curl -X POST http://localhost:8000/tts \
  -H "Content-Type: application/json" \
  -d '{"text": "Hello from SwarAI!"}'

WebSocket Connection

const ws = new WebSocket('ws://localhost:8000/ws');

ws.onmessage = (event) => {
  const data = JSON.parse(event.data);
  console.log('Received:', data);
};

ws.send(JSON.stringify({
  type: 'command',
  data: { command: 'Hello SwarAI!' }
}));

🤖 AI Agents

Available Agents

Agent	Description	Capabilities
WhatsApp	Message automation	Send messages, contact search, URL generation
FileSearch	File management	Search files, open files, fuzzy matching
Conversation	Natural dialogue	Context-aware chat, personality, memory
System Control	System operations	Volume, brightness, battery, system info
Email	Email automation	Compose, send emails (Gmail integration)
Calendar	Calendar management	Create events, reminders (Google Calendar)
Payment	Payment processing	PayPal, Google Pay, UPI integration
WebSearch	Web searching	Google, Bing, DuckDuckGo, YouTube
Phone	Phone operations	Make calls, SMS (platform-dependent)
App Launcher	Application control	Launch apps, manage windows
Screenshot	Screen capture	Take screenshots, save images
Task	Task management	Create, manage tasks and reminders
Multi-Task	Workflow orchestration	Complex multi-step operations

Agent Architecture

Each agent follows a consistent pattern:

class Agent:
    def __init__(self):
        self.llm = ChatGroq(...)  # Groq LLM
        self.tools = [...]        # Agent-specific tools
        
    def process_command(self, user_input: str) -> Dict:
        # 1. Parse command
        # 2. Execute action
        # 3. Return result
        pass

📚 API Reference

REST Endpoints

`POST /process-command`

Process a text or voice command.

Request:

{
  "command": "Send WhatsApp to Jay: Hello!",
  "use_voice": false
}

Response:

{
  "success": true,
  "message": "WhatsApp message ready for Jay!",
  "agent": "whatsapp",
  "data": {
    "whatsapp_url": "https://wa.me/911234567890?text=Hello!"
  }
}

`POST /tts`

Convert text to speech.

Request:

{
  "text": "Hello from SwarAI!",
  "engine": "edge"
}

Response:

{
  "success": true,
  "audio_file": "path/to/audio.mp3"
}

`GET /health`

Check API health status.

Response:

{
  "status": "healthy",
  "version": "2.0.0",
  "agents_loaded": 13,
  "llm_available": true
}

`GET /agents`

List all available agents.

Response:

{
  "agents": [
    {
      "name": "WhatsApp Agent",
      "status": "active",
      "capabilities": ["send_message", "contact_search"]
    },
    ...
  ]
}

WebSocket Events

Client → Server

{
  "type": "command",
  "data": {
    "command": "Hello SwarAI!"
  }
}

Server → Client

{
  "type": "response",
  "data": {
    "message": "Hello! How can I help you?",
    "agent": "conversation"
  }
}

🛠️ Development

Project Structure

SwarAI/
├── backend/
│   ├── agents/                 # AI Agents
│   │   ├── __init__.py
│   │   ├── agent_manager.py    # Main coordinator
│   │   ├── whatsapp_agent.py
│   │   ├── filesearch_agent.py
│   │   ├── conversation_agent.py
│   │   └── ...
│   ├── utils/                  # Utilities
│   │   ├── enhanced_speech_processor.py
│   │   ├── conversational_tts.py
│   │   ├── conversation_memory.py
│   │   └── ...
│   ├── config.py               # Configuration
│   ├── main.py                 # FastAPI server
│   ├── crew_main.py            # CrewAI server
│   ├── crew_config.py          # CrewAI configuration
│   ├── requirements.txt        # Python dependencies
│   └── .env.example            # Environment template
├── frontend/
│   ├── src/
│   │   ├── app/                # Next.js app
│   │   ├── components/         # React components
│   │   └── hooks/              # Custom hooks
│   ├── package.json
│   └── next.config.ts
├── .gitignore
└── README.md

Running Tests

# Backend tests
cd backend
pytest

# Frontend tests
cd frontend
npm test

Code Style

# Python (Black, isort)
cd backend
black .
isort .

# TypeScript (ESLint, Prettier)
cd frontend
npm run lint
npm run format

Adding a New Agent

Create agent file in backend/agents/:

from langchain.tools import BaseTool
from langchain_groq import ChatGroq

class MyAgent:
    def __init__(self):
        self.llm = ChatGroq(...)
        
    def process_command(self, user_input: str):
        # Implementation
        pass

Register in agent_manager.py:

from agents.my_agent import MyAgent

self.my_agent = MyAgent()

Add routing logic in process_command().

🐛 Troubleshooting

Common Issues

1. ModuleNotFoundError: No module named 'streamlit'

Solution:

pip install streamlit pycaw comtypes psutil screen-brightness-control

2. MongoDB Connection Failed

Solution: The app works without MongoDB (uses in-memory storage). To fix:

# Install MongoDB locally or use MongoDB Atlas
# Update MONGODB_URL in .env

3. Voice Recognition Not Working

Solution:

# Install audio dependencies
pip install pyaudio  # May need system libraries

# Windows: Download PyAudio wheel
# Linux: sudo apt-get install portaudio19-dev python3-pyaudio
# Mac: brew install portaudio

4. Frontend Won't Start

Solution:

cd frontend
rm -rf node_modules package-lock.json
npm install --legacy-peer-deps
npm run dev

5. GROQ_API_KEY Error

Solution:

Get API key from https://console.groq.com/
Add to backend/.env:
```
GROQ_API_KEY=your_actual_key_here
```

Debug Mode

Enable debug logging:

# .env
LOG_LEVEL=DEBUG
DEBUG_MODE=true

🤝 Contributing

We welcome contributions! Here's how you can help:

Ways to Contribute

Report Bugs: Open an issue with details
Suggest Features: Share your ideas
Submit Pull Requests: Fix bugs or add features
Improve Documentation: Help others understand
Share Feedback: Tell us what works and what doesn't

Development Workflow

Fork the repository
Create a feature branch:
```
git checkout -b feature/amazing-feature
```
Make your changes
Test thoroughly
Commit with clear messages:
```
git commit -m "Add amazing feature"
```
Push to your fork:
```
git push origin feature/amazing-feature
```
Open a Pull Request

Code Guidelines

Follow PEP 8 for Python
Use TypeScript for frontend
Write clear commit messages
Add tests for new features
Update documentation

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

CrewAI - Multi-agent orchestration framework
LangChain - AI application framework
Groq - Fast LLM inference
FastAPI - Modern Python web framework
Next.js - React framework
All contributors - Thank you!

📞 Support

Issues: GitHub Issues
Discussions: GitHub Discussions
Email: [Your Email]

🗺️ Roadmap

Planned Features

🎉 Recent Improvements & New Features

Version 2.4 - UX Improvements

🔊 Speech Control on Page Refresh

Problem: Speech would continue playing in background after page refresh.

Solution: Added cleanup handlers to stop speech immediately on:

Page refresh
Navigation away
Tab close
Component unmount

Result: Clean, professional user experience with controllable audio.

📝 Concise Introduction Responses

Problem: Introduction responses were too long (45+ seconds).

Solution: Shortened to 2-3 sentences maximum with key capabilities only.

Result: Quick, clear introductions (~10 seconds) without overwhelming users.

Version 2.3 - AI Grammar Correction

✨ AI-Powered WhatsApp Message Grammar

Automatically improves grammar in all WhatsApp messages:

Features:

Capitalizes first letter of sentences
Adds proper punctuation (. ? !)
Fixes grammar mistakes naturally
Preserves conversational tone
Smart context-aware punctuation

Examples:

"how are you" → "How are you?"
"i am coming home" → "I am coming home."
"meeting at 5" → "Meeting at 5."
"gonna be late" → "Gonna be late."

Workflow Integration:

Parse Command → AI Grammar Correction → Search Contact → Generate URL → Send

Version 2.2 - Smart Features

📁 Smart "Latest File" Detection

When you specify only a file type (without filename), SwarAI opens the most recent file of that type from Downloads.

Examples:

"open pdf" → Opens newest PDF from Downloads
"open word" → Opens latest .docx from Downloads
"open excel" → Opens latest .xlsx from Downloads
"open powerpoint" → Opens latest .pptx from Downloads

Specific file search still works:

"open NPTEL certificates" → Finds and opens specific file

Version 2.1 - Enhanced Speech Quality

🗣️ File Path Filtering

Problem: SwarAI was reading file paths in speech.

Solution: Intelligent removal of:

Windows paths: C:\Users\...
Unix paths: /home/user/...
Path patterns: Path: ...
Special symbols with paths

Result:

Before: "Successfully opened: NPTEL.pdf Path: C:\Users\Shashank Gupta\Downloads\NPTEL.pdf"
After: "Opened NPTEL.pdf"

👋 Natural Greeting Flow

Problem: Redundant "Got it!" before greeting responses.

Solution: Skip acknowledgment for greetings (hi, hello, hey, good morning, etc.)

Result:

Before: "Got it!" → pause → "Hello, I'm SwarAI..."
After: "Hello, I'm SwarAI, nice to meet you..."

Version 2.0 - Major UX Improvements

🔍 Fuzzy Contact Name Matching

Problem: Required exact contact names, failing for variations.

Solution: Intelligent fuzzy matching that handles:

Partial names: "Shivam" → Finds "Shivam Patel"
Common suffixes: "clg", "college", "mam", "sir", "bro", "sis"
First name only: "Jay" → Finds "Jay Sharma"
Substring matching

Examples:

✅ "Shivam clg" → Finds "Shivam Patel"
✅ "Gitanjali mam" → Finds "Gitanjali"
✅ "Jay" → Finds "Jay Sharma"

🎤 Intelligent Speech Filtering

Problem: SwarAI was reading URLs and technical content.

Solution: Smart filtering that removes:

URLs: https://wa.me/...
Phone numbers: +919876543219
Technical instructions: "Click the link to send"
Keeps full text in chat for visual reference

Result:

Before: "WhatsApp message ready for Gitanjali! Click the link to send: https://wa.me/+919876543219?text=..."
After: "WhatsApp message ready for Gitanjali. Opening WhatsApp now."

📏 Context-Aware Speech Length

Problem: Speech cut off at 200 characters regardless of content.

Solution: Smart length limits based on content type:

Agent Type	Max Length	Purpose
Conversation	1000 chars	Full responses
WebSearch	1000 chars	Complete information
WhatsApp	100 chars	Brief confirmations
FileSearch	100 chars	Quick feedback
Email	100 chars	Action confirmations
Payment	100 chars	Transaction confirmations
Default	500 chars	Balanced

Result: Full educational content and conversations, brief action confirmations.

🎯 Complete Filtering & Enhancement Pipeline

The speech system now applies intelligent filtering:

Remove Emojis - Clean visual symbols
Remove URLs - No "https colon slash slash"
Remove File Paths - No directory structures
Remove Technical Patterns - No wa.me links, phone numbers
Remove Instructions - No "Click here" messages
Agent-Specific Simplification - Context-aware brevity
Context-Aware Length Limiting - Appropriate for content type

🧪 Test the New Features

WhatsApp with Fuzzy Matching:

"send WhatsApp to Shivam clg that AI is working"
"message Gitanjali mam hello"
"WhatsApp Jay that meeting at 5"

Smart File Opening:

"open pdf" → Opens latest PDF
"open NPTEL certificates" → Opens specific file

Natural Conversations:

"hello" → Direct greeting (no "Got it!")
"give me details about Harappan civilization" → Full response

Made with ❤️ by the SwarAI Team

⭐ Star us on GitHub if you find this helpful!

⬆ Back to Top

Name		Name	Last commit message	Last commit date
backend		backend
frontend		frontend
.gitignore		.gitignore
AUTHENTICATION_GUIDE.md		AUTHENTICATION_GUIDE.md
BACKEND_FIXES.md		BACKEND_FIXES.md
IMPLEMENTATION_SUMMARY.md		IMPLEMENTATION_SUMMARY.md
QUICKSTART.md		QUICKSTART.md
README.md		README.md
start-swarai.bat		start-swarai.bat

Latest commit

History

Folders and files

Repository files navigation

🤖 SwarAI

Multi-Agent AI Task Automation Assistant

A sophisticated multi-agent AI system powered by CrewAI, LangChain, and Groq LLM

📖 Table of Contents

🌟 Overview

Key Highlights

✨ Features

🎯 Core Capabilities

1. Conversational AI

2. Voice Recognition & TTS

3. WhatsApp Automation

4. File Management

5. System Control

6. Multi-Agent Orchestration

🏗️ Architecture

System Architecture

Technology Stack

Backend

Frontend

🚀 Quick Start

Prerequisites

1. Clone the Repository

2. Backend Setup

3. Frontend Setup

4. Run the Application

5. Access the Application

📦 Installation

Detailed Backend Installation

Detailed Frontend Installation

Optional Dependencies

⚙️ Configuration

Environment Variables

Core Configuration

Voice & Speech

Database

Agent Configuration

💻 Usage

Voice Commands

WhatsApp

File Search

System Control

Conversation

API Usage

Process Command

Text-to-Speech

WebSocket Connection

🤖 AI Agents

Available Agents

Agent Architecture

📚 API Reference

REST Endpoints

POST /process-command

POST /tts

GET /health

GET /agents

WebSocket Events

Client → Server

Server → Client

🛠️ Development

Project Structure

Running Tests

Code Style

Adding a New Agent

🐛 Troubleshooting

Common Issues

1. ModuleNotFoundError: No module named 'streamlit'

2. MongoDB Connection Failed

3. Voice Recognition Not Working

4. Frontend Won't Start

5. GROQ_API_KEY Error

Debug Mode

🤝 Contributing

Ways to Contribute

Development Workflow

Code Guidelines

📄 License

`POST /process-command`

`POST /tts`

`GET /health`

`GET /agents`