Air-to-Art Sculptor

Transform hand motion in the air into interpreted generative artwork using AI-powered motion analysis and image generation.

This is NOT gesture recognition. This is motion INTENT interpretation powered by Google Gemini.

Philosophy

• Motion → Features → Gemini AI → Art Prompt → Generated Image
• Same motion may produce different outputs (this is a feature)
• No fixed gesture classification - AI interprets each motion uniquely
• AI reasoning is always visible and explainable
• Graceful handling of ambiguous motion

Demo

demo.mp4

System Architecture

┌──────────────────────────┐     HTTP POST      ┌───────────────────────────────────────────┐
│   ESP8266 + MPU6050      │ ──────────────────▶│          Flask Backend                    │
│                          │    Raw IMU Data    │                                           │
│  5-sec @ 20Hz            │                    │  ┌──────────────┐  ┌─────────────────┐   │
│  Noise Filtering         │                    │  │   Motion     │  │    Gemini       │   │
│  Oversampling & Median   │                    │  │   Processor  │─▶│   Interpreter   │   │
└──────────────────────────┘                    │  │              │  │                 │   │
                                                │  │  - Accel/    │  │  - Interpret    │   │
                                                │  │    Gyro      │  │  - Generate     │   │
                                                │  │  - Filtering │  │    Prompt       │   │
                                                │  │  - Features  │  │  - Reasoning    │   │
                                                │  └──────────────┘  └────────┬────────┘   │
                                                │                             │            │
                                                │  ┌──────────────────────────▼───────┐   │
                                                │  │     Image Generator              │   │
                                                │  │  (Stability AI / Placeholder)    │   │
                                                │  └──────────────────────────────────┘   │
                                                └───────────────────────────────────────────┘
                                                                    │
                                                                    ▼
                                                ┌───────────────────────────────────────────┐
                                                │          Web Frontend                     │
                                                │                                           │
                                                │  • Real-time Motion Display   • Generated │
                                                │  • Motion Features           │ Artwork    │
                                                │  • Gemini Interpretation     • AI Prompt  │
                                                │  • Reasoning Chain           • Status     │
                                                └───────────────────────────────────────────┘

Project Structure

Team_Security_Guardz/
├── firmware/
│   └── air_to_art_sculptor/
│       ├── platformio.ini              # PlatformIO configuration
│       └── src/
│           └── main.cpp                # ESP8266 firmware with noise filtering
│
├── backend/
│   ├── app.py                          # Flask application
│   ├── motion_processor.py             # IMU data processing & feature extraction
│   ├── gemini_interpreter.py           # Google Gemini AI integration
│   ├── image_generator.py              # Image generation (Stability AI)
│   ├── motion_features.py              # Feature calculation utilities
│   ├── requirements.txt                # Python dependencies
│   ├── templates/
│   │   └── index.html                  # Web UI
│   └── static/
│       ├── css/
│       │   └── style.css
│       └── js/
│           └── app.js
│
└── README.md

Hardware Requirements

Components

• ESP8266 (NodeMCU or Wemos D1 Mini)
• MPU6050 6-axis IMU (accelerometer + gyroscope)

Wiring

MPU6050	ESP8266
VCC	3.3V
GND	GND
SDA	D2 (GPIO4)
SCL	D1 (GPIO5)

Software Setup

1. Backend Setup

# Navigate to backend directory
cd Team_Security_Guardz

# Create virtual environment
python3 -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

2. Environment Configuration

Create a .env file in the backend directory with your API keys:

# Google Gemini API (REQUIRED - for AI interpretation)
GEMINI_API_KEY="your-gemini-api-key-here"

# Stability AI (OPTIONAL - for image generation)
# Get key from: https://platform.stability.ai/
STABILITY_API_KEY="your-stability-ai-key-here"

Getting Gemini API Key:

1. Go to Google AI Studio
2. Click "Create API Key"
3. Copy the key and paste into .env file

Getting Stability AI Key (Optional):

1. Sign up at Stability AI Platform
2. Create an API key
3. Add to .env file for real image generation

3. Run Backend

# From the same directory with venv activated
python3 backend/app.py

The server will start on http://0.0.0.0:5000

4. Firmware Setup

Using PlatformIO (Recommended)

1. Install PlatformIO

   pip install platformio

2. Configure WiFi & Backend URL

   Edit firmware/air_to_art_sculptor/src/main.cpp:

   const char* WIFI_SSID = "YOUR_WIFI_SSID";
   const char* WIFI_PASSWORD = "YOUR_WIFI_PASSWORD";
   const char* BACKEND_URL = "http://YOUR_COMPUTER_IP:5000/upload_motion";

3. Build and Upload

   cd firmware/air_to_art_sculptor
   platformio run --target upload -e nodemcuv2

Using Arduino IDE (Alternative)

1. Install Board Support

   • Add to Board Manager URLs: http://arduino.esp8266.com/stable/package_esp8266com_index.json
   • Install "ESP8266 by ESP8266 Community"

2. Install Libraries (via Library Manager)

   • ArduinoJson (v6.21.3+)
   • Wire (built-in)
   • ESP8266WiFi (included)
   • ESP8266HTTPClient (included)

3. Configure and Upload

   • Edit WiFi credentials in src/main.cpp
   • Select Board: "NodeMCU 1.0" or "WEMOS D1 Mini"
   • Upload

Usage

With Hardware

1. Power on ESP8266 (connects to WiFi automatically)
2. Open http://YOUR_COMPUTER_IP:5000 in browser
3. Hold device and make a motion for 5 seconds
4. Watch Gemini interpret and generate artwork

Demo Mode (No Hardware)

1. Start backend: python app.py
2. Open http://localhost:5000
3. Use demo buttons:
   • Circular: Smooth rotational motion
   • Linear: Straight movement
   • Energetic: Fast, dynamic bursts
   • Gentle: Subtle, smooth motion
   • Random: Chaotic patterns

How It Works

1. Motion Capture (ESP8266)

Captures 5 seconds of IMU data at 20Hz with advanced noise filtering:

• 4x Oversampling: Read sensor 4 times per sample, average for stability
• Median Filter: Reject outliers without blurring sharp motions
• Motion Detection: Skip stationary periods to reduce noise
• Optimal DLPF Settings: Hardware low-pass filter configured for human gesture bandwidth

Data sent as JSON to backend with raw readings:

{
  "samples": [
    {"ax": 123, "ay": -456, "az": 16384, "gx": 10, "gy": -5, "gz": 2},
    ...
  ]
}

2. Motion Processing (Backend)

Signal Processing Pipeline:

• Calibrate accelerometer readings (convert to m/s²)
• Remove gravity using exponential filter
• Apply temporal smoothing
• Integrate to get velocity and position
• Normalize trajectories

Feature Extraction:

• Path geometry (length, efficiency, bounding box)
• Curvature analysis (mean, max, variance)
• Closure detection (open vs closed)
• Speed profile (mean, max, variance)
• Energy calculation
• Smoothness scoring
• Symmetry detection
• Dimensionality (linear/planar/3D)
• Rhythm regularity

3. AI Interpretation (Gemini)

Sends motion features to Google Gemini with structured prompt:

Interpret this motion data and describe what you perceive:
- Path geometry: [features]
- Speed profile: [features]
- Energy level: [features]
... (comprehensive feature set)

Respond with:
1. Interpretation (what the motion expresses)
2. Confidence level
3. Abstract art prompt for image generation

Gemini returns:

• Semantic interpretation (what the motion conveys)
• Abstract art prompt (detailed, creative description)
• Confidence score
• Reasoning chain

4. Image Generation

Uses the Gemini-generated prompt with Stability AI to create artwork. Falls back to placeholder SVG if image API unavailable.

API Reference

POST /upload_motion

Process motion data from ESP8266 or demo.

Request:

{
  "device_id": "esp8266_001",
  "samples": [
    {"ax": 123, "ay": -456, "az": 16384, "gx": 10, "gy": -5, "gz": 2}
  ]
}

Response:

{
  "status": "success",
  "timestamp": "2026-02-08T10:30:00",
  "motion_features": {
    "path_length": 2.45,
    "curvature_mean": 0.18,
    "speed_mean": 0.52
  },
  "ai_interpretation": {
    "interpretation": "Flowing, expressive motion with rhythmic quality...",
    "art_prompt": "Abstract digital artwork with flowing forms...",
    "confidence": 0.92
  },
  "generated_image": "data:image/png;base64,..."
}

POST /demo_motion?type={type}

Generate demo motion without hardware.

Types: circular, linear, energetic, gentle, random

GET /latest_result

Retrieve most recent processing result.

GET /health

Health check endpoint.

Configuration

Firmware (main.cpp)

Parameter	Default	Description
CAPTURE_DURATION_MS	5000	Capture window (ms)
SAMPLE_INTERVAL_MS	50	Sample interval (20Hz)
OVERSAMPLE_COUNT	4	Oversampling factor
MEDIAN_WINDOW_SIZE	3	Median filter window
MOTION_THRESHOLD	800	Motion detection threshold

Backend (Environment Variables)

Variable	Required	Description
GEMINI_API_KEY	Yes	Google Gemini API key
STABILITY_API_KEY	No	Stability AI key (for real images)
FLASK_ENV	No	Set to 'development' for debug mode

Troubleshooting

ESP8266 Issues

WiFi Connection Failed

• Verify SSID and password in source code
• Ensure 2.4GHz network (ESP8266 doesn't support 5GHz)
• Check router MAC filtering isn't blocking ESP8266

MPU6050 Not Detected

• Verify I2C wiring (SDA/SCL not swapped)
• Check MPU6050 address: 0x68 (default) or 0x69 (if AD0 pulled HIGH)
• Ensure 3.3V power (not 5V on VCC pin)
• Use serial monitor to debug I2C issues

Upload Fails

• Verify correct board selected (NodeMCU 1.0 or WEMOS D1 Mini)
• Try different COM port
• Hold GPIO0 to GND while uploading (some boards)

Backend Issues

Module Import Errors

• Activate virtual environment: source venv/bin/activate
• Reinstall dependencies: pip install -r requirements.txt

Gemini API Errors

• Verify GEMINI_API_KEY is set correctly in .env
• Check key has GenAI API enabled in Google Cloud Console
• Ensure .env file is in backend directory

Image Generation Not Working

• System falls back to placeholder SVGs (this is normal)
• Optional: Add STABILITY_API_KEY for real images
• Check API key has sufficient credits

General

Motion Looks Flat/Boring

• Try more varied motions (circular, figure-8, jabbing)
• Ensure MPU6050 is properly calibrated
• Check motion confidence in UI

Different Results for Same Motion

• This is intentional! Gemini adds semantic variation
• Motion features are interpreted, not matched

License

MIT License

Acknowledgments

• Google Generative AI team for Gemini API
• Stability AI for image generation
• ESPHome/PlatformIO teams
• Flask framework

---

Remember: This is motion INTENT interpretation, not classification. Every motion is unique and generates unique art.