🤖 Hybrid Hand+Face Controlled Lamp

An intelligent robotic lamp that combines computer vision, voice AI, and ROS2 control for natural human-robot interaction.

✨ Features

🎯 Priority-Based Tracking

• 🖐️ Hand Tracking (Primary): Follows index fingertip with MediaPipe
• 👁️ Face Tracking (Fallback): Switches to face tracking when no hand detected
• 🔄 Seamless Switching: Intelligent priority system for smooth transitions

🎤 Voice AI Integration

• 🗣️ Speech-to-Text: Deepgram real-time transcription
• 🧠 Emotion Analysis: Cerebras LLM for natural language understanding
• 🎭 Emotion Control: Dynamic lamp expressions (happy, sad, angry, confused, etc.)

🔧 Dual-Mode Operation

• 🤖 AI Mode: Automatic tracking and movement
• 🖐️ Manual Mode: Voice-activated manual teaching mode
• 🎤 Voice Commands: "AI mode" / "manual mode" for seamless switching

🔌 Hardware Integration

• ROS2 Control: Professional robot control system
• ESP32 Communication: Eye position and emotion display
• Waveshare Servo Control: Multi-axis positioning via persistent symlinks
• Camera Support: USB cameras with auto-focus optimization
• Serial Protocol: Arduino/ESP32 compound commands

🚀 Quick Start

One-Command Installation

git clone <your-repo-url>
cd Robo-Lamp_AI
./install.sh

The installation script automatically:

• ✅ Creates fresh virtual environment with all dependencies
• ✅ Detects and configures ESP32 and Servo Board hardware
• ✅ Sets up persistent device symlinks
• ✅ Configures user permissions
• ✅ Creates test and launcher scripts
• ✅ Generates comprehensive documentation

Configure API Keys

Edit hybrid_hand_face_controlled_lamp.py:

DEEPGRAM_API_KEY = "your_deepgram_key_here"
CEREBRAS_API_KEY = "your_cerebras_key_here"

Launch the System

# Interactive mode (shows output, good for development)
./run_lamp.sh

# Background service mode (logs to file, good for production)
./start_lamp_service.sh &

📋 Prerequisites

System Requirements

• Ubuntu 20.04+ / Linux
• Python 3.8+
• ROS2 Humble (optional but recommended)
• USB Camera
• ESP32 with CP2102 USB-to-UART Bridge (optional)
• Waveshare Servo Driver Board with CH340 (optional)

ROS2 Installation

# Install ROS2 Humble
sudo apt update
sudo apt install ros-humble-desktop
source /opt/ros/humble/setup.bash

# Install additional ROS2 packages
sudo apt install ros-humble-cv-bridge ros-humble-std-srvs

🔧 Hardware Automation

Serial Devices USB Automation

The installation script automatically sets up reliable communication with all serial devices:

ESP32 Lamp Controller

• 🔗 Persistent Symlink: Always available at /dev/esp32-lamp
• Function: Emotion display and eye position control
• Hardware: CP2102 USB-to-UART Bridge (VID: 10c4, PID: ea60)

Waveshare Servo Driver Board

• 🔗 Persistent Symlink: Always available at /dev/waveshare-servo
• Function: Multi-axis servo control for lamp positioning
• Hardware: QinHeng CH340 USB Serial Controller (VID: 1a86, PID: 55d3)

Automation Features

• 🚫 No More USB Port Issues: Works regardless of which USB ports you use
• ⚡ Automatic Detection: Finds devices based on hardware identifiers
• 🔐 Permission Setup: Adds user to dialout group for serial access
• 🧪 Built-in Testing: Includes communication test scripts for all devices
• 🔄 Reliable Reconnection: Handles unplugging/reconnecting automatically

Hardware Testing

# Test all devices
python3 test_all_devices.py

# Test individual devices
python3 esp32_test.py      # ESP32 only
python3 servo_test.py      # Servo board only

🎮 Usage

Voice Commands

Mode Switching

• "Manual mode" / "Switch to manual" → Enable manual teaching
• "AI mode" / "Automatic mode" → Resume automatic tracking

Emotion Control

• "I'm happy" → Happy expression
• "This is confusing" → Confused expression
• "Stop tracking" → Angry expression
• Natural speech is analyzed for emotional content

Visual Tracking

Hand Priority

1. Show your index finger → Lamp follows hand immediately
2. Hide hand → Automatically switches to face tracking
3. Show face → Lamp follows face movements

Manual Mode

1. Say "manual mode"
2. Physically move the lamp (hardware unlocks when force applied)
3. Say "AI mode" to resume automatic tracking

🔧 Configuration

Camera Settings

CAMERA_INDEX = 2        # USB camera index
CAMERA_WIDTH = 640      # Resolution width
CAMERA_HEIGHT = 480     # Resolution height
TARGET_FPS = 30         # Frame rate

API Configuration

DEEPGRAM_API_KEY = "your_key"    # Speech-to-text
CEREBRAS_API_KEY = "your_key"    # LLM emotion analysis

ROS2 Topics

• Published: /servo_node/delta_twist_cmds (lamp movement)
• Published: /camera/image_raw (camera feed)
• Subscribed: /joint_states (current lamp position)
• Service: /lamp/set_manual_mode (mode switching)

Serial Communication

Format: #idle#position#curiosity#emotion#
Example: #off#DEFAULT#off#happy#

📁 Project Structure

Robo-Lamp_AI/
├── hybrid_hand_face_controlled_lamp.py  # Main application
├── hand_tracking_controlled_lamp.py     # Original hand tracking  
├── voice_camera_controlled_lamp.py      # Original face tracking
├── requirements.txt                     # Python dependencies
├── install.sh                          # Comprehensive installation script
├── run_lamp.sh                         # Interactive launcher (auto-generated)
├── start_lamp_service.sh               # Background service launcher
├── test_all_devices.py                 # Combined hardware test
├── esp32_test.py                       # ESP32 communication test
├── servo_test.py                       # Servo board communication test
├── config.example.py                   # Configuration template
├── setup.py                            # Python package setup
├── venv/                               # Virtual environment (created by install.sh)
├── image-analysis/                     # Gemini AI image analysis tool
│   ├── src/
│   │   ├── index.js                    # CLI interface
│   │   └── imageAnalyzer.js            # Core analysis functionality
│   ├── images/                         # Input images
│   ├── output/                         # Processed images
│   └── package.json                    # Node.js dependencies
└── README.md                           # This file

🛠️ Development

Running in Development Mode

# Activate environment
source venv/bin/activate
source /opt/ros/humble/setup.bash

# Run with custom Arduino port
python3 hybrid_hand_face_controlled_lamp.py /dev/ttyUSB1

# Run with debug output
ROS_LOG_LEVEL=DEBUG python3 hybrid_hand_face_controlled_lamp.py

Key Components

Priority Tracking System

# Hand detection has priority
if hand_detected:
    mode = "hand"
    target = hand_center
elif face_detected:
    mode = "face" 
    target = face_center
else:
    mode = "search"

Voice Command Processing

# Mode switching checked before emotion analysis
async def process_voice_command(transcript):
    if await check_mode_commands(transcript):
        return  # Mode command processed
    
    # Continue with emotion analysis
    emotion = await analyze_emotion_with_cerebras(transcript)

Movement Control

# Respects manual mode setting
def publish_velocity_command(self, x, y, z):
    if not self.should_move_lamp():
        return  # Blocked in manual mode
    
    # Publish ROS2 TwistStamped message

🔍 Troubleshooting

Camera Issues

# Check available cameras
ls /dev/video*

# Test camera access
python3 -c "import cv2; cap = cv2.VideoCapture(2); print('Camera OK' if cap.isOpened() else 'Camera Failed')"

Hardware Communication Issues

# Check all persistent symlinks
ls -la /dev/esp32-lamp /dev/waveshare-servo

# Test all devices
python3 test_all_devices.py

# Check if devices are detected
lsusb | grep -E "10c4:ea60|1a86:55d3"  # ESP32 and Servo Board

# Reload udev rules (after connecting devices)
sudo udevadm control --reload-rules
sudo udevadm trigger

# Check user permissions
groups  # Should include 'dialout'

Serial Device Troubleshooting

Symlinks Missing

# Reload udev rules
sudo udevadm control --reload-rules
sudo udevadm trigger

# Monitor udev events
sudo udevadm monitor

Permission Issues

# Check group membership
groups

# Add to dialout group if needed
sudo usermod -a -G dialout $(whoami)
newgrp dialout  # Apply immediately

Communication Issues

# Test with different baud rates
python3 -c "
import serial
with serial.Serial('/dev/esp32-lamp', 9600) as s:
    print('9600 baud OK')
with serial.Serial('/dev/esp32-lamp', 115200) as s:
    print('115200 baud OK')
"

MediaPipe Installation

# Reinstall MediaPipe
pip uninstall mediapipe
pip install mediapipe==0.10.21

ROS2 Connection

# Check ROS2 nodes
ros2 node list

# Check topics
ros2 topic list

# Test service
ros2 service call /lamp/set_manual_mode std_srvs/srv/SetBool "data: true"

Audio Issues

# Check audio devices
python3 -c "import pyaudio; p = pyaudio.PyAudio(); print(f'Audio devices: {p.get_device_count()}')"

# Test microphone
arecord -l

Python Environment Issues

# Recreate virtual environment
rm -rf venv
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Raspberry Pi Camera Setup (Ubuntu 22.04)

For CSI ribbon camera on Raspberry Pi running Ubuntu 22.04:

Boot Configuration

Edit /boot/firmware/config.txt:

# Enable camera
start_x=1
gpu_mem=128

# Camera module overlay (adjust for your camera)
dtoverlay=ov5647  # For Pi Camera v1
# dtoverlay=imx219  # For Pi Camera v2
# dtoverlay=imx708  # For Pi Camera v3

Package Installation

# Install libcamera and tools
sudo apt update
sudo apt install libcamera-apps libcamera-tools

# Install Python libraries
pip install picamera2 opencv-python

Testing

# Test camera detection
libcamera-hello --list-cameras

# Capture test image
libcamera-still -o test.jpg

🔧 Image Analysis Tool

The repository includes a Gemini AI-powered image analysis tool:

Setup

cd image-analysis
npm install
cp .env.example .env
# Add your GOOGLE_API_KEY to .env

Usage

# General image analysis
npm start images/photo.jpg

# Focus on specific objects
npm start images/photo.jpg "notebook"
npm start images/photo.jpg "phone"

Features

• General Image Description: Comprehensive descriptions
• Object Detection: Automatic detection with bounding boxes
• Focused Analysis: Crop and analyze specific objects
• Text Recognition: OCR and text analysis

📊 System Scripts Reference

Installation & Setup

• ./install.sh - Comprehensive installation (Python + Hardware + Testing)
• python3 verify_setup.py - Verify installation completeness

Running the System

• ./run_lamp.sh - Interactive mode (development, shows output)
• ./start_lamp_service.sh - Background service (production, logs to file)

Hardware Testing

• python3 test_all_devices.py - Test all hardware
• python3 esp32_test.py - Test ESP32 only
• python3 servo_test.py - Test servo board only

Logs & Monitoring

# Check service logs
tail -f /home/$(whoami)/lamp_service.log

# Monitor hardware detection
sudo udevadm monitor

# Check system status
systemctl status  # If running as systemd service

🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/amazing-feature)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing-feature)
5. Open a Pull Request

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

• MediaPipe for hand and face tracking
• ROS2 for robot control framework
• Deepgram for speech-to-text
• Cerebras for LLM emotion analysis
• OpenCV for computer vision
• Google Gemini AI for image analysis

📞 Support

For issues and questions:

• 🐛 Bug Reports: Open an issue on GitHub
• 💬 Discussions: Use GitHub Discussions
• 📧 Direct Contact: [Your Contact Info]

---

Built with ❤️ for natural human-robot interaction