Skip to content
BranchesTags

Repository files navigation

Table of Contents

Overview

This project consists of three progressive tasks that demonstrate the implementation of reinforcement learning for robot control:

  • Task 0: Basic robot interface setup and demonstration
  • Task 1: Discrete action space reinforcement learning using Soft Actor-Critic (SAC)
  • Task 2: Continuous action space with vision transformer (ViT) for food detection and collection

Each task builds upon the previous one, introducing more sophisticated techniques and capabilities.

Project Structure

Learning_machine/
├── task0/          # Basic setup and robot interface
├── task1/          # SAC with discrete actions
├── task2/          # SAC with continuous actions + ViT vision
├── .venv/          # Python virtual environment
└── README.md       # This file

Each task directory contains:

  • catkin_ws/ - ROS workspace with learning_machines package
  • scenes/ - CoppeliaSim simulation scenes
  • scripts/ - Setup and run scripts
  • results/ - Training results and model checkpoints
  • requirements.txt - Python dependencies
  • Dockerfile - Docker configuration for ROS environment

Prerequisites

Required Software

  1. Python 3.8+

    python3 --version  # Should be 3.8 or higher

  2. Docker Desktop

    • Download from Docker Desktop
    • For Apple Silicon Macs: Enable experimental features and set CPU limit to 1

  3. CoppeliaSim (Educational version)

  4. Virtual Environment

    python3 -m venv .venv
source .venv/bin/activate

System Requirements

  • macOS (Intel or Apple Silicon), Linux, or Windows
  • Minimum 8GB RAM (16GB recommended)
  • Docker Desktop with sufficient resources allocated

Task 0: Basic Setup and Robot Interface

Purpose: Establish the foundation for robot interaction with both simulation and hardware.

Features

  • Basic robot interface setup using robobo_interface
  • Connection to CoppeliaSim simulator
  • Connection to physical Robobo hardware (via ROS)
  • Demonstration of basic robot movements and sensor readings
  • IR sensor reading and camera access

Key Components

  • robobo_interface: Abstraction layer for hardware/simulation
  • Basic action scripts demonstrating robot capabilities
  • Multiple simulation scenes for testing

Running Task 0

  1. Setup IP Address:

    cd task0
# Get your IP address
ipconfig getifaddr en0  # macOS
# Update scripts/setup.bash with your IP

  2. Start CoppeliaSim:

    zsh ./scripts/start_coppelia_sim.zsh ./scenes/Robobo_Scene.ttt

  3. Run the Code:

    # For Intel Mac or Linux
bash ./scripts/run.sh --simulation

# For Apple Silicon Mac
zsh ./scripts/run_apple_sillicon.zsh --simulation

Task 1: Reinforcement Learning with SAC

Purpose: Implement reinforcement learning for obstacle avoidance using discrete action spaces.

Features

  • Soft Actor-Critic (SAC) algorithm implementation
  • Discrete action space: 6 primitive movements
  • State space: 8 normalized IR sensor values
  • Gymnasium environment wrapper (RoboboIREnv)
  • Experience replay buffer
  • Training statistics and model checkpointing

Algorithm Details

  • Agent: SAC with discrete actions using Gumbel-Softmax
  • Network Architecture:
    • Actor: 2 hidden layers (64 neurons each)
    • Critic: 2 Q-networks with target networks
  • Training: 150 episodes, 10 steps per episode
  • Exploration: Epsilon-greedy with decay

Key Files

  • agent.py: SAC agent implementation
  • env.py: Gymnasium environment wrapper
  • train.py: Training script
  • validation.py: Model evaluation script

Running Task 1

  1. Train the Model:

    cd task1
# Start CoppeliaSim first
zsh ./scripts/start_coppelia_sim.zsh ./scenes/arena_approach.ttt

# Run training (in another terminal)
bash ./scripts/run.sh --simulation

  2. Validate Trained Model:

    # Edit validation.py to set MODEL_PATH
python3 catkin_ws/src/learning_machines/scripts/validation.py --simulation

Results

Training results are saved in task1/results/ including:

  • Model checkpoints (.h5 files)
  • Training statistics (JSON)
  • Training plots (.png)
  • Validation results

Task 2: Vision-Based Food Collection

Purpose: Extend Task 1 with continuous actions and vision transformer for food detection.

Features

  • Continuous action space: Normalized wheel speeds [-1, 1]
  • Vision Transformer (ViT): Zero-shot image classification using CLIP
  • Enhanced state space:
    • 8 IR sensor values
    • 3 green detection bands (horizontal image analysis)
    • 1 ViT food score
  • Food collection task: Collect green/red blocks in simulation
  • Reward function: Combines food collection, collision avoidance, and movement

Algorithm Details

  • Agent: SAC with continuous actions (Gaussian policy)
  • Vision: CLIP-ViT for zero-shot food detection
  • State Processing: Multi-modal (IR sensors + image analysis)
  • Action Space: Continuous [left_speed, right_speed]

Key Improvements Over Task 1

  1. Continuous Control: Smooth wheel speed control instead of discrete movements
  2. Vision Integration: Real-time food detection using transformer models
  3. Multi-modal State: Combines proprioceptive (IR) and exteroceptive (vision) sensing
  4. Task Complexity: Food collection with reward shaping

Running Task 2

  1. Train the Model:

    cd task2
# Start CoppeliaSim with food collection scene
zsh ./scripts/start_coppelia_sim.zsh ./scenes/arena_approach_sparse.ttt

# Run training
bash ./scripts/run.sh --simulation

  2. Test Vision Detection:

    # Test ViT food detection
python3 catkin_ws/src/learning_machines/scripts/test_vit.py

Environment Details

  • Observation Space: Box(0.0, 1.0, shape=(12,))
    • 8 IR sensors (normalized)
    • 3 green detection bands
    • 1 ViT food score
  • Action Space: Box(-1.0, 1.0, shape=(2,))
    • Left wheel speed
    • Right wheel speed
  • Reward Components:
    • +10.0 per food item collected
    • -5.0 for collisions
    • +0.2 for forward movement
    • +0.5 for green detection
    • +0.5 for ViT food score
    • -0.1 for obstacle proximity

Quick Start

Complete Setup (macOS)

  1. Clone and Setup:

    git clone <repository-url>
cd Learning_machine
python3 -m venv .venv
source .venv/bin/activate

  2. Install CoppeliaSim:

    • Download CoppeliaSim for your platform
    • Copy to task0/coppeliaSim.app, task1/coppeliaSim.app, task2/coppeliaSim.app

  3. Configure IP Address:

    # Get your IP
ipconfig getifaddr en0

# Update in each task
# Edit task0/scripts/setup.bash
# Edit task1/scripts/setup.bash
# Edit task2/scripts/setup.bash

  4. Run Task 0 (Basic Setup):

    cd task0
zsh ./scripts/start_coppelia_sim.zsh ./scenes/Robobo_Scene.ttt
# In another terminal:
bash ./scripts/run.sh --simulation

  5. Run Task 1 (RL Training):

    cd task1
zsh ./scripts/start_coppelia_sim.zsh ./scenes/arena_approach.ttt
bash ./scripts/run.sh --simulation

  6. Run Task 2 (Vision RL):

    cd task2
zsh ./scripts/start_coppelia_sim.zsh ./scenes/arena_approach_sparse.ttt
bash ./scripts/run.sh --simulation

Results

Task 1 Results

Located in task1/results/:

  • Multiple training runs with different configurations
  • Model checkpoints saved every 20 episodes
  • Training statistics showing:
    • Episode rewards
    • Episode lengths
    • Loss curves
    • Exploration rate decay

Task 2 Results

Located in task2/results/:

  • Vision transformer test outputs
  • State visualization images
  • Food detection examples

Troubleshooting

Common Issues

  1. CoppeliaSim Connection Failed

    • Check IP address in scripts/setup.bash
    • Ensure CoppeliaSim is running before starting Docker
    • Verify network connectivity

  2. Docker Build Fails

    • Ensure Docker Desktop is running
    • Check available disk space (Docker needs ~10GB)
    • For Apple Silicon: Set CPU limit to 1 in Docker settings

  3. Python Import Errors

    • Activate virtual environment: source .venv/bin/activate
    • Install requirements: pip install -r requirements.txt
    • Check Python version (3.8+ required)

  4. Hardware Connection Issues

    • Verify ROS_MASTER_URI in setup.bash
    • Ensure phone and computer are on same network
    • Check Robobo app is running on phone

  5. Vision Transformer Not Working

    • Install transformers library: pip install transformers
    • Check internet connection (downloads model on first use)
    • Verify image capture is working

Getting Help

  • Check individual task READMEs in task0/README.md, task1/README.md, task2/README.md
  • Review quick_setup_macos.md in each task directory
  • Check Docker logs: docker logs <container_id>

Additional Resources

License

This project is part of the Learning Machines course. See individual task directories for specific licensing information.

About

A comprehensive project implementing reinforcement learning algorithms for the Robobo robot platform, progressing from basic setup to advanced vision-based food collection tasks.

Resources

Readme
License
Activity

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

Packages

Contributors

Languages