Navigation with QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning

Paper   Preprint

Overview

This repository is the official implementation of the Quantized Planner for Hierarchical Implicit Learning (QPHIL) published at IJCNN 2025. QPHIL is a offline goal-conditioned reinforcement learning method which performs state-space discretization to plan a navigation path in the discretized space, improving long-horizon goal reaching performance.

Installation

1. Create and activate conda environment:

conda create -n qphil python=3.10 -y -q && conda activate qphil

2. Download Mujoco Download the MuJoCo version 2.1 binaries for Linux or Mac and extract the downloaded mujoco210 directory into ~/.mujoco/mujoco210.
3. Install conda dependencies:

conda install -c conda-forge mesalib=24.2.6 glew=2.1.0 glfw=3.4.0 libgcc=14.2.0 patchelf=0.17.2 -y

4. Install pip dependencies:

pip install --no-cache-dir -r requirements.txt

5. Download the datasets for AntMaze-Extreme from [link] and place them in the ~/.d4rl/datasets/ folder:

~/
└── .d4rl/
    └── datasets/
        └── Ant_maze_extreme-maze_noisy_multistart_True_multigoal_False
        └── Ant_maze_extreme-maze_noisy_multistart_True_multigoal_True

6. Setup the environment variables:

conda env config vars set PYTHONPATH="$PYTHONPATH:$PWD"
conda env config vars set D4RL_SUPPRESS_IMPORT_ERROR=1
conda env config vars set D4RL_DATASET_DIR=~/.d4rl/datasets
conda env config vars set LD_LIBRARY_PATH=~/.mujoco/mujoco210/bin
conda env config vars set LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/lib/nvidia
conda deactivate && conda activate qphil

Usage

To reproduce QPHIL's results, launch the experiments with the following commands:

# antmaze-medium-diverse-v2
python sgcrl/train.py -cn antmaze-medium env_name=antmaze-medium-diverse-v2

# antmaze-medium-play-v2
python sgcrl/train.py -cn antmaze-medium env_name=antmaze-medium-play-v2

# antmaze-large-diverse-v2
python sgcrl/train.py -cn antmaze-large env_name=antmaze-large-diverse-v2

# antmaze-large-play-v2
python sgcrl/train.py -cn antmaze-large env_name=antmaze-large-play-v2

# antmaze-ultra-diverse-v0
python sgcrl/train.py -cn antmaze-ultra env_name=antmaze-ultra-diverse-v0

# antmaze-ultra-play-v0
python sgcrl/train.py -cn antmaze-ultra env_name=antmaze-ultra-play-v0

# antmaze-extreme-diverse-v0
python sgcrl/train.py -cn antmaze-extreme env_name=antmaze-extreme-diverse-v0

# antmaze-extreme-play-v0
python sgcrl/train.py -cn antmaze-extreme env_name=antmaze-extreme-play-v0

Acknowledgments

This codebase uses code from the pytorch, vector_quantize_pytorch and d4rl library.

Citation

@INPROCEEDINGS{11227725,
  author={Canesse, Alexi and Petitbois, Mathieu and Denoyer, Ludovic and Lamprier, Sylvain and Portelas, Rémy},
  booktitle={2025 International Joint Conference on Neural Networks (IJCNN)}, 
  title={Navigation With QPHIL: Quantizing Planner for Hierarchical Implicit Q-Learning}, 
  year={2025},
  volume={},
  number={},
  pages={1-8},
  keywords={Training;Quantization (signal);Q-learning;Navigation;LoRa;Transformers;Planning;Trajectory;Robots;Signal to noise ratio;Goal Conditioned Reinforcement Learning;Offline Reinforcement Learning;Navigation;Locomotion},
  doi={10.1109/IJCNN64981.2025.11227725}}

© [2026] Ubisoft Entertainment. All Rights Reserved.