Skip to content

Releases: dimensionalOS/dimos

Release list

Release v0.0.5: Pre-Launch Release

Release v0.0.5: Pre-Launch Release

Choose a tag to compare

View all tags
@spomichter spomichter released this 28 Oct 04:59
v0.0.5
fffc1d4

Test pre-release changelog

What's Changed

Read more

Contributors

leshy, paul-nechifor, and 5 other contributors
Loading

Release v0.0.4: ClaudeAgent thinking models with new physical RobotSkills, vector SpatialMemory for emergent world reasoning, major new RobotSkills

Release v0.0.4: ClaudeAgent thinking models with new physical RobotSkills, vector SpatialMemory for emergent world reasoning, major new RobotSkills

Choose a tag to compare

View all tags
@spomichter spomichter released this 10 May 02:26
v0.0.4
10e1688

🚀 The Dimensional Framework v0.0.4

The universal framework for AI-native generalist robotics

🧠 Core Enhancement Details

🗺️ Spatial Memory System

SpatialMemory gives robots an emergent understanding of the physical world via a rich embedding and associated metadata. This includes temporality, world geometry, object semantics, physical characteristics, and more.

Key Files and Classes:

  • /dimos/types/robot_location.py - New structured RobotLocation type
  • /dimos/agents/memory/spatial_vector_db.py - Vector database implementation for spatial memory
  • /dimos/agents/memory/image_embedding.py - CLIP-based visual embedding for image similarity
  • /dimos/perception/spatial_perception.py - Semantic spatial perception implementation

Notable Changes:

  • Extracted SpatialMemory as a standalone class with modular architecture (PR #264)
  • Implemented multi-modal querying by text, image, or location with semantic matching
  • Added chromaDB persistence with support for existing memory loading and new memory creation
  • Introduced frame filtering based on distance and time thresholds for improved memory quality
  • Added rotation vector support as VectorDB metadata for more accurate retrieval (PR #216)
  • Implemented RobotLocation tracking to associate names with coordinates
  • Created reactive stream processing API for continuous memory building from video
  • Added support for semantic text queries for spatial navigation (e.g., "where is the kitchen")

💭 Claude Agent Thinking

Implemented ClaudeAgent with support for continuous thinking blocks with real-time visualization and parallel tool execution. Claude 3.7 thinking models now allow for incredible performance in general spatial reasoning and planning of Robot Skill action primitives, enabling sophisticated skill orchestration, planning, and reasoning capabilities.

Key Files and Classes:

  • /dimos/agents/claude_agent.py - Streaming API integration for continuous thinking blocks
  • /assets/agent/prompt.txt - Master dimOS prompt for Claude agent

Notable Changes:

  • Implemented streaming architecture with real-time thinking block visualization (PR #200)
  • Added thinking_budget_tokens parameter for controlling Claude's reasoning depth
  • Developed continuous writing to memory.txt as thinking and response chunks arrive
  • Created ResponseMessage class with support for thinking_blocks and tool_calls
  • Built event handling for streaming API responses
  • Parallel/Concurrent tool calling supported with lock on conversation_history

👁️ Object Detection Stream

Added unified object detection streaming with support for both YOLO and Detic backends, enabling real-time perception integration with LLM agents.

Key Files and Classes:

  • /dimos/perception/object_detection_stream.py - Main ObjectDetectionStream implementation
  • /dimos/models/Detic/ - Added Detic object detection model
  • /dimos/perception/detection2d/detic_2d_det.py - Detic detector implementation
  • /dimos/perception/detection2d/yolo_2d_det.py - YOLO detector implementation
  • /tests/test_object_detection_stream.py - Test file for object detection stream

Notable Changes:

  • Added Detic and YOLO support to ObjectDetectionStream (PR #243, PR #239)
  • Implemented get_formatted_stream() for easier agent interpretation (PR #261)
  • Added integration with LLMAgent via input_data_stream parameter to allow for Agent

Implementation:

object_detector = ObjectDetectionStream(
    camera_intrinsics=robot.camera_intrinsics,
    min_confidence=min_confidence,
    class_filter=class_filter,
    transform_to_map=robot.ros_control.transform_pose,
    detector=detector,
    video_stream=video_stream
)
object_stream = object_detector.get_stream()

Agent Integration:

agent = LLMAgent(
    input_data_stream=object_detection_stream,
    # other parameters...
)

🧩 Skills Architecture

Major skills refactoring with standardized interfaces for movement, perception, and navigation.

Key Files and Classes:

  • /dimos/skills/ - New centralized skills directory
  • /dimos/skills/navigation.py - Navigation skills implementation
  • /dimos/skills/kill_skill.py - Skill to terminate running skills
  • /dimos/skills/observe_stream.py - Stream observation skill
  • /dimos/skills/speak.py - Text-to-speech skill
  • /dimos/skills/visual_navigation_skills.py - Visual navigation skills
  • /dimos/skills/rest/rest.py - REST API integration skills

Notable Changes:

  • Skills refactor (PR #154) with standardized interface
  • Added GenericRestSkill for Basic GET/POST Requests (PR #225)
  • Added Speak() skill with enhanced TTS (PR #233)
  • Created ObserveStream and KillSkills for Claude thinking agent (PR #183)

🤖 New RobotSkills

🧭 Navigation Skills

  • NavigateWithText (/dimos/skills/navigation.py): General semantic navigation command, uses both SpatialMemory and
  • NavigateToGoal (/dimos/skills/navigation.py): Navigates to specific coordinates
  • GetPose (/dimos/skills/navigation.py): Gets current robot pose

🏃‍♂️ Movement Skills

  • Move (/dimos/robot/unitree/unitree_skills.py): Forward movement using velocity commands
  • Reverse (/dimos/robot/unitree/unitree_skills.py): Backward movement using velocity commands
  • SpinLeft (/dimos/robot/unitree/unitree_skills.py): Rotation using degree commands
  • SpinRight (/dimos/robot/unitree/unitree_skills.py): Rotation using degree commands
  • Wait (/dimos/robot/unitree/unitree_skills.py): Pauses execution for specified time

👁️ Perception Skills

  • ObserveStream (/dimos/skills/observe_stream.py): Streams observations to agent
  • FollowHuman (/dimos/skills/visual_navigation_skills.py): Person tracking and following

🛑 Management Skills

  • KillSkill (/dimos/skills/kill_skill.py): Terminates running skills safely

🔌 API Integration

  • GenericRestSkill (/dimos/skills/rest/rest.py): GET/POST requests to external APIs

🗣️ Interaction

  • Speak (/dimos/skills/speak.py): Text-to-speech with enhanced TTS support

🧭 Navigation & Planning Details

🛣️ Symbolic Navigation

Integrated global and local planners with path tracking and goal orientation, featuring visual navigation to any object and native 2D mapping.

Key Files and Classes:

  • /dimos/robot/global_planner/ - Global path planning implementation
  • /dimos/robot/local_planner/ - Local path planning with VFH algorithm
  • /dimos/robot/local_planner/local_planner.py - Base local planner class
  • /dimos/robot/local_planner/vfh_local_planner.py - VFH local planner implementation
  • /dimos/types/costmap.py - Costmap implementation for planning
  • /dimos/types/path.py - Path representation types
  • /dimos/types/vector.py - Native vector type implementation

Notable Changes:

  • Improved A* implementation with more conservative parameters (PR #226)
  • Introduced navigate to anything in camera view, using Qwen as backbone, falling back to memory map if no object found in frame
  • Integrated VFH+ for obstacle avoidance with pure pursuit controller for path tracking
  • Implemented dimOS native 2D mapping and global planning (moving away from ROS/Nav2)
  • Created dimOS native typing for common data structures (Vector, Costmap, etc.)

🔍 Semantic Navigation

Navigate to named locations or objects using natural language queries.

Key Files and Classes:

  • /dimos/skills/navigation.py - Implemented BuildSemanticMap and Navigate skills
  • /dimos/skills/visual_navigation_skills.py - Visual navigation implementation

Notable Changes:

  • Added NavigateWithText skill (formerly Navigate) for language-based navigation
  • Added GetPose and NavigateToGoal skills (PR #229)
  • Fixed goal theta orientation for Navigation (PR #227)
  • Added Use metric3d for distance estimate for navigate to object skill (PR #235)

⚙️ Hardware & Performance Details

🖥️ Jetson Support

Added compatibility for NVIDIA Jetson with Jetpack 6.2 and CUDA 12.6.

Key Files and Classes:

  • /docker/jetson/ - Jetson-specific Docker configuration
  • /docker/jetson/huggingface_local/ - HuggingFace models on Jetson
  • /tests/test_agent_huggingface_local_jetson.py - Jetson-specific tests

Notable Changes:

  • Added working Jetson Pytorch/torchvision wheels for CUDA 12.6
  • Created Jetson-specific Dockerfile and Docker Compose files
  • Added fix_jetson.sh script for ARM/import issues

💾 Model Persistence

Added Docker volume caching for ML models to prevent repeated downloads.

Key Files and Classes:

  • /docker/unitree/agents_interface/docker-compose.yml - Volume configuration

Notable Changes:

  • Added three persistent volumes:
    • torch-hub-cache: For PyTorch Hub models (Metric3D)
    • iopath-cache: For Detic models
    • ultralytics-cache: For YOLO models
  • Mounted to respective cache directories to persist downloaded models

🛠️ Developer Experience Details

📊 Visualization

Improved real-time visualization for robot position and planning.

Key Files and Classes:

  • /dimos/web/websocket_vis/ - WebSocket visualization implementation
  • /dimos/web/websocket_vis/server.py - Visualization server

Notable Changes:

  • Added WebSocket visualization system (PR #198)
  • Improved visualization API to be realtime (PR #207)
  • Cleaner global planner API with faster rendering (PR #210)

Released on May 8, 2025

What's Changed

  • Update supervisord.conf to output dimos logs to regular terminal by @lukasapaukstys in #153
  • Feature: SentenceTransformers implemented as local embedding model for AgentMemory by @spomichter in #166
  • DIM-136: Jetson support for running local Agents, models on Jetpack 6.2, CUDA ...
Read more

Contributors

leshy, spomichter, and 2 other contributors
Loading

Release v0.0.3: Local Models via CTransformers (GGUF) and HF + Object tracking and Semantic Segmentation with YOLO, Qwen2.5-VL, Metric3D, OpenCV + TTS/STT Support

Release v0.0.3: Local Models via CTransformers (GGUF) and HF + Object tracking and Semantic Segmentation with YOLO, Qwen2.5-VL, Metric3D, OpenCV + TTS/STT Support

Choose a tag to compare

View all tags
@spomichter spomichter released this 08 May 13:25
v0.0.3
ebd3760

Enable Local & Remote Hugging Face and GGUF Ctransformer Agents (GPU-Ready)

Introduces Two New Agent Classes:

By @lukasapaukstys

  • HuggingFaceLocalAgent: Provides local inference capabilities using Hugging Face models. Supports GPU acceleration and is optimized for execution within Docker environments.
    Fully tested with: ./run.sh hf-local
  • HuggingFaceRemoteAgent: Enables remote inference via the Hugging Face API. Functionality is endpoint-dependent.
    Fully tested with: ./run.sh hf-remote
  • CTransformersGGUFAgent: Enables local inference via the CTransformers.
    Fully tested with: ./run.sh gguf

Running the Agents Locally

To run the agents in a Docker container with GPU (CUDA) support:

  1. Comment out all lines in dimos/robot/__init__.py to disable default initialization.

  2. (HuggingFace Local) From the project root, run:

    ./run.sh hf-local

    (GGUF Local) From the project root, run:

    ./run.sh gguf

Other Changes

  • Added licensing headers to newly created files and any existing files that were missing them.
  • Added a sample video to the assets folder: assets/trimmed_video_office.mov
  • Added a convenience run.sh shell script to the root directory.

Object tracking and Semantic Segmentation with YOLO, Qwen2.5-VL, Metric3D, OpenCV

By @alexlin2

Changes:

  • Introduces person following and semantic segmentation in dimos/perception
  • Integrates semantic segmentation, monocular depth, rich labels to Agent stack as observable streams

TTS/STT Audio Integrations to Agent stack

By @leshy

Changes

  • Created audio stack in stream/audio with OpenAI whisper powered TTS/STT
  • Text streaming out as Observable for consumption by agents or other processes
  • Modular pipeline
def stt():
    # Create microphone source, recorder, and audio output
    mic = SounddeviceAudioSource()
    normalizer = AudioNormalizer()
    recorder = KeyRecorder(always_subscribe=True)
    whisper_node = WhisperNode()  # Assign to global variable

    # Connect audio processing pipeline
    normalizer.consume_audio(mic.emit_audio())
    recorder.consume_audio(normalizer.emit_audio())
    monitor(recorder.emit_audio())
    whisper_node.consume_audio(recorder.emit_recording())

    user_text_printer = TextPrinterNode(prefix="USER: ")
    user_text_printer.consume_text(whisper_node.emit_text())

    return whisper_node


def tts():
    tts_node = OpenAITTSNode()
    agent_text_printer = TextPrinterNode(prefix="AGENT: ")
    agent_text_printer.consume_text(tts_node.emit_text())

    response_output = SounddeviceAudioOutput(sample_rate=24000)
    response_output.consume_audio(tts_node.emit_audio())

    return tts_node

Full Changelog: https://github.com/dimensionalOS/dimos/commits/v0.0.3

Contributors

leshy, lukasapaukstys, and alexlin2
Loading

Release v0.0.2: ClaudeAgent, Genesis Docker, Logging, and Direct Movement Velocity Support Updates

Release v0.0.2: ClaudeAgent, Genesis Docker, Logging, and Direct Movement Velocity Support Updates

Choose a tag to compare

View all tags
@spomichter spomichter released this 29 Mar 02:15
v0.0.2
d76c656

Release v0.0.2: ClaudeAgent, Genesis Docker, Logging, and Direct Movement Velocity Support Updates

Version Updates

  • pyproject.toml: 0.0.1 -> 0.0.2

ClaudeAgent

  • Implemented Claude Agent with input query streaming and system query support. Image support WIP.
  • Added skills/tool calling capabilities with thinking model integration
  • Introduced run_observable_query() helper method in base LLMAgent
  • Enhanced web interface integration for input queries
  • Improved handling of Pydantic generic classes in observable query
  • Streamlined text streaming for Agents using FastAPIServer
  • Fixed system query handling and initialization

Robot Control & Skills Framework

  • Re-implemented direct movement velocity controls in ROSControl
  • Added new Robot skill for move_vel() functionality

Logging & Infrastructure

  • Standardized logging system with DIMOS_LOG_LEVEL environment variable
  • Fixed debug message handling and logger configuration
  • Cleaned up terminal output and removed redundant logging
  • Added standardized test file headers

Docker & Simulation

  • Added support for Genesis simulator alongside Isaac
  • Created separate folders for simulation docker files
  • Updated Docker configurations for both simulators
  • Changed web server interface address from localhost to 0.0.0.0
Loading