Human Behavior Recognition in Video Streams

End-to-end human behavior recognition system for video streams and MP4 clips. The project combines a FastAPI backend, React operator dashboard, PostgreSQL event history, reusable inference pipeline, Docker Compose orchestration, and documentation for demo, benchmark, and module workflows.

Quick Start

Requirements:

• Docker Desktop or Docker Engine with Docker Compose
• Local clone of this repository
• Local data folders used by Compose

Create the expected local folders when starting from a clean clone:

mkdir data\raw
mkdir data\logs
mkdir data\uploads
mkdir data\subset
mkdir data\logs\checkpoints

Start the CPU-safe stack:

docker compose up --build

For local NVIDIA GPU testing, layer the optional override:

docker compose -f compose.yaml -f compose.gpu.yaml up --build

CI and the default local stack use compose.yaml, so GPU access is optional.

Local Services

• Frontend
• Swagger UI / OpenAPI docs
• ReDoc API docs
• API health

If .env overrides PORT, use that API port instead of 8000.

Documentation Index

• Architecture - Compose topology, Mermaid diagram, API/WebSocket surface, demo paths, checkpoint handling, and limitations.
• Final Demo Runbook - Infrastructure smoke verification and final MP4 fallback runbook.
• Performance Benchmark - Repeatable MP4 benchmark path and delivered-system smoke measurements.
• Backend - API contracts, WebSocket contracts, event schema, persistence, and logging details.
• Frontend - React dashboard, live camera mode, MP4 session mode, and operator workflow.
• Integration and DevOps - Docker Compose wiring, logs, environment variables, and smoke test flow.
• Inference - MP4 inference CLI, runtime service API, checkpoint metadata, tracking, and context module.
• Data Pipeline - Dataset preparation and visualization utilities.
• ML Baseline - Baseline training, validation, checkpoint output, and selected classes.
• CI Workflows - Automated validation paths.
• Contributing - Contributor workflow.

Demo Paths

Primary operator path:

• Live camera mode in the Frontend streams browser camera frames to WS /ws/camera.
• The backend returns live detection/status messages and broadcasts generated events through WS /ws/live.
• Live camera behavior depends on a valid local checkpoint, config path, local camera/browser behavior, and target runtime conditions.

Fallback demo path:

• MP4 session mode uploads an operator-selected .mp4 to POST /api/videos/upload.
• The frontend starts processing with POST /api/sessions/, polls GET /api/sessions/{session_id}, and then reads GET /api/events/sessions/{session_id}.
• The final runbook also supports direct MP4-to-JSON fallback inference with scripts/run_mp4_inference.ps1.

Infrastructure smoke path:

.\scripts\final_demo_smoke.ps1

The smoke script verifies API health, session creation, WebSocket event flow, database persistence, and logs using a generated dummy checkpoint and dummy MP4. It is not final model validation.

Checkpoint Status

Place model checkpoints under:

data/logs/checkpoints/<checkpoint>.pth

Inside Docker, the same file is visible as:

/app/data/logs/checkpoints/<checkpoint>.pth

The Compose inference service can read INFERENCE_CHECKPOINT, and the demo/benchmark scripts accept checkpoint paths through their -Checkpoint arguments. The delivered demo and benchmark documentation use data/logs/checkpoints/baseline_epoch_50.pth as the current checkpoint path.

Performance Status

See Performance Benchmark for the repeatable benchmark command and current delivered-system smoke measurements captured with baseline_epoch_50.pth.

Known Limitations

• CPU mode may be slower than GPU mode.
• Smoke benchmarks are throughput checks, not accuracy or model-quality validation.
• Performance depends on hardware, device, checkpoint, config, input video, and cold-start/cache state.
• Live camera mode depends on local camera/browser behavior and target runtime conditions.