English | 中文 | Changelog | Contributing

The ZDEM Particle Tracker Application

ZDEM Particle Tracker is a desktop research tool for inspecting ZDEM two-dimensional discrete-element frame dumps and tracking one permanent particle ID through a selected simulation interval. It is designed for salt-tectonics analogue / numerical experiments where morphology evolves through layered overburden deformation, mobile salt, basal constraints, wall-bounded model geometry, and particle loss by erosion or removal.

The tool keeps the scope deliberately narrow: streaming all_<step>.dat parsing, wall-first domain detection, true-radius particle rendering, permanent-ID trajectory extraction, displacement and path-length curves, erosion / missing-file classification, and reproducible visual exports for figures or audit trails. It does not claim to be a published package, a general DEM solver, or a multi-tracer linking framework.

MainViewer after loading a real lab sample path used by tests (~tens of thousands of particles, layered groups). Particles are not hand-drawn in the capture script.

Research background

ZDEM salt-tectonic simulations commonly produce long sequences of text frames (all_<step>.dat) containing particle coordinates, radii, groups, wall information, and additional solver sections. For structural interpretation, the central question is often not only what the final cross-section looks like, but how material points travelled while the model shortened, uplifted, folded, or eroded:

Where did permanent particle ID = N move between two simulation steps, how much displacement and cumulative path length did it accumulate, and was its disappearance a physical removal event or a data problem?

This repository supports that workflow with explicit assumptions:

• Permanent ID is the material marker. Tracking uses the particle id; file row order and row index are never treated as identity.
• The experimental domain is wall defined. Walls are preferred for the model box, with metadata fallback and manual lock where needed.
• Kinematics are relative to the chosen start step. Displacement, increments, and path length are measured from the user-selected reference frame.
• Removed material is reported distinctly. A missing particle after valid frames can be classified as eroded / removed, while unreadable frames remain file errors.
• Visual evidence is reproducible. The GUI and evidence scripts provide traceable screenshots, curves, and tables without silently writing into experiment directories.

Getting started

These instructions run the repository from source. They do not install a published package.

Prerequisites

• Python 3.11+
• Windows 10/11 for primary interactive desktop use
• Linux for headless CI-style checks (QT_QPA_PLATFORM=offscreen)
• Optional: uv for fast environment creation

Installation

Windows with uv (recommended)

This creates a source-checkout environment with runtime dependencies and dev/test tools; it does not install a published package.

git clone https://github.com/Phoenix0531-sudo/ZDEM_ParticleTracker.git
cd ZDEM_ParticleTracker
uv sync --dev
uv run python main.py

Windows with Python venv + pip

git clone https://github.com/Phoenix0531-sudo/ZDEM_ParticleTracker.git
cd ZDEM_ParticleTracker
py -3.11 -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
pip install -r requirements.txt
python main.py

Linux headless environment

git clone https://github.com/Phoenix0531-sudo/ZDEM_ParticleTracker.git
cd ZDEM_ParticleTracker
python3.11 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt

Core stack: PySide6, NumPy, SciPy, PyQtGraph, Matplotlib (+ scienceplots), and VisPy. CI forces the PyQtGraph rendering path with ZDEM_FORCE_PYQTGRAPH=1.

Quick start

1. Start the application from the repository root:

uv run python main.py
# or, inside an activated venv:
python main.py

2. Choose an experiment directory containing files named all_<digits>.dat (non-recursive scan).
3. Confirm the start / end steps and the experimental region (walls preferred; metadata fallback).
4. Select a particle by click (KD-tree) or by entering the permanent particle ID.
5. Inspect trajectory, displacement components, cumulative path length, and present / eroded / file-error status as frames advance.

Headless test commands:

# Windows PowerShell
$env:QT_QPA_PLATFORM = "offscreen"
$env:ZDEM_FORCE_PYQTGRAPH = "1"
uv run pytest -q tests

# Linux / CI shell
QT_QPA_PLATFORM=offscreen ZDEM_FORCE_PYQTGRAPH=1 uv run pytest -q tests

Screenshots and evidence

GUI hero captures use a real orogenic-scale salt-tectonics ZDEM experiment (41794 balls, 4 groups: base / salt / roof / ***, step 365000) when available; the samples/ mini sequence (12 particles, 3 steps) remains the clone-runnable fallback.

Benchmark — real parse performance

Measured on the same experiment (33 frames, ~42k balls each, 35 MB per file, Windows + PyQtGraph path, single core):

Operation	Wall time	Throughput
Metadata + basics parse (BASIC_FRAME)	≈ 0.12–0.18 s / frame	~230k particles / s
Full-properties parse (FULL_PARTICLE_PROPERTIES)	≈ 0.20 s / frame	~210k particles / s
KD-tree build (41794 finite balls)	< 0.01 s	—
31-frame trajectory extract (single ID)	≈ 4.5 s total	~7 files / s

Plans to switch to FIND_SINGLE_PARTICLE mode for trajectory extract reduce per-frame bytes touched and should roughly halve the wall time.

MainViewer — real ZDEM experiment capture; 41794 particles, 4 groups	Trajectory — tracked salt particle path (coloured by step)
Displacement vs step — ux / uy / |u| / cumulative path length (km)	Erosion policy — present / eroded / file_error distinctly classified
Particle field — 41794 balls, 4 groups, walls, region	Evidence pack — 2×2 collage of the panels above

uv run python scripts/capture_real_shots.py
uv run python scripts/generate_evidence_pack.py

Application architecture

A typical interactive session follows:

1. Scan the directory for ^all_(\d+)\.dat$ only; ignore .sav, vtk_inters_*.vtk, and subfolders.
2. Parse metadata, wall records, particle basics, and groups in a streaming fashion; stop before large contact blocks when possible.
3. Render particles as batched geometry (VisPy mesh discs by default; PyQtGraph / CPU path for CI).
4. Select one permanent ID present at the session start frame.
5. Track across the chosen step range: displacement relative to start, increments, path length; classify present, eroded, and file_error separately.
6. Export only on user request (CSV / PNG); nothing is auto-written into the experiment folder.

Package layout:

main.py
zdem_particle_tracker/
  app.py
  parsers/          # DAT stream parse + scan
  rendering/        # VisPy / pyqtgraph / backend selection
  services/         # region, trajectory, export, quality, project config
  widgets/          # MainViewer, selection helpers
  workers/          # frame load / trajectory workers
  ui/               # side panels, about, legend
  utils/            # cache, colors, logging helpers
tests/              # parser, gate, region, trajectory, GUI smoke (subprocess)

Concern	Approach
Identity	Permanent id only — never row index for tracking
Region	User lock > walls AABB > metadata (RegionDetector)
Velocity	v = Δx / Δstep (simulation step), not wall-clock seconds
Scale	NumPy arrays + one/few draw calls — no per-particle Qt items
GUI freeze	Workers + signals; LRU frame cache; cancelable trajectory

Performance

• Streaming / section-based DAT parse avoids loading multi-GB contact tails for basic playback.
• Viewport culling and optional decimation reduce draw pressure; picking still uses the full spatial index.
• Mesh buffers are reused while scrubbing frames.
• Linux CI isolates some Qt constructs in a subprocess (tests/qt_subprocess.py) to avoid process-wide OpenGL aborts.

Help and support

1. Read this README and the in-app status messages (scan / load / trajectory / cancel).
2. Reproduce with the headless commands above where possible.
3. Open a GitHub issue with OS, Python version, a minimal DAT snippet or metadata, and whether the failure is parse / select / track / render.

Related lab tools in the same ecosystem:

Repo	Role
ZDEM_Salt_Kinematics	Salt geometry metrics
ZDEM_Area_Conservation	Area conservation / triangulation
ZDEM_Bond_Fracture	Bond damage series
ZDEM_Model_Editor	Model file visual editor

Contributing

Useful contributions:

• Parser robustness on real DAT variants (headers, scientific notation, missing property rows).
• Tests that stay offline unless explicitly gated for private lab samples.
• Documentation and bilingual README alignment.
• GUI paths that remain cancelable and thread-safe.

Please open an issue before large refactors of MainViewer. Keep permanent-ID tracking and region policy intact.

Scope / limitations (v1)

In

• ZDEM 2D all_*.dat sequences, walls, group / color modes
• Single permanent-ID tracking, erosion vs file-error distinction
• Displacement, increments, path length; curves and table
• Manual project config (.zdemtrack.json) and manual export

Out (by design for v1)

• Simultaneous multi-particle tracking
• Binary .sav reverse engineering
• vtk_inters_*.vtk contact network as primary source
• 3D solver UI, cloud sync, auto-export into experiment folders
• Treating PEPT / fluorescence multi-tracer linking as the core product (see pept for that domain)

Continuous integration

GitHub Actions workflow .github/workflows/ci.yml:

1. Logic job — ruff critical rules + pytest excluding heavy GUI/render modules first.
2. Full offscreen job — installs XCB/EGL libs, QT_QPA_PLATFORM=offscreen, ZDEM_FORCE_PYQTGRAPH=1, full tests/.
3. pip-audit — advisory only (continue-on-error).

Badge at the top tracks the CI workflow on main.

License

MIT. See LICENSE.

Acknowledgements

README section layout is intentionally aligned with concise research-oriented particle-tracking projects such as pept: a clear problem statement, installation paths, architecture, performance notes, support, and honest scope. Domain algorithms and ZDEM formats here are independent of PEPT.