All numbers marked TODO: unverified have not been measured from trained models.
They represent implementation capability or design targets. Do not cite them as
empirical results.
The pytest suite and ablation runner are structured to produce the numbers below once trained weights are available.
These tests verify that the generative process is numerically correct and can be run without trained parameters.
| Test | Result | Command | Notes |
|---|---|---|---|
| Prior sampling uniformity | Pass | pytest validator/test_model.py -k "test_prior_sampling_statistics" |
1000 prior samples verified uniform within tolerance |
| Guidance perturbation | Pass | pytest validator/test_model.py -k "test_guidance_perturbs_logits" |
Guidance changes logits measurably (diff > 0.01) |
| Guidance normalization | Pass | pytest validator/test_model.py -k "test_guidance_composer_normalization" |
Unnormalized 100× scale yields 57× ratio; normalized keeps ratio < 20 |
| Categorical sample validity | Pass | pytest validator/test_model.py -k "test_sample_categorical_validity" |
One-hot decoding verified |
| Flow update bias direction | Pass | pytest validator/test_model.py -k "test_flow_update_bias_direction" |
First atom type probability increases after biased update |
| Flow update preserves simplex | Pass | pytest validator/test_model.py -k "test_flow_update_preserves_probability_simplex" |
Valid probability simplex preserved after dt ∈ {0.05, 0.1, 0.2} |
| Noising (t=0 identity) | Pass | pytest validator/test_model.py -k "test_forward_process_variance or test_forward_process_interpolation" |
t=0 unchanged, t=0.5 exact interpolation, t=1 uniform |
| No-NaN sampling (all modes) | Pass | pytest validator/test_model.py -k "test_sampling_loop_no_nan" |
Unconditional, guided, and guided+projected all produce finite output |
| Different guidance strengths | Pass (shape only) | pytest validator/test_model.py -k "test_different_guidance_strengths" |
Verifies sampling paths complete; property improvement requires trained models |
| Test | Result | Command | Notes |
|---|---|---|---|
| Expressiveness proxy (path vs. star) | Pass | pytest validator/test_model.py -k "test_expressiveness_proxy" |
Path graph vs star graph produce different outputs (diff > 1e-6) |
| Permutation invariance (backbone) | Pass | pytest validator/test_model.py -k "test_permutation_invariance_backbone" |
Output permutes identically with input |
| Permutation invariance (predictor) | Pass | pytest validator/test_model.py -k "test_property_predictor_permutation_invariance" |
Graph-level predictions unchanged under node permutation |
| Manifold projection consistency | Pass | pytest validator/test_model.py -k "test_manifold_projection_consistency" |
Deterministic output with same input |
| Manifold projection disabled | Pass | pytest validator/test_model.py -k "test_manifold_projection_disabled" |
Disabled projection = identity transformation |
| Edge-biased attention shape | Pass | pytest validator/test_model.py -k "test_backbone_output_shape" |
Correct output shapes with and without edge features |
| Test | Result | Command | Notes |
|---|---|---|---|
| Noising → denoising consistency | Pass | pytest validator/test_model.py -k "test_noising_then_denoising_consistency" |
Loss < 0.1 at t=0 (clean input → near-perfect reconstruction) |
| Manifold projection ramp schedule | Pass | pytest validator/test_model.py -k "test_manifold_projection_ramp_schedule" |
Low noise (t=0.1) produces stronger perturbation than high noise (t=0.9) |
| Manifold projection valid probabilities | Pass | pytest validator/test_model.py -k "test_manifold_projection_produces_valid_probabilities" |
Valid simplex across all noise levels, no NaN/Inf |
| Validity rate (RDKit) | TODO: unverified |
python -c "from eval import *; m=MetricsCollector(...); m.evaluate_batch(...)" |
Requires trained weights + RDKit |
| Property attainment rates | TODO: unverified |
Same as above | Requires trained weights + RDKit |
| FCD | TODO: unverified |
Requires ChemNet model | Not wired; external dependency |
| Diversity (Tanimoto) | TODO: unverified |
eval.py::MetricsCollector.compute_diversity |
Requires trained weights + RDKit |
All ablations are single-field config changes defined in
validator/run_ablations.py. Results require trained models — none have been
executed. The table below states the design hypotheses.
| Ablation | Config delta | Hypothesis | Expected Δ vs. baseline | Status |
|---|---|---|---|---|
| No manifold projection | projection.enabled: True → False |
Projection improves validity | Validity drops ≥10% at same property attainment | TODO: unverified |
| No valence constraints | projection.enforce_valence: True → False |
Valence is the most impactful constraint | Validity drop; especially at low sampling steps | TODO: unverified |
| No guidance | predictor.enabled: True → False |
Guidance improves property satisfaction | Property attainment drops toward unconditional | TODO: unverified |
| Product of experts | composer.composition: 'weighted_sum' → 'product_of_experts' |
PoE gives better multi-objective control | Pareto front shifts | TODO: unverified |
| Sampling steps = 10 | backbone.sampling_steps: 50 → 10 |
Steerability decreases with fewer steps | Property attainment lower at 10 steps | TODO: unverified |
| Sampling steps = 25 | backbone.sampling_steps: 50 → 25 |
Intermediate tradeoff | Property attainment between 10 and 50 | TODO: unverified |
| Sampling steps = 100 | backbone.sampling_steps: 50 → 100 |
Diminishing returns | Small improvement over 50 | TODO: unverified |
| Separate GNN predictor | predictor.architecture: 'mlp_on_backbone' → 'separate_gnn' |
Backbone features are property-predictive | Comparable property attainment; slower | TODO: unverified |
| Clean-only training | predictor.train_on_clean_only: False → True |
Multi-noise training is necessary | Guidance decays at 10-25 sampling steps | TODO: unverified |
| No gradient normalization | composer.normalize_gradients: True → False |
Normalization prevents dominance | Single property dominates; Pareto front degrades | TODO: unverified |
| Constant ramp | projection.ramp_type: 'linear' → 'constant' |
Ramping is necessary | Property satisfaction degrades at comparable validity | TODO: unverified |
Reproduce all: python validator/run_ablations.py (requires trained models).
TODO: unverified— profiling results below are structural expectations, not measured numbers. Usepython validator/profile_model.pyto obtain real values.
GPU: (tested on NVIDIA GPU with ≥4 GB VRAM when available) Precision: float32
| Phase | Expected time per step | Expected peak mem |
|---|---|---|
| Forward (backbone only) | ~10–30 ms (256-d, 64 nodes) | ~500 MB |
| Forward + backward (training) | ~30–60 ms | ~1.5 GB |
| Guided sampling (50 steps, B=16) | ~1–5 s total | ~2 GB |
Estimated FLOPs (backbone forward, per step):
- Attention: 2 × B × n_heads × N² × d_head = 2 × 128 × 8 × 4096 × 32 = 268M
- FFN: 2 × B × N × d_model × d_ff × 3 = 2 × 128 × 64 × 256 × 1024 × 3 = 12.9B
- Total per step: ~13G FLOPs (forward)
- Training (6× FLOPs): ~78G FLOPs per step
Reproduce: python validator/profile_model.py --mode forward
| Dimension | Score/status | Evidence | Gaps |
|---|---|---|---|
| Novelty | 4/5 | Gap analysis identifies 5 well-grounded gaps; DeFoG explicitly leaves guidance as future work | No concurrent work search since research synthesis; gradient guidance is methodologically analogous to classifier guidance in diffusion |
| Experimental comprehensiveness | 3/5 | 20 smoke tests, 30+ pytest tests, 9 defined ablations, 3 profiling modes | DeFoG baseline not reproduced; CFG baseline missing; OOD pipeline incomplete; FCD dependency not wired |
| Theoretical foundation | 3/5 | CTMC flow theory correctly implemented; guidance gradient composition correct; valence constraints chemically grounded | No formal proof of manifold preservation; gradient guidance formulation lacks trajectory analysis; PoE scaling is ad-hoc |
| Result analysis | 2/5 | Smoke tests verify numerical correctness; ramp schedule empirically verified; composer normalization characterized | No quantitative results from trained models (validity, property satisfaction, FCD, diversity, OOD gap, Pareto frontier) |
| Implementation reproducibility | 4/5 | All configs fully specified with defaults; 20 smoke tests pass; structured for CI; seeded RNG for reproducibility | Smoke tests use small config; full-scale behavior may differ; RDKit and trained weights required for full reproduction |
| Writing readiness | 3/5 | Architecture document comprehensive (1200+ lines); configs have inline docstrings; research synthesis covers 10+ key papers | No experiment results; no publication-ready figures; scaffold-split OOD narrative lacks supporting data |
Required next experiments (from validator scorecard):
- Reproduce DeFoG unconditional results on MOSES (validity ≥ 92.8%)
- Run
no_projectionandno_guidanceablations (tests core claims) - Train property predictor on multi-noise levels, evaluate guidance effectiveness
- Sampling steps sweep {10, 25, 50, 100, 200}
- Single-objective vs. multi-objective Pareto frontier
- Implement CFG baseline and compare
- Implement scaffold-split OOD evaluation
| Baseline | Status | Target metric | Command |
|---|---|---|---|
| Unconditional DeFoG (MOSES) | Implemented, TODO: unverified |
Validity ≥ 92.8%, FCD comparable | Requires full training on MOSES |
| Classifier-free guidance (DiGress-adapted) | TODO: not implemented |
— | Would require conditional backbone training |
| Post-hoc filtering | Partially implemented | — | Evaluation infrastructure exists; filtering pipeline absent |
| OOD scaffold-split | Config-supported, TODO: not implemented |
Validity gap < 10% | Requires dataset preprocessing |