Fine-tune PAT-Conv-L to reach 0.625 AUC for depression (3.2% gap)

[The-Obstacle-Is-The-Way]

Summary

We've successfully implemented PAT-Conv-L for depression classification achieving 0.5929 AUC, but we're 3.2% below the paper's reported 0.625 AUC (n=2800). This issue tracks efforts to close this gap through systematic hyperparameter tuning and training improvements.

Current Status

Our Results

• Model: PAT-Conv-L (PAT-L with Conv1D patch embedding)
• Best AUC: 0.5929 (epoch 2)
• Dataset: NHANES 2013-2014 (n=3,077)
• Target: 0.625 AUC as reported in paper's Supplemental Table 5

Paper's Results (Supplemental Table 5)

Model	n=500	n=1000	n=2500	n=2800	Avg AUC
PAT Conv-L (FT)	0.602	0.599	0.617	0.625	0.610
PAT Conv-L (LP)	0.602	0.599	0.617	0.625	0.610

Technical Details

What We've Implemented Correctly

1. ✅ Data preprocessing: Log(x+1) transformation → StandardScaler normalization
2. ✅ Architecture: Conv1D patch embedding (kernel_size=9, stride=9)
3. ✅ Class weighting: pos_weight=9.91 for imbalanced dataset (~9% positive)
4. ✅ Basic training: AdamW optimizer, CosineAnnealingLR scheduler

Training Configuration

batch_size = 32
learning_rate = 1e-4
optimizer = AdamW(lr=1e-4, betas=(0.9, 0.95), weight_decay=0.01)
scheduler = CosineAnnealingLR(T_max=15_epochs, eta_min=1e-5)
criterion = BCEWithLogitsLoss(pos_weight=9.91)

Key Discovery

We fixed a critical normalization bug that was causing AUC 0.4756 (worse than random). Using proper StandardScaler immediately improved results to 0.57+.

Hypotheses for the 3.2% Gap

1. Data Augmentation (Most Likely)

The paper likely uses augmentation techniques we haven't implemented:

• Temporal jittering/warping
• Gaussian noise injection
• Time-shift augmentation
• Mixup/CutMix for sequences
• SMOTE for class imbalance

2. Training Strategy Refinements

• Multiple random seeds → ensemble average
• Different train/val splits
• K-fold cross-validation
• Learning rate warmup
• Gradient accumulation for larger effective batch size

3. Sample Size Discrepancy

• Paper reports n=2,800 for depression
• We found n=3,077 with standard filters
• Investigate additional exclusion criteria

4. Optimization Details

• Try paper's exact LP→FT strategy (though our tests showed worse results)
• Different optimizers (Lion, AdaFactor)
• Label smoothing for better generalization
• Stochastic Weight Averaging (SWA)
• Focal loss instead of weighted BCE

Action Items

Phase 1: Data Augmentation (Priority)

1. Implement time-series specific augmentations
2. Test each augmentation individually
3. Combine best performing augmentations

Phase 2: Training Refinements

1. Run with 5 different random seeds
2. Implement k-fold cross-validation
3. Test gradient accumulation (batch_size 64/128)
4. Add learning rate warmup

Phase 3: Advanced Techniques

1. Ensemble multiple model checkpoints
2. Try different loss functions
3. Implement progressive training strategies

Success Criteria

• Achieve ≥0.620 AUC on validation set (within 0.5% of paper)
• Results reproducible across multiple seeds
• Document exact configuration that achieves target

Resources

• Training script: scripts/pat_training/train_pat_conv_l_simple.py
• Current best checkpoint: model_weights/production/pat_conv_l_v0.5929.pth
• Training logs: training/logs/pat_conv_l_v0.5929_20250725.log

References

• Original PAT paper: Foundation Models for Wearable Movement Data in Mental Health Research
• Supplemental Table 5 shows PAT-Conv-L achieving 0.625 AUC at n=2800

---

Note: Even at 0.5929 AUC, our implementation provides valuable clinical insights. This issue focuses on reaching paper parity for research validation.

[The-Obstacle-Is-The-Way]

Update: Data Leakage Discovered

We've identified why we're stuck at 0.593 AUC and can't reach the target 0.625. See Issue #85 for full details.

Root Cause: We're using PAT models pretrained on 29k participants (which includes NHANES 2013-2014) and then fine-tuning on the same NHANES 2013-2014 dataset. This creates severe data leakage where the model has already "seen" the test data during pretraining.

PAT authors confirmed they used 21k weights (excludes NHANES 2013-2014) for their results.

Next Steps:

1. Fix data leakage by switching to 21k weights (Issue Critical: PAT Fine-Tuning Uses Wrong Pretrained Weights (Data Leakage) #85)
2. Retrain depression head with correct setup
3. Expect initial drop in performance (healthy sign)
4. Target realistic 0.60-0.61 AUC based on authors' actual results

This issue is blocked until we resolve #85.