Skip to content

Speed up CI testing #1021

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mattcieslak wants to merge 3 commits into
base: main
Choose a base branch
from
Open

Speed up CI testing #1021

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
3fe0511
speed up testing?
mattcieslak Jan 16, 2026
d1f4518
update parameters
mattcieslak Jan 16, 2026
97cb773
cleanup
mattcieslak Jan 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
142 changes: 89 additions & 53 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@ version: 2.1
orbs:
codecov: codecov/codecov@3.2.4

parameters:
run_full_drbuddi:
type: boolean
default: false

.dockersetup: &dockersetup
docker:
- image: pennlinc/qsiprep_build:26.1.7
Expand All @@ -14,8 +19,13 @@ runinstall: &runinstall
if [[ -n "$CIRCLE_TAG" ]]; then
VERSION="$CIRCLE_TAG"
fi
git checkout $CIRCLE_BRANCH
pip install .[tests] --progress-bar off
WHEEL=$(ls dist/qsiprep-*.whl 2>/dev/null | head -n 1)
if [[ -n "$WHEEL" ]]; then
pip install "${WHEEL}[tests]" --progress-bar off
else
git checkout $CIRCLE_BRANCH
pip install .[tests] --progress-bar off
fi

# Precaching fonts, set 'Agg' as default backend for matplotlib
python -c "from matplotlib import font_manager"
Expand All @@ -34,66 +44,60 @@ runinstall: &runinstall
jobs:
build:
resource_class: small
docker:
- image: cimg/python:3.10.9
working_directory: /tmp/src/qsiprep
steps:
- checkout
- run: *runinstall

download_drbuddi_rpe_series:
docker:
- image: cimg/python:3.10.9
working_directory: /tmp/src/qsiprep
steps:
- checkout
- restore_cache:
key: drbuddi_rpe_series-01
- run: *runinstall
key: pip-{{ checksum "pyproject.toml" }}
- run:
name: Download drbuddi_rpe_series test data
name: Build wheel
command: |
cd /tmp/src/qsiprep/.circleci
python get_data.py $PWD/data drbuddi_rpe_series
python -m pip install --upgrade pip build --progress-bar off
python -m build --wheel
- save_cache:
key: drbuddi_rpe_series-01
key: pip-{{ checksum "pyproject.toml" }}
paths:
- /tmp/src/qsiprep/.circleci/data/drbuddi_rpe_series
- /root/.cache/pip
- persist_to_workspace:
root: /tmp/src/qsiprep
paths:
- dist

download_drbuddi_epi:
download_drbuddi_data:
resource_class: small
docker:
- image: cimg/python:3.10.9
- image: python:3.10-slim
working_directory: /tmp/src/qsiprep
steps:
- checkout
- restore_cache:
key: drbuddi_epi-01
- run: *runinstall
key: drbuddi-data-01
- run:
name: Download drbuddi_epi test data
name: Download DRBUDDI test data
command: |
cd /tmp/src/qsiprep/.circleci
python get_data.py $PWD/data drbuddi_epi
cd .circleci
python get_data.py $PWD/data drbuddi_rpe_series drbuddi_epi
- save_cache:
key: drbuddi_epi-01
key: drbuddi-data-01
paths:
- /tmp/src/qsiprep/.circleci/data/drbuddi_rpe_series
- /tmp/src/qsiprep/.circleci/data/drbuddi_epi

download_DSDTI:
resource_class: small
docker:
- image: cimg/python:3.10.9
- image: python:3.10-slim
working_directory: /tmp/src/qsiprep
steps:
- checkout
- restore_cache:
key: DSDTI-01
- run: *runinstall
- run:
name: Download DSDTI test data
command: |
cd /tmp/src/qsiprep/.circleci
cd .circleci
python get_data.py $PWD/data DSDTI
- save_cache:
key: DSDTI-01
Expand All @@ -103,17 +107,16 @@ jobs:
download_DSCSDSI:
resource_class: small
docker:
- image: cimg/python:3.10.9
- image: python:3.10-slim
working_directory: /tmp/src/qsiprep
steps:
- checkout
- restore_cache:
key: DSCSDSI-01
- run: *runinstall
- run:
name: Download DSCSDSI test data
command: |
cd /tmp/src/qsiprep/.circleci
cd .circleci
python get_data.py $PWD/data DSCSDSI
- save_cache:
key: DSCSDSI-01
Expand All @@ -123,17 +126,16 @@ jobs:
download_twoses:
resource_class: small
docker:
- image: cimg/python:3.10.9
- image: python:3.10-slim
working_directory: /tmp/src/qsiprep
steps:
- checkout
- restore_cache:
key: twoses-02
- run: *runinstall
- run:
name: Download twoses test data
command: |
cd /tmp/src/qsiprep/.circleci
cd .circleci
python get_data.py $PWD/data twoses
- save_cache:
key: twoses-02
Expand All @@ -143,17 +145,16 @@ jobs:
download_forrest_gump:
resource_class: small
docker:
- image: cimg/python:3.10.9
- image: python:3.10-slim
working_directory: /tmp/src/qsiprep
steps:
- checkout
- restore_cache:
key: forrestgump-01
- run: *runinstall
- run:
name: Download forrest_gump test data
command: |
cd /tmp/src/qsiprep/.circleci
cd .circleci
python get_data.py $PWD/data forrest_gump
- save_cache:
key: forrestgump-01
Expand All @@ -167,6 +168,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: DSCSDSI-01
- run: *runinstall
Expand Down Expand Up @@ -194,6 +197,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: DSDTI-01
- run: *runinstall
Expand Down Expand Up @@ -221,6 +226,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: DSDTI-01
- run: *runinstall
Expand Down Expand Up @@ -248,9 +255,20 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: drbuddi-data-01
- restore_cache:
key: DSDTI-01
- run: *runinstall
- run:
name: Skip unless run_full_drbuddi
command: |
if [ "<< pipeline.parameters.run_full_drbuddi >>" != "true" ]; then
echo "Skipping optional DRBUDDI TENSORLine job"
circleci-agent step halt
fi
- run:
name: Run TENSORLine with epi fmaps and DRBUDDI
no_output_timeout: 2h
Expand All @@ -275,9 +293,18 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: drbuddi_epi-01
key: drbuddi-data-01
- run: *runinstall
- run:
name: Skip unless run_full_drbuddi
command: |
if [ "<< pipeline.parameters.run_full_drbuddi >>" != "true" ]; then
echo "Skipping optional DRBUDDI SHORELine job"
circleci-agent step halt
fi
- run:
name: Run SHORELine with epi fmaps and DRBUDDI
no_output_timeout: 2h
Expand All @@ -302,8 +329,10 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: drbuddi_rpe_series-01
key: drbuddi-data-01
- run: *runinstall
- run:
name: Run Eddy with rpe series fmaps and DRBUDDI
Expand All @@ -329,6 +358,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- run: *runinstall
- run:
name: Run QSIPrep on multi-shell dataset with GRE field maps
Expand All @@ -354,6 +385,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: forrestgump-01
- run: *runinstall
Expand Down Expand Up @@ -381,6 +414,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: forrestgump-01
- run: *runinstall
Expand Down Expand Up @@ -408,6 +443,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: forrestgump-01
- run: *runinstall
Expand Down Expand Up @@ -442,6 +479,8 @@ jobs:
<<: *dockersetup
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: forrestgump-01
- run: *runinstall
Expand All @@ -464,6 +503,8 @@ jobs:
CIRCLE_CPUS: 4
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: twoses-02
- run: *runinstall
Expand All @@ -481,6 +522,8 @@ jobs:
CIRCLE_CPUS: 4
steps:
- checkout
- attach_workspace:
at: /tmp/src/qsiprep
- restore_cache:
key: DSDTI-01
- run: *runinstall
Expand Down Expand Up @@ -598,14 +641,7 @@ workflows:
tags:
only: /.*/

- download_drbuddi_rpe_series:
requires:
- build
filters:
tags:
only: /.*/

- download_drbuddi_epi:
- download_drbuddi_data:
requires:
- build
filters:
Expand Down Expand Up @@ -649,21 +685,25 @@ workflows:

- DRBUDDI_SHORELine_EPI:
requires:
- download_drbuddi_epi
- download_drbuddi_data
- build
filters:
tags:
only: /.*/

- DRBUDDI_eddy_rpe_series:
requires:
- download_drbuddi_rpe_series
- download_drbuddi_data
- build
filters:
tags:
only: /.*/

- DRBUDDI_TENSORLine_EPI:
requires:
- download_drbuddi_data
- download_DSDTI
- build
filters:
tags:
only: /.*/
Expand Down Expand Up @@ -736,9 +776,7 @@ workflows:
- DSCSDSI
- DSDTI_nofmap
- DSDTI_synfmap
- DRBUDDI_SHORELine_EPI
- DRBUDDI_eddy_rpe_series
- DRBUDDI_TENSORLine_EPI
- maternal_brain_project
- forrest_gump
- forrest_gump_patch2self
Expand All @@ -759,9 +797,7 @@ workflows:
- DSCSDSI
- DSDTI_nofmap
- DSDTI_synfmap
- DRBUDDI_SHORELine_EPI
- DRBUDDI_eddy_rpe_series
- DRBUDDI_TENSORLine_EPI
- maternal_brain_project
- forrest_gump
- forrest_gump_patch2self
Expand Down
Loading