Skip to content

Harden walkthrough CI tests: snapshots + fault-tolerant harness #79

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
torwager merged 2 commits into from
Jun 26, 2026
Merged

Harden walkthrough CI tests: snapshots + fault-tolerant harness #79

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
5f93d4b
Harden walkthrough CI tests: snapshots + fault-tolerant harness
torwager Jun 19, 2026
0567b10
Walkthrough CI: handle missing Image Processing Toolbox
torwager Jun 19, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .github/workflows/tests-walkthroughs.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ jobs:
products: |
Statistics_and_Machine_Learning_Toolbox
Signal_Processing_Toolbox
Image_Processing_Toolbox
cache: true

- name: Run walkthrough tests
Expand Down
168 changes: 168 additions & 0 deletions CanlabCore/Unit_tests/helpers/canlab_classify_environment_error.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
function category = canlab_classify_environment_error(ME, had_env_skip)
%CANLAB_CLASSIFY_ENVIRONMENT_ERROR Bucket a caught error for CI test handling.
%
% category = canlab_classify_environment_error(ME)
% category = canlab_classify_environment_error(ME, had_env_skip)
%
% Classifies an MException caught while running toolbox example/walkthrough
% code into one of a small set of buckets, so that test harnesses can decide
% whether to SKIP (mark Incomplete) or FAIL. The goal is to keep headless CI
% from going red over missing-display / missing-data / interactive-prompt
% conditions that are not real regressions, while still surfacing genuine
% code errors.
%
% This centralizes the heuristics previously hand-coded inside
% canlab_test_help_examples.m (skip_on_environment_error) so the walkthrough
% harness and the per-method help-example tests share one definition.
%
% :Inputs:
%
% **ME:**
% an MException (or any struct with .identifier, .message, and
% optionally .stack fields).
%
% **had_env_skip:**
% logical (default false). When true, "undefined variable/function"
% and "unrecognized field" errors are treated as CASCADE fallout from
% an earlier skipped section rather than genuine failures. Pass the
% running "have we already skipped something upstream" flag here.
%
% :Output:
%
% **category:**
% one of:
% 'graphics' - needs a display / OpenGL / Java / figure window, or
% the error originates inside a graphics-only code path
% (orthviews, surface rendering, etc.).
% 'input' - code tried to prompt for interactive user input,
% unavailable in batch/CI.
% 'capability' - an UndefinedFunction error for a known optional
% MATLAB toolbox function (e.g. niftiinfo) that is not
% provisioned on this runner.
% 'data' - an optional data file (signature, atlas, feature set)
% is not on the CI path.
% 'cascade' - an undefined-variable/field error following an earlier
% environment skip (downstream fallout, not a new bug).
% 'genuine' - none of the above; treat as a real failure.
%
% :See also: canlab_run_walkthrough_snapshot, canlab_test_help_examples

% ..
% Author: CANlab. Part of the CanlabCore Unit_tests headless-CI harness.
% ..

if nargin < 2 || isempty(had_env_skip)
had_env_skip = false;
end

msg = lower(ME.message);
id = '';
if isprop(ME, 'identifier') || isfield(ME, 'identifier')
id = ME.identifier;
end

% ---------------------------------------------------------------------
% Graphics: explicit graphics error ids, telltale message tokens, or a
% stack frame inside a known graphics-only function. spm_orthviews and the
% surface renderers fail or behave erratically on a headless runner.
% ---------------------------------------------------------------------
gfx_ids = {'MATLAB:graphics:opengl:Unavailable', ...
'MATLAB:graphics:initialize', ...
'MATLAB:class:InvalidHandle'};
is_gfx = any(strcmp(id, gfx_ids)) || ...
strncmp(id, 'MATLAB:hg:', 10) || ... % handle-graphics errors
contains(msg, 'opengl') || contains(msg, 'display') || ...
contains(msg, 'java') || contains(msg, 'jvm') || ...
contains(msg, 'figure window') || contains(msg, 'no display') || ...
contains(msg, 'graphics') || ...
contains(msg, 'invalid or deleted object'); % set/get on a handle
% whose graphics section
% was skipped upstream

gfx_fns = {'spm_orthviews', 'orthviews', 'spm_check_registration', ...
'spm_figure', 'surface', 'render_on_surface', 'addbrain', ...
'cluster_surf', 'isosurface', 'riverplot', 'tor_3d', ...
'render_blobs', 'cluster_orthviews'};
if ~is_gfx && isfield_or_prop(ME, 'stack') && ~isempty(ME.stack)
stack_names = {ME.stack.name};
if any(ismember(stack_names, gfx_fns))
is_gfx = true;
end
end
if is_gfx
category = 'graphics';
return
end

% ---------------------------------------------------------------------
% Interactive input: input() / keyboard prompts are unavailable in -batch.
% MATLAB reports MissingRequiredCapability with a "support for user input"
% message in that case.
% ---------------------------------------------------------------------
is_input = strcmp(id, 'MATLAB:services:MissingRequiredCapability') || ...
contains(msg, 'support for user input') || ...
contains(msg, 'input is not available');
if is_input
category = 'input';
return
end

% ---------------------------------------------------------------------
% Missing optional MATLAB toolbox. An UndefinedFunction error for a known
% MathWorks toolbox function (e.g. niftiinfo/niftiread from the Image
% Processing Toolbox) means that toolbox is not provisioned on this runner,
% not that the code is broken. Keep this list to functions that are *only*
% ever provided by a toolbox, so we never mask a genuine missing-CanlabCore
% function bug.
% ---------------------------------------------------------------------
optional_toolbox_fns = {'niftiinfo', 'niftiread', 'niftiwrite', ...
'cfg_getfile'};
if strcmp(id, 'MATLAB:UndefinedFunction')
undef = regexp(ME.message, "Undefined function '([^']+)'", 'tokens', 'once');
if ~isempty(undef) && any(strcmpi(undef{1}, optional_toolbox_fns))
category = 'capability';
return
end
end

% ---------------------------------------------------------------------
% Missing optional data files (NPS+ signatures, Neurosynth feature set,
% Bianciardi atlas sources, etc.). load_image_set / annotate_* abort with a
% disp() + bare error('Exiting'), so the message is literally "Exiting".
% ---------------------------------------------------------------------
bare_exiting = isempty(id) && ...
ismember(strtrim(strrep(ME.message, '.', '')), {'Exiting', 'exiting'});
is_data = contains(msg, 'cannot find images') || ...
contains(msg, 'find and add the file') || ...
contains(msg, 'not found in matlab path') || ...
contains(msg, 'no such file or directory') || ...
contains(msg, 'cannot find') || ...
bare_exiting;
if is_data
category = 'data';
return
end

% ---------------------------------------------------------------------
% Cascade: an earlier section was skipped, so a variable/field it would
% have defined is now missing. Treat as fallout, not a new regression.
% ---------------------------------------------------------------------
if had_env_skip && ( ...
strcmp(id, 'MATLAB:UndefinedFunction') || ...
strcmp(id, 'MATLAB:undefinedVarOrClass') || ...
strcmp(id, 'MATLAB:nonExistentField') || ...
contains(msg, 'undefined') || ...
contains(msg, 'unrecognized field'))
category = 'cascade';
return
end

category = 'genuine';

end


function tf = isfield_or_prop(obj, name)
% MException is an object (use isprop); a plain struct uses isfield.
tf = (isobject(obj) && isprop(obj, name)) || (isstruct(obj) && isfield(obj, name));
end
207 changes: 207 additions & 0 deletions CanlabCore/Unit_tests/helpers/canlab_run_walkthrough_snapshot.m
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,207 @@
function canlab_run_walkthrough_snapshot(tc, wt__snapshot_path)
%CANLAB_RUN_WALKTHROUGH_SNAPSHOT Run a frozen walkthrough snapshot under CI hardening.
%
% canlab_run_walkthrough_snapshot(tc, snapshot_path)
%
% Executes a copied CANlab_help_examples walkthrough script (a "snapshot"
% bundled under Unit_tests/walkthroughs/private/) one %%-cell at a time, in
% a single shared workspace, with headless-graphics and fault-tolerant error
% handling. Reports outcomes to the supplied matlab.unittest TestCase `tc`.
%
% Why snapshots instead of running the real tutorials. The walkthroughs in
% the CANlab_help_examples repo are teaching scripts, full of graphics
% (orthviews, surface), optional data sets, and the occasional interactive
% prompt. They should stay clean tutorials, not be retrofitted with test
% scaffolding, and CanlabCore CI should not depend on an external repo's
% un-hardened scripts. So we keep verbatim copies here and apply the
% hardening at run time, in one reusable place (this function) rather than
% editing each copy. Refresh a snapshot by overwriting the file under
% private/ from example_help_files/ — no re-hardening needed.
%
% Hardening applied:
% * Headless graphics: DefaultFigureVisible is forced 'off' for the run
% (callers also set this; doing it here makes the harness safe to call
% directly). Figures are closed between sections.
% * Section isolation: the script is split at its %% cell boundaries and
% each cell is run in its own try/catch, so a graphics-only section that
% fails on a headless runner does not abort the compute sections.
% * Error classification (see canlab_classify_environment_error):
% - graphics / input / data / cascade errors -> section SKIPPED,
% recorded, execution continues.
% - genuine errors -> the harness stops and the test FAILS with an
% informative message (which section, error id, message, and the
% offending top stack frame), because downstream cells usually
% cascade once a real error occurs.
% * Outcome mapping to matlab.unittest:
% - any genuine failure -> tc.verifyFail (test FAILS)
% - every section skipped (env) -> tc.assumeFail (test INCOMPLETE)
% - at least one section ran ok -> PASS, with a logged skip summary.
%
% Variable hygiene: every internal variable in this function is prefixed
% `wt__` because snapshot code is eval'd in this workspace and would
% otherwise clobber ordinary loop variables (i, n, t, r, ...).
%
% :Inputs:
% **tc:** a matlab.unittest.TestCase (from functiontests).
% **snapshot_path:** absolute path to the snapshot .m file to run.
%
% :See also: canlab_classify_environment_error, canlab_run_all_tests

% ..
% Author: CANlab. Part of the CanlabCore Unit_tests headless-CI harness.
% ..

tc.assertTrue(exist(wt__snapshot_path, 'file') == 2, ...
sprintf('Walkthrough snapshot not found: %s', wt__snapshot_path));

[~, wt__name] = fileparts(wt__snapshot_path);
wt__src = fileread(wt__snapshot_path);
wt__sections = local_split_sections(wt__src);
wt__nsec = numel(wt__sections);

% Headless: callers set this too, but be self-sufficient.
wt__prev_vis = get(0, 'DefaultFigureVisible');
set(0, 'DefaultFigureVisible', 'off');
wt__cleanup = onCleanup(@() set(0, 'DefaultFigureVisible', wt__prev_vis));

wt__skipped = {}; % cellstr of human-readable skip notes
wt__ran_real = 0; % count of sections that executed without error
wt__had_env_skip = false; % drives cascade classification downstream
wt__failure = []; % first genuine failure, if any (struct)

for wt__s = 1:wt__nsec

wt__code = wt__sections{wt__s};

% Skip cells that are pure comment/whitespace, or that define a
% function (snapshots are scripts; a trailing function would not eval).
if local_is_noop(wt__code)
continue
end

try
evalc(wt__code); % runs in THIS workspace; vars persist
wt__ran_real = wt__ran_real + 1;
catch wt__ME
wt__cat = canlab_classify_environment_error(wt__ME, wt__had_env_skip);
if strcmp(wt__cat, 'genuine')
wt__failure = struct( ...
'section', wt__s, ...
'id', wt__ME.identifier, ...
'message', wt__ME.message, ...
'where', local_top_frame(wt__ME), ...
'snippet', local_first_stmt(wt__code));
break % stop; later cells will cascade
else
wt__had_env_skip = true;
wt__skipped{end+1, 1} = sprintf(' section %d/%d [%s]: %s', ...
wt__s, wt__nsec, wt__cat, local_oneline(wt__ME.message)); %#ok<AGROW>
end
end

close all force
end

% ---------------------------------------------------------------------
% Map collected outcomes onto the TestCase.
% ---------------------------------------------------------------------
if ~isempty(wt__failure)
wt__report = sprintf([ ...
'%s: genuine error at section %d of %d.\n' ...
' identifier: %s\n' ...
' message: %s\n' ...
' section starts: %s\n' ...
' at: %s\n' ...
' (%d earlier section(s) skipped for environment reasons)'], ...
wt__name, wt__failure.section, wt__nsec, ...
wt__failure.id, local_oneline(wt__failure.message), ...
wt__failure.snippet, wt__failure.where, numel(wt__skipped));
tc.verifyFail(wt__report);

elseif wt__ran_real == 0
tc.assumeFail(sprintf( ...
'%s: all %d section(s) skipped for environment reasons:\n%s', ...
wt__name, wt__nsec, strjoin(wt__skipped, newline)));

else
if ~isempty(wt__skipped)
tc.log(1, sprintf('%s: %d of %d section(s) skipped (environment):\n%s', ...
wt__name, numel(wt__skipped), wt__nsec, strjoin(wt__skipped, newline)));
end
tc.verifyTrue(true, sprintf('%s ran %d section(s) without genuine error', ...
wt__name, wt__ran_real));
end

end


% =====================================================================
% Local helpers
% =====================================================================

function sections = local_split_sections(src)
% Split source at %% cell markers. Each section keeps its leading marker
% line (a comment, harmless to eval). Everything before the first marker is
% section 1.
lines = regexp(src, '\r\n|\r|\n', 'split');
is_head = ~cellfun('isempty', regexp(lines, '^\s*%%', 'once'));
starts = find(is_head);
if isempty(starts) || starts(1) ~= 1
starts = [1, starts];
end
starts = unique(starts);
ends = [starts(2:end) - 1, numel(lines)];
sections = cell(numel(starts), 1);
for k = 1:numel(starts)
sections{k} = strjoin(lines(starts(k):ends(k)), newline);
end
end


function tf = local_is_noop(code)
% True if the cell has no executable content: blank, comment-only, or a
% function definition (which cannot be eval'd as a statement).
stripped = regexprep(code, '%[^\n]*', ''); % drop line comments
stripped = strtrim(stripped);
tf = isempty(stripped) || ~isempty(regexp(stripped, '^function\b', 'once'));
end


function s = local_oneline(msg)
% Collapse a multi-line error message to a single trimmed line.
s = strtrim(regexprep(msg, '\s+', ' '));
end


function s = local_top_frame(ME)
% "funcname (line N)" for the deepest stack frame that is not this harness
% (errors raised at a section's top level otherwise just point back at the
% evalc call site here, which is unhelpful). Falls back to a marker.
if isempty(ME.stack)
s = '<top level of section>';
return
end
this_fn = mfilename;
for k = 1:numel(ME.stack)
if ~strcmp(ME.stack(k).name, this_fn)
s = sprintf('%s (line %d)', ME.stack(k).name, ME.stack(k).line);
return
end
end
s = '<top level of section>';
end


function s = local_first_stmt(code)
% First non-comment, non-blank line of a section, trimmed, for the report.
lines = regexp(code, '\r\n|\r|\n', 'split');
s = '<none>';
for k = 1:numel(lines)
ln = strtrim(lines{k});
if ~isempty(ln) && ~strncmp(ln, '%', 1)
s = ln;
if numel(s) > 100, s = [s(1:97) '...']; end
return
end
end
end
Loading
Loading