Rally tracks can register custom param sources, runners, and schedulers via register(registry) in track.py. There is currently no equivalent hook for validating track parameters before a benchmark run starts.
When a track uses array parameters that must satisfy a structural constraint - for example, all scheduling arrays must have either 1 element or exactly N elements where N is the number of configured phases - there is no clean way to enforce this.
The proposal is to add a register_validator(challenge_name, fn) function to esrally/track/params.py, following the same pattern as the existing register_param_source_for_name. The validator function receives the resolved params dict and raises TrackConfigError with a readable message if the params are invalid. Rally calls all registered validators for the active challenge after the track spec is loaded and params are merged, but before the schedule is built. TrackConfigError already exists in esrally/exceptions.py for this purpose, so no new exception type is needed.
# In track.py
def _validate_autoscale_params(params):
phases = params.get("as_phases", 5)
for name, default in [("as_warmup_time_periods", [600]), ("as_time_periods", [1800])]:
arr = params.get(name, default)
if len(arr) != 1 and len(arr) != phases:
raise TrackConfigError(f"{name} must have 1 or {phases} elements, got {len(arr)}")
def register(registry):
for challenge in ("search-autoscale", "ingest-autoscale", "ingest-search-autoscale"):
registry.register_validator(challenge, _validate_autoscale_params)
This produces a clean error like TrackConfigError: as_search_clients must have 1 or 4 elements, got 2 before any cluster operations run.
Without a validation hook, track authors either skip validation (silent wrong results) or implement workarounds that give users misleading error messages. A first-class register_validator hook is a small addition that makes the plugin API complete and consistent with the rest of the extension points Rally already provides.
Rally tracks can register custom param sources, runners, and schedulers via
register(registry)intrack.py. There is currently no equivalent hook for validating track parameters before a benchmark run starts.When a track uses array parameters that must satisfy a structural constraint - for example, all scheduling arrays must have either 1 element or exactly N elements where N is the number of configured phases - there is no clean way to enforce this.
The proposal is to add a
register_validator(challenge_name, fn)function toesrally/track/params.py, following the same pattern as the existingregister_param_source_for_name. The validator function receives the resolved params dict and raisesTrackConfigErrorwith a readable message if the params are invalid. Rally calls all registered validators for the active challenge after the track spec is loaded and params are merged, but before the schedule is built.TrackConfigErroralready exists inesrally/exceptions.pyfor this purpose, so no new exception type is needed.This produces a clean error like
TrackConfigError: as_search_clients must have 1 or 4 elements, got 2before any cluster operations run.Without a validation hook, track authors either skip validation (silent wrong results) or implement workarounds that give users misleading error messages. A first-class
register_validatorhook is a small addition that makes the plugin API complete and consistent with the rest of the extension points Rally already provides.