Skip to content

Add extension point to support per-track parameter validation #2143

Description

@reugn

Rally tracks can register custom param sources, runners, and schedulers via register(registry) in track.py. There is currently no equivalent hook for validating track parameters before a benchmark run starts.

When a track uses array parameters that must satisfy a structural constraint - for example, all scheduling arrays must have either 1 element or exactly N elements where N is the number of configured phases - there is no clean way to enforce this.

The proposal is to add a register_validator(challenge_name, fn) function to esrally/track/params.py, following the same pattern as the existing register_param_source_for_name. The validator function receives the resolved params dict and raises TrackConfigError with a readable message if the params are invalid. Rally calls all registered validators for the active challenge after the track spec is loaded and params are merged, but before the schedule is built. TrackConfigError already exists in esrally/exceptions.py for this purpose, so no new exception type is needed.

# In track.py
def _validate_autoscale_params(params):
    phases = params.get("as_phases", 5)
    for name, default in [("as_warmup_time_periods", [600]), ("as_time_periods", [1800])]:
        arr = params.get(name, default)
        if len(arr) != 1 and len(arr) != phases:
            raise TrackConfigError(f"{name} must have 1 or {phases} elements, got {len(arr)}")

def register(registry):
    for challenge in ("search-autoscale", "ingest-autoscale", "ingest-search-autoscale"):
        registry.register_validator(challenge, _validate_autoscale_params)

This produces a clean error like TrackConfigError: as_search_clients must have 1 or 4 elements, got 2 before any cluster operations run.

Without a validation hook, track authors either skip validation (silent wrong results) or implement workarounds that give users misleading error messages. A first-class register_validator hook is a small addition that makes the plugin API complete and consistent with the rest of the extension points Rally already provides.

Metadata

Metadata

Assignees

Labels

enhancementImproves the status quo

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions