Issue encountered

When developers create custom evaluation tasks using LightevalTaskConfig, there is currently no fast way to perform static structural validation before executing the pipeline. If a developer forgets the TASKS_TABLE module export, requests an evaluation_split that isn't declared in hf_avail_splits, or fails to return a Doc object from their prompt_function, the error is only caught at runtime (usually triggering a messy traceback when lighteval tasks inspect attempts to iterate over eval_docs()). This delays developer feedback loops, especially when pushing heavy evaluation jobs to headless compute nodes.

Solution/Feature

I propose introducing a lightweight static validation utility, exposed via a new lighteval tasks lint <custom_task.py> CLI command.

This feature would statically validate both the LightevalTaskConfig structure and the module export boundary without downloading datasets or executing the pipeline.

Specifically, the pure-Python linter would verify:

1. Module Export Integrity: Asserts that the script correctly exposes a list named TASKS_TABLE.
2. Split Verification: Cross-references evaluation_splits and few_shots_split strictly against the declared hf_avail_splits.
3. Metric Conformity: Validates that config.metrics contains populated grading instances.
4. Prompt Function Signature: Uses inspect.signature to statically ensure the prompt_function accepts the required row dictionary and correctly returns the expected Doc class.

I have already built the pure-Python validation logic locally, along with a pytest suite that uses types.ModuleType to mock the namespace boundaries and verify the exports. If this aligns with the core roadmap, I would be more than happy to open a Pull Request!

Possible alternatives

1. Relying strictly on lighteval tasks inspect: Leaving the current behavior as-is. The downside is that inspect actually instantiates the datasets and loops through data, which is computationally heavier than a fast static check for spelling/typing errors.
2. Pydantic Validation: Refactoring LightevalTaskConfig to inherit from pydantic.BaseModel to catch typing errors on instantiation. However, this introduces a heavier external dependency footprint and still wouldn't solve the TASKS_TABLE module-export boundary issue.