Problem
Task specs are currently composable colon-delimited variants (task[:variant1[:variant2...]]), but the semantics of each segment are not explicit or enforced.
Concrete example from current quickstart docs:
humaneval:3shot conveys prompt format / few-shot regime, but not metric.humaneval:bpb conveys metric, but not format/regime.
Proposal
Adopt an explicit minimal schema for task specs and apply it consistently. For example:
<dataset>:<format>:<metric>
Examples:
humaneval:3shot:bpbhumaneval:0shot:pass_at_1
There's an open question, of course, for what mc|rc count as vs 0shot|3shot|8shot count as
Problem
Task specs are currently composable colon-delimited variants (
task[:variant1[:variant2...]]), but the semantics of each segment are not explicit or enforced.Concrete example from current quickstart docs:
humaneval:3shotconveys prompt format / few-shot regime, but not metric.humaneval:bpbconveys metric, but not format/regime.Proposal
Adopt an explicit minimal schema for task specs and apply it consistently. For example:
<dataset>:<format>:<metric>Examples:
humaneval:3shot:bpbhumaneval:0shot:pass_at_1There's an open question, of course, for what
mc|rccount as vs0shot|3shot|8shotcount as