Skip to content

Commit 236f0e1

Browse files
authored
[query] Benchmark configurations (#14762)
Added benchmark parameters for all benchmark tests in the query codebase in `batch-config.yaml`. This file lists, for all benchmarks: - Number of burn in iterations (untimed trials) - The mean runtime and coefficient of variance when the analyses were performed - Configurations for benchmarking at differing fidelity: - The minimal detectable `slowdown` factor - Number of instances (batch jobs) to use - Number of iterations (how many timed trials of the benchmark per instance) These parameters allow for more precise and controlled benchmark measurements by explicitly defining the statistical requirements and execution parameters for each test. These parameters are now used by the benchmark_in_batch.py script. Currently only high fidelity (expensive) configurations are supported. Collecting these configurations was a long boring process so I included others while I was working on it. To run the benchmarks, simply run from the hail root directory: ```bash make benchmark ``` When run last, the benchmark suite took 14 hours to complete and cost 50 USD. Much of this runtime comes from failing to time out when at the memory limit and premptions. This change has no impact on the broad-managed batch service in gcp.
1 parent c7a3f17 commit 236f0e1

2 files changed

Lines changed: 1260 additions & 6 deletions

File tree

0 commit comments

Comments
 (0)