Commit 236f0e1
authored
[query] Benchmark configurations (#14762)
Added benchmark parameters for all benchmark tests in the query codebase
in `batch-config.yaml`. This file lists, for all benchmarks:
- Number of burn in iterations (untimed trials)
- The mean runtime and coefficient of variance when the analyses were
performed
- Configurations for benchmarking at differing fidelity:
- The minimal detectable `slowdown` factor
- Number of instances (batch jobs) to use
- Number of iterations (how many timed trials of the benchmark per
instance)
These parameters allow for more precise and controlled benchmark
measurements by explicitly defining the statistical requirements and
execution parameters for each test.
These parameters are now used by the benchmark_in_batch.py script.
Currently only high fidelity (expensive) configurations are supported.
Collecting these configurations was a long boring process so I included
others while I was working on it.
To run the benchmarks, simply run from the hail root directory:
```bash
make benchmark
```
When run last, the benchmark suite took 14 hours to complete and cost 50
USD.
Much of this runtime comes from failing to time out when at the memory
limit and premptions.
This change has no impact on the broad-managed batch service in gcp.1 parent c7a3f17 commit 236f0e1
2 files changed
Lines changed: 1260 additions & 6 deletions
0 commit comments