Benchmarking #3864
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| FROM python:3.12 | ||
|
|
||
| RUN pip install --no-cache-dir matplotlib netcdf4 numpy pandas pint pooch pyproj scipy traitlets xarray asv pysu metpy | ||
|
Check warning on line 3 in benchmarks/Dockerfile
|
||
|
blue-jaye-121 marked this conversation as resolved.
Outdated
|
||
|
|
||
| COPY --chmod=700 entrypoint.sh / | ||
|
|
||
| ENTRYPOINT ["/entrypoint.sh"] | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,86 @@ | ||
| pipeline { | ||
| agent { label 'main' } | ||
| environment { | ||
| CLONE_DIR = "temp_repo_results" | ||
| } | ||
| stages { | ||
| // checks out the results repo using secret stored in Jenkins | ||
| stage('Checkout results repo') { | ||
| steps { | ||
| sh 'git config --global credential.helper cache' | ||
| sh 'git config --global push.default simple' | ||
| checkout scmGit(branches: [[name: '*/main']], extensions: [[$class: 'RelativeTargetDirectory', relativeTargetDir: 'temp_repo_results'], cloneOption(depth: 1, noTags: false, reference: '', shallow: true)], userRemoteConfigs: [[credentialsId: 'GH_DEPLOY_KEY_METPY_BENCH_RESULTS', url: 'https://github.com/unidata/metpy-benchmark.git']]) | ||
| } | ||
| } | ||
| // copies past results into the asv/results folder on the main repo | ||
| stage('Copy past results') { | ||
| steps { | ||
| sh ''' | ||
| if [ -d ${CLONE_DIR}/results ]; then | ||
| echo "-------Copying results--------" | ||
| cp -r ${CLONE_DIR}/results/* benchmarks/asv/results | ||
| fi | ||
| ''' | ||
| } | ||
| } | ||
| // generates the hashes to run and stores them in a text file | ||
| stage('Setup for ASV run') { | ||
| steps { | ||
| sh ''' | ||
| cd benchmarks | ||
| bash generate_hashes.sh | ||
| cd .. | ||
| ''' | ||
| } | ||
| } | ||
| // Runs ASV in the docker container | ||
| // The catch error ensures that the build works even if some ASV fail | ||
| stage('Run ASV') { | ||
| steps { | ||
| catchError(buildResult: 'SUCCESS') { | ||
| sh ''' | ||
| cd benchmarks | ||
| docker build -t metpy-benchmarks:latest . | ||
| cd .. | ||
| docker run --rm -v .:/container-benchmarks --hostname Docker_Container -e DUID=$(id -u) -e DGID=$(id -g) metpy-benchmarks:latest benchmark | ||
| ''' | ||
| } | ||
| } | ||
| } | ||
| // Copies results from the asv/results into the results repo | ||
| stage('Copy results') { | ||
| steps{ | ||
| sh ''' | ||
| if [ -d "${CLONE_DIR}/results" ]; then | ||
| echo "--------results repo exist-------" | ||
| else | ||
| mkdir ${CLONE_DIR}/results | ||
| fi | ||
| cp -r benchmarks/asv/results/* ${CLONE_DIR}/results | ||
| ''' | ||
| } | ||
| } | ||
| // Pushes to the git repo if there have been changes | ||
| stage('Update results repo') { | ||
| steps { | ||
| withCredentials([gitUsernamePassword(credentialsId: 'ASV_RESULTS_REPO_PAT', gitToolName: 'Default')]) { | ||
| sh ''' | ||
| if [ -n "$(git status --porcelain)" ]; then | ||
| cd ${CLONE_DIR} | ||
| git add --all | ||
| git commit -m "Jenkins Updating Benchmark Results BUILD-NUMBER:${BUILD_NUMBER}" || echo "-----no changes to commit-----" | ||
| git push origin HEAD:main --force | ||
| fi | ||
| ''' | ||
| } | ||
| } | ||
| } | ||
| } | ||
| post { | ||
| // always removes the temporary repo regardless of build status | ||
| always { | ||
| echo "---Cleaning up temporary repo---" | ||
| sh 'rm -rf "${CLONE_DIR}"' | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,208 @@ | ||
| { | ||
| // The version of the config file format. Do not change, unless | ||
| // you know what you are doing. | ||
| "version": 1, | ||
|
|
||
| // The name of the project being benchmarked | ||
| "project": "metpy", | ||
|
|
||
| // The project's homepage | ||
| "project_url": "https://unidata.github.io/MetPy/latest/", | ||
|
|
||
| // The URL or local path of the source code repository for the | ||
| // project being benchmarked | ||
| "repo": "..", | ||
|
|
||
| // The Python project's subdirectory in your repo. If missing or | ||
| // the empty string, the project is assumed to be located at the root | ||
| // of the repository. | ||
| //"repo_subdir": "benchmarks", | ||
|
|
||
| // Customizable commands for building the project. | ||
| // See asv.conf.json documentation. | ||
| // To build the package using pyproject.toml (PEP518), uncomment the following lines | ||
| "build_command": [ | ||
| "python -m pip install build", | ||
| "python -m build", | ||
| "python -mpip wheel -w {build_cache_dir} {build_dir}" | ||
| ], | ||
| // To build the package using setuptools and a setup.py file, uncomment the following lines | ||
| // "build_command": [ | ||
| // "python setup.py build", | ||
| // "python -mpip wheel -w {build_cache_dir} {build_dir}" | ||
| // ], | ||
|
|
||
| // Customizable commands for installing and uninstalling the project. | ||
| // See asv.conf.json documentation. | ||
| "install_command": ["in-dir={env_dir} python -mpip install {build_dir}"], | ||
| "uninstall_command": ["return-code=any python -mpip uninstall -y {project}"], | ||
|
|
||
| // List of branches to benchmark. If not provided, defaults to "main" | ||
| // (for git) or "default" (for mercurial). | ||
| "branches": ["HEAD"], // for git | ||
| // "branches": ["default"], // for mercurial | ||
|
|
||
| // The DVCS being used. If not set, it will be automatically | ||
| // determined from "repo" by looking at the protocol in the URL | ||
| // (if remote), or by looking for special directories, such as | ||
| // ".git" (if local). | ||
| // "dvcs": "git", | ||
|
|
||
| // The tool to use to create environments. May be "conda", | ||
| // "virtualenv", "mamba" (above 3.8) | ||
| // or other value depending on the plugins in use. | ||
| // If missing or the empty string, the tool will be automatically | ||
| // determined by looking for tools on the PATH environment | ||
| // variable. | ||
| "environment_type": "virtualenv", | ||
|
|
||
| // timeout in seconds for installing any dependencies in environment | ||
| // defaults to 10 min | ||
| //"install_timeout": 600, | ||
|
|
||
| // the base URL to show a commit for the project. | ||
| "show_commit_url": "http://github.com/unidata/metpy/commit/", | ||
|
|
||
| // The Pythons you'd like to test against. If not provided, defaults | ||
| // to the current version of Python used to run `asv`. | ||
| //"pythons": ["3.8", "3.12"], | ||
|
|
||
| // The list of conda channel names to be searched for benchmark | ||
| // dependency packages in the specified order | ||
| "conda_channels": ["conda-forge"], | ||
|
|
||
| // A conda environment file that is used for environment creation. | ||
| // "conda_environment_file": "environment.yml", | ||
|
|
||
| // The matrix of dependencies to test. Each key of the "req" | ||
| // requirements dictionary is the name of a package (in PyPI) and | ||
| // the values are version numbers. An empty list or empty string | ||
| // indicates to just test against the default (latest) | ||
| // version. null indicates that the package is to not be | ||
| // installed. If the package to be tested is only available from | ||
| // PyPi, and the 'environment_type' is conda, then you can preface | ||
| // the package name by 'pip+', and the package will be installed | ||
| // via pip (with all the conda available packages installed first, | ||
| // followed by the pip installed packages). | ||
| // | ||
| // The ``@env`` and ``@env_nobuild`` keys contain the matrix of | ||
| // environment variables to pass to build and benchmark commands. | ||
| // An environment will be created for every combination of the | ||
| // cartesian product of the "@env" variables in this matrix. | ||
| // Variables in "@env_nobuild" will be passed to every environment | ||
| // during the benchmark phase, but will not trigger creation of | ||
| // new environments. A value of ``null`` means that the variable | ||
| // will not be set for the current combination. | ||
| // | ||
| "matrix": { | ||
| "req": { | ||
| "matplotlib": [ | ||
| ], | ||
| "numpy": [ | ||
| ], | ||
| "pandas": [ | ||
| ], | ||
| "pint": [ | ||
| ], | ||
| "pooch": [ | ||
| ], | ||
| "pyproj": [ | ||
| ], | ||
| "scipy": [ | ||
| ], | ||
| "traitlets": [ | ||
| ], | ||
| "xarray": [ | ||
| ], | ||
| "netcdf4": [ | ||
| ], | ||
| } | ||
| }, | ||
| // Combinations of libraries/python versions can be excluded/included | ||
| // from the set to test. Each entry is a dictionary containing additional | ||
| // key-value pairs to include/exclude. | ||
| // | ||
| // An exclude entry excludes entries where all values match. The | ||
| // values are regexps that should match the whole string. | ||
| // | ||
| // An include entry adds an environment. Only the packages listed | ||
| // are installed. The 'python' key is required. The exclude rules | ||
| // do not apply to includes. | ||
| // | ||
| // In addition to package names, the following keys are available: | ||
| // | ||
| // - python | ||
| // Python version, as in the *pythons* variable above. | ||
| // - environment_type | ||
| // Environment type, as above. | ||
| // - sys_platform | ||
| // Platform, as in sys.platform. Possible values for the common | ||
| // cases: 'linux2', 'win32', 'cygwin', 'darwin'. | ||
| // - req | ||
| // Required packages | ||
| // - env | ||
| // Environment variables | ||
| // - env_nobuild | ||
| // Non-build environment variables | ||
| // | ||
| // "exclude": [ | ||
| // {"python": "3.2", "sys_platform": "win32"}, // skip py3.2 on windows | ||
| // {"environment_type": "conda", "req": {"six": null}}, // don't run without six on conda | ||
| // {"env": {"ENV_VAR_1": "val2"}}, // skip val2 for ENV_VAR_1 | ||
| // ], | ||
| // | ||
| // "include": [ | ||
| // // additional env for python3.12 | ||
| // {"python": "3.12", "req": {"numpy": "1.26"}, "env_nobuild": {"FOO": "123"}}, | ||
| // // additional env if run on windows+conda | ||
| // {"platform": "win32", "environment_type": "conda", "python": "3.12", "req": {"libpython": ""}}, | ||
| // ], | ||
|
|
||
| // The directory (relative to the current directory) that benchmarks are | ||
| // stored in. If not provided, defaults to "benchmarks" | ||
| // "benchmark_dir": "benchmarks", | ||
|
|
||
| // The directory (relative to the current directory) to cache the Python | ||
| // environments in. If not provided, defaults to "env" | ||
| "env_dir": "asv/env", | ||
|
|
||
| // The directory (relative to the current directory) that raw benchmark | ||
| // results are stored in. If not provided, defaults to "results". | ||
| "results_dir": "asv/results", | ||
|
|
||
| // The directory (relative to the current directory) that the html tree | ||
| // should be written to. If not provided, defaults to "html". | ||
| "html_dir": "asv/html", | ||
|
|
||
| // The number of characters to retain in the commit hashes. | ||
| // "hash_length": 8, | ||
|
|
||
| // `asv` will cache results of the recent builds in each | ||
| // environment, making them faster to install next time. This is | ||
| // the number of builds to keep, per environment. | ||
| "build_cache_size": 52 | ||
|
|
||
| // The commits after which the regression search in `asv publish` | ||
| // should start looking for regressions. Dictionary whose keys are | ||
| // regexps matching to benchmark names, and values corresponding to | ||
| // the commit (exclusive) after which to start looking for | ||
| // regressions. The default is to start from the first commit | ||
| // with results. If the commit is `null`, regression detection is | ||
| // skipped for the matching benchmark. | ||
| // | ||
| // "regressions_first_commits": { | ||
| // "some_benchmark": "352cdf", // Consider regressions only after this commit | ||
| // "another_benchmark": null, // Skip regression detection altogether | ||
| // }, | ||
|
|
||
| // The thresholds for relative change in results, after which `asv | ||
| // publish` starts reporting regressions. Dictionary of the same | ||
| // form as in ``regressions_first_commits``, with values | ||
| // indicating the thresholds. If multiple entries match, the | ||
| // maximum is taken. If no entry matches, the default is 5%. | ||
| // | ||
| // "regressions_thresholds": { | ||
| // "some_benchmark": 0.01, // Threshold of 1% | ||
| // "another_benchmark": 0.5, // Threshold of 50% | ||
| // }, | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.