CORE-Bench Evaluation

Evaluation toolkit for CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding

Overview | Data | Evaluation | Outputs | Citation

CORE-Bench studies code retrieval as requirement-driven repository search. Given a bug report, feature request, refactoring request, or other development query, a retriever must identify the code and repository context that a coding agent should inspect. This setting is broader than conventional docstring-to-function or isolated snippet retrieval because it uses concrete repository states, multiple relevant locations, long code chunks, and dense in-repository distractors.

Resources

Resource	Link	Description
Paper	arXiv:2606.11864	Benchmark design, construction, and evaluation results
Dataset	zhangfw123/CORE-Bench	Level-2, Level-3, and rewritten-query benchmark data
Evaluation code	zhangfw123/CORE-Bench-Eval	Dense retrieval, BM25, prediction export, and result collection
Upstream framework	embeddings-benchmark/mteb	Original MTEB task and evaluation framework

Benchmark Overview

CORE-Bench evaluates retrieval at three complementary levels:

Level	Goal	Scale reported in the paper
Level-1: Code Understanding	Foundational text-to-code, code-to-text, code-to-code, hybrid, and competitive-programming retrieval	172,961 queries; 2,412,296 corpus items
Level-2: Issue-to-Edit Localization	Retrieve files or chunks that must be modified for a requested change	632 repositories; 5,061 queries; 9,377,120 chunks
Level-3: Broader Context Retrieval	Retrieve auxiliary code, tests, documentation, configuration, and other context needed to complete the change	97 repositories; 2,580 queries; 2,609,581 chunks

Registered Benchmarks

Benchmark name	Included tasks
CORE_BENCH_LEVEL1	11 tasks from original benchmarks
CORE_BENCH_LEVEL2	SWE-bench Pro, Verified, Live, SWE-Bench++, SWE-bench+, Multi-SWE-bench, and SWE-bench Multilingual
CORE_BENCH_LEVEL3	SWE-bench Pro, Verified, Multi-SWE-bench, and SWE-bench Multilingual
CORE_BENCH_LEVEL2_REWRITE	Rewritten queries for all registered Level-2 tasks
CORE_BENCH_LEVEL3_REWRITE	Rewritten queries for all registered Level-3 tasks

Repository Layout

CORE-Bench-Eval/
  instruction/              # Model-family query instructions
  src/models/               # Dense model backends and MTEB wrappers
  src/tasks/                # CORE-Bench task loaders
  collect_results/          # Prediction-to-Excel aggregation
  scripts/                  # Dense and BM25 benchmark runners
  run_mteb.py               # Dense retrieval entry point
  run_bm25.py               # BM25 entry point

Environment

The following package versions have been validated with this evaluation pipeline:

torch==2.10.0
transformers==4.57.6
sentence-transformers==5.1.2
mteb==1.38.9
datasets==3.2.0
bm25s==0.2.14
pandas==2.3.3
openpyxl==3.1.5

Data Setup

Download the benchmark data from Hugging Face:

git lfs install
git clone https://huggingface.co/datasets/zhangfw123/CORE-Bench

Then point CORE_BENCH_DATA_ROOT at the directory containing the packaged Level-2, Level-3, and rewrite data:

export CORE_BENCH_DATA_ROOT=/path/to/CORE-Bench/data

The expected layout is:

$CORE_BENCH_DATA_ROOT/
  LEVEL-2/<dataset>/<repository>/
  LEVEL-3/<dataset>/<repository>/
  Rewrite-LEVEL-2/<dataset>/<repository>/
  Rewrite-LEVEL-3/<dataset>/<repository>/

Each repository directory contains corpus.jsonl, queries.jsonl, and qrels.jsonl. Rewrite directories provide replacement queries and reuse the original corpus and qrels.

The custom CLARC and CPRet Level-1 data can be stored separately:

export CORE_BENCH_LEVEL1_ROOT=/path/to/Code-Retrieval-Bench/LEVEL-1

The other nine Level-1 tasks use their revision-pinned MTEB/Hugging Face dataset loaders and the standard Hugging Face cache.

Dense Retrieval

Clone this evaluation repository and enter its root directory:

git clone https://github.com/zhangfw123/CORE-Bench-Eval.git
cd CORE-Bench-Eval

Run Qwen3-Embedding-0.6B on Level-2:

python run_mteb.py \
  --model Qwen/Qwen3-Embedding-0.6B \
  --model_type qwen3 \
  --precision fp16 \
  --batch_size 16 \
  --benchmark CORE_BENCH_LEVEL2 \
  --model_kwargs '{"max_length":8192,"pooler_type":"last","do_norm":true,"add_eos_id":true,"use_instruction":true,"instruction_template":"Instruct: {task_description}\nQuery: {query}","instruction_dict_path":"instruction/task_prompts_multilingual.json","trust_remote_code":true}' \
  --run_kwargs '{"save_predictions":true,"top_k":50000}' \
  --output_dir results/Qwen3-Embedding-0.6B

The Qwen3 helper runs one or more benchmark groups with the same settings:

bash scripts/run_core_bench.sh \
  Qwen/Qwen3-Embedding-0.6B \
  Qwen3-Embedding-0.6B \
  CORE_BENCH_LEVEL2 CORE_BENCH_LEVEL3

When no benchmark names are supplied, the helper attempts all five registered groups. The corresponding data roots must exist.

Outputs

Dense evaluation writes two forms of output:

results/<model>/
  <task>_<subset>[_<split>]_predictions.json
  <model-name>/<revision>/<task>.json
  <model-name>/<revision>/model_meta.json

The flat prediction JSON files are consumed by the result collector. The nested files are standard MTEB task results containing metrics and model metadata.

Collect complete benchmark results from multiple model directories:

python collect_results/collect_core_bench_results.py \
  --model_dirs results/Qwen3-Embedding-0.6B results/bm25 \
  --benchmarks CORE_BENCH_LEVEL2 CORE_BENCH_LEVEL3 \
  --output_file core_bench_results.xlsx

The collector intentionally requires every task and split in each requested benchmark.

Citation

@misc{zhang2026corebench,
  title         = {CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Coding},
  author        = {Fuwei Zhang and Yanzhao Zhang and Mingxin Li and Dingkun Long and Lexiang Hu and Pengjun Xie and Zhao Zhang and Fuzhen Zhuang},
  year          = {2026},
  eprint        = {2606.11864},
  archivePrefix = {arXiv},
  primaryClass  = {cs.IR},
  url           = {https://arxiv.org/abs/2606.11864}
}

Acknowledgements

This evaluation toolkit is built on the original MTEB (Massive Text Embedding Benchmark) framework. We thank the MTEB contributors for providing the task abstractions, evaluation interfaces, and retrieval infrastructure.