Learning to Rank LLM Expertise with Behavioral Profiles

TREC 2025 Million LLM Track — submission notebook Run ID: lambdamart_profiles

This repository contains the code and notebook paper for a behavioral profiling approach to ranking LLMs by predicted expertise on a given query. Each of the 1,131 LLMs in the track is represented by a compact 64-dimensional behavioral profile derived from its response behavior, and a LambdaMART model is trained to rank these profiles against incoming queries.

Abstract

Behavioral profiles are derived by embedding responses from the discovery dataset with the Qwen3-Embedding-8B model, producing 64-dimensional vectors that encode response statistics, linguistic patterns, refusal behavior, and domain- and task-specific expertise. For each query, interaction features are built by matching domain and task signals extracted from the query against the corresponding behavioral scores. These feature vectors serve as input to a LambdaMART ranking model trained to directly optimize NDCG.

Task

The TREC 2025 Million LLM Track introduces a ranking task in which the goal is to identify the most relevant LLMs for a given query based solely on historical query–response pairs — without issuing new queries to the models. The track provides two datasets:

• Discovery dataset — queries with precomputed responses and token-level log probabilities from all 1,131 models.
• Development dataset — 342 queries with relevance judgments (qrels) labeling the expertise of each LLM per query.

Pipeline

flowchart TD
    A1[Discovery Dataset<br/>1,131 LLMs<br/>responses + logprobs]
    A2[Development Dataset<br/>342 queries<br/>with qrels]

    subgraph Profile["Behavioral Profile Construction"]
        direction TB
        B1[Sample 50% of<br/>discovery responses]
        B2[Qwen3-Embedding-8B<br/>encode responses]
        B3["Extract feature groups:<br/>• Response metrics<br/>• Lexical diversity<br/>• Structural features<br/>• Query–response interaction<br/>• Semantic embeddings<br/>• Domain scores (10)<br/>• Task scores (7)"]
        B4[(64-dim<br/>Behavioral Profile)]
        B1 --> B2 --> B3 --> B4
    end

    subgraph QueryFeat["Query Feature Extraction"]
        direction TB
        C1["• Query descriptors<br/>• Domain keyword match<br/>• Task keyword match"]
    end

    subgraph Interaction["Interaction Features"]
        direction TB
        D1[Match query vs.<br/>profile scores]
        D2[Elementwise products]
        D3[Aggregate<br/>mean, std, max]
        D4[Concatenate with<br/>behavioral profile]
        D1 --> D2 --> D3 --> D4
    end

    subgraph Training["Learning to Rank"]
        direction TB
        E1[Top-200 LLMs<br/>per query]
        E2[LightGBM<br/>LambdaMART<br/>rank_xendcg]
        E3[Train: 273 queries<br/>Validate: 75 queries]
        E1 --> E2 --> E3
    end

    subgraph Deliverable["Output"]
        direction TB
        F1[["Scoring function<br/>f query, LLM → score"]]
        F2[/"lambdamart_profiles<br/>TREC 2025 run"/]
        F1 --> F2
    end

    A1 --> Profile
    A2 --> QueryFeat
    A2 --> E1
    Profile --> D1
    QueryFeat --> D1
    D4 --> E2
    E3 --> F1

    classDef data fill:#e1f5ff,stroke:#0277bd,color:#01579b
    classDef process fill:#fff4e1,stroke:#ef6c00,color:#e65100
    classDef model fill:#f3e5f5,stroke:#7b1fa2,color:#4a148c
    classDef output fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20

    class A1,A2 data
    class B1,B2,B3,C1,D1,D2,D3,D4,E1 process
    class B4,E2,E3 model
    class F1,F2 output

Loading

Methodology

Behavioral profiles

Each LLM is represented by a behavioral profile: a vector summarizing how the model tends to answer, how confident it appears, and how its responses are distributed in embedding space. Profiles are derived from 50% of the discovery responses (using more did not improve profile quality). The following signals feed into the profile:

Feature group	Description
Response metrics	Token and character counts, log probabilities, punctuation ratios
Lexical diversity	Type–token ratio, word repetition rates
Structural features	Sentence counts, code/list usage, formatting, uppercase/digit frequency
Query–response interaction	Jaccard similarity, keyword overlap
Semantic embeddings	Centroid embeddings, dispersion measures, similarity statistics from Qwen3-Embedding-8B

Each LLM's signals are distilled into a 64-dimensional profile encoding domain expertise (10 domains), task strengths (7 task types), quality indicators, semantic descriptors, and statistical summaries.

• Domain labels: technology, science, medicine, law, business, education, arts, sports, politics, history
• Task labels: explanation, comparison, analysis, instruction, creative, technical, factual

Interaction features

For each query–LLM pair, interaction features are built by combining query descriptors (length, type–token ratio) with the LLM's behavioral profile. The same ten domain labels and seven task types used in the profiles are inferred from the query via keyword matching, and those matches are appended alongside the profile's domain/task scores together with their elementwise products. Domain and task interactions are then aggregated with simple statistics (mean, standard deviation, maximum) so the ranker gets a summary of how well each model fits the query. These interaction features, together with the behavioral profile, form the input feature vector for the learning-to-rank stage.

Learning to rank

The ranking model is LightGBM's LambdaMART, used with the rank_xendcg objective to directly optimize NDCG. Training uses only the top 200 LLMs per query (chosen by development-set relevance) so the model focuses on high-quality comparisons, while all 1,131 models are ranked at inference for the final submission. This keeps the model learning from strong candidates and avoids noisy labels and easy negatives.

Results

The model is trained on 273 queries with labels from the development dataset and validated on 75 held-out queries, using behavioral profiles for all 1,131 LLMs. On a five-fold cross-validation over the development set:

NDCG@10 = 0.7674 ± 0.0250

An initial feature-importance analysis with LightGBM's built-in function suggests that only a subset of the 64 features contributes meaningfully to the ranking. An ablation study with removing feature groups and measuring the impact on the evaluation metrics is planned.

Limitations

The ten domain and seven task categories are handcrafted. Queries outside the predefined labels are forced into the closest match, which may bias how the models are scored. Future work will address this alongside a deeper analysis of how well behavioral profiles work across domains and task types.