[Question] Why does LLaVA evaluation assert batch_size == 1 for benchmark inference?

[AustinKol]

Question

Hi LLaVA team,

We are evaluating LLaVA-1.5-7B on MME and noticed batch-size-dependent variance in a small number of benchmark predictions.

In the official LLaVA evaluation code, several benchmark loaders appear to assert or assume batch_size == 1 during evaluation. We wanted to ask whether this was only for implementation simplicity, or whether it was intentional because batched generation can produce different predictions.

Our setup:

• Model: llava-hf/llava-1.5-7b-hf
• Decoding: greedy, do_sample=False, num_beams=1
• Precision: fp16
• Padding: left padding
• Benchmark: MME
• Compared batch size 1 vs batch size 16

We found that most predictions match exactly, but two MME rows flipped between batch size 1 and batch size 16:

• commonsense_reasoning/0042/0
  • Question: “Are there usually cars in the area shown in the picture? Please answer yes or no.”
  • Gold: yes
  • Batch size 1: Yes
  • Batch size 16: No
• numerical_calculation/0014/0
  • Question: “Is the answer to the arithmetic question in the image 200? Please answer yes or no.”
  • Gold: yes
  • Batch size 1: Yes
  • Batch size 16: No

This changes MME-C and total MME score slightly:

• Batch size 1: MME-C around 350.36, total around 1859.87
• Batch size 16: MME-C around 340.71, total around 1850.23

We ran a focused diagnostic on these rows. After removing left padding, the target row’s input_ids, position_ids, and pixel_values are identical between the single-row input and the real batch-16 input. However, the first-token logits are extremely close in fp16, so small batch-shape/kernel differences can change the Yes/No argmax.

Our questions:

1. Why does the official LLaVA evaluation path assert or assume batch_size == 1 for benchmark inference?
2. Did you observe batch-size-dependent output variance during evaluation?
3. For reproducing reported benchmark numbers, should evaluations always use batch size 1?
4. If batching is intended to be valid, are there specific settings you recommend for stable batched evaluation, such as fp32, eager attention, explicit position IDs, or another generation setup?

Tagging teammates for context: @ATATC @charlesgchen @Victor-W-X @RobyIm @perctrix

Thanks.