Interpolate ViT positional embeddings for variable resolutions

[Thakor-Yashpal]

Modifies load_pretrained and interpolate_posembed functions to support loading pre-trained Vision Transformer (ViT) checkpoints into models initialized with different input resolutions.

Changes include:

• Detecting shape mismatches in positional embeddings (Transformer/posembed_input/pos_embedding) between the checkpoint and the target model.
• Calling interpolate_posembed to resize the embedding grid using scipy.ndimage.zoom (bilinear interpolation).
• Adding logic to handle the presence or absence of the class token during separation and concatenation.
• Including error checks for non-square grid sizes and shape mismatches after interpolation.

This allows fine-tuning or inference with ViT models on image resolutions different from their pre-training resolution.

[sylvesterkaczmarek]

The class-token heuristic misclassifies a common no-token 14x14 embedding. For n=196, the first check says no class token, but the fallback tests 195 % int(sqrt(195)) == 0; int(sqrt(195)) is 13 and 195 % 13 == 0, so it flips to True even though 195 is not square. interpolate_posembed then rejects the 195-token grid. Could this use an actual perfect-square check (s*s == n / s*s == n-1) and cover 196 tokens?

[junleen]

您好，这是来自QQ邮箱的自动回复邮件。我将尽快阅读您的邮件，谢谢！