On Group size

[John-Ge]

I am interested in what would be if G=16 or more. Is the group size scalable?

[MKJia]

Thanks for your interest! You can refer to Figure 8 for an ablation study on varying group sizes $$G$$. As discussed in our paper, increasing the number of sub-codebook groups ($$G$$) generally improves reconstruction quality; however, a higher $$G$$ reduces the dimension of each sub-token, which may negatively impact performance. Thus, scaling to $$G=16$$ or beyond is feasible, but it typically requires increasing the latent dimension $$C$$ accordingly to maintain or enhance performance.

[John-Ge]

I am very interested in the performances how could we scale up both G and dim also with large enough vocabulary size. Apart from rFID, gFID is also very cirtical metric. Would you like to explore the upper bound of your proposed method.

Thanks.

[MKJia]

Hi, thanks for the thoughtful suggestion! Exploring gFID performances with larger values of G, dimension, and vocabulary size would indeed be interesting directions. At the moment, we don’t have enough time and resources to fully investigate these scaling aspects. For now, we’ve only tested our tokenizers with a simple Llamagen generator, but we agree that pushing toward the upper bound is a promising avenue for future work.