🌟 CVPR Challenge: Zero-Shot Image Classification

🔍 Overview

This repository contains the solution for the CVPR 2025 Zero-Shot Image Classification Challenge. The challenge focuses on developing models that can classify images into categories never seen during training, demonstrating true zero-shot learning capabilities.

graph TD
    A[Input Image] --> B[CLIP Image Encoder]
    C[Class Names/Descriptions] --> D[CLIP Text Encoder]
    B --> E[Image Embeddings]
    D --> F[Text Embeddings]
    E --> G[Zero-Shot Matching]
    F --> G
    G --> H[Class Prediction]
    
    style A fill:#f9d5e5,stroke:#333,stroke-width:2px
    style H fill:#eeeeee,stroke:#333,stroke-width:2px

Loading

Zero-Shot Classification Pipeline

❗ Problem Statement

Zero-shot learning addresses the fundamental challenge of recognizing objects or categories that were not present in the training data. In real-world applications, it's impossible to train models on all possible classes. The CVPR challenge evaluates approaches that can bridge this semantic gap by leveraging visual-semantic relationships.

Challenge Link : https://vizwiz.org/tasks-and-datasets/image-classification/

graph TD
    A[Zero-Shot Learning] --> B[Image Classification]
    A --> C[Text Classification]
    C --> D[NLP Models]
    B --> E[Vision Models]

Loading

Traditional vs. Zero-Shot Learning

💻 Technologies Used

• Python: Primary programming language
• PyTorch: Deep learning framework
• Transformers: Hugging Face library for state-of-the-art models
• scikit-learn: For clustering and data analysis
• CLIP (Contrastive Language-Image Pre-training): OpenAI's model connecting text and images
• NumPy/Pandas: For data handling and manipulation
• Matplotlib/Seaborn: For visualization

graph TD
    A[Python] --> B[PyTorch]
    A --> C[Transformers]
    C --> D[CLIP]
    B --> E[scikit-learn]

Loading

Technology Stack Overview

📊 Model Architecture

The solution primarily utilizes CLIP (Contrastive Language-Image Pre-training), a neural network trained on a variety of image-text pairs. Key components include:

1. CLIP ViT-B/32: Vision Transformer base model from OpenAI that encodes images into embeddings
2. Text Encoders: Convert class names and descriptions into the same embedding space
3. Embedding Alignment: Methods to align visual and textual embeddings

graph TD
    subgraph Image_Encoder[Image Encoder]
        A1[Input Image]
        A2[Vision Transformer / CNN]
        A3[Image Embedding]
        A1 --> A2
        A2 --> A3
    end

    subgraph Text_Encoder[Text Encoder]
        B1[Input Text]
        B2[Transformer-based Language Model]
        B3[Text Embedding]
        B1 --> B2
        B2 --> B3
    end

    A3 --> C[Shared Embedding Space]
    B3 --> C

    C --> D[Contrastive Loss]
    D --> E[Similarity Score / Prediction]

Loading

CLIP Model Architecture

🔍 Methodological Approach

The implementation follows several key steps:

1. Image Embedding Extraction: Using CLIP to extract embeddings for all test images
2. Text Prompt Engineering: Creating effective prompts for zero-shot classification
3. Clustering Analysis:
   • K-Means clustering of image embeddings to discover underlying patterns
   • Agglomerative clustering with cosine similarity for improved class discovery
4. Ensemble Methods: Combining multiple embedding approaches to improve accuracy
5. Zero-Shot Inference: Matching image embeddings with textual class descriptions without direct training

graph TD
    A[Data Collection] --> B[Data Preprocessing]
    B --> C[Feature Engineering]
    C --> D[Model Training]
    D --> E[Model Evaluation]
    E --> F[Model Deployment]

Loading

Methodological Approach Pipeline

VizWiz Classification Dataset (CVPR 2025)

The Dataset is selected from this Website (this is our challenge website as well). • https://vizwiz.org/tasks-and-datasets/image-classification/

We Downloaded the Training Dataset and Upload into Drive to further use in Project implementation. • https://drive.google.com/drive/folders/12b4Xm9NxRNe7IdYxqrX6NUQ9irgSwJf?usp=sharing

📂 Files in the Repository

• CVPR_Final_SOTA.ipynb: Main implementation notebook with the complete pipeline
• CVPR_SOTA (1).ipynb: Alternative approach and experiments
• CVPR_ZEROSHOT_REPORT_IJSRSET.pdf: Detailed research report on the approach
• ANN PRESENTATION CVPR.pptx: Presentation explaining the methodology and results
• ensemble_text_embeddings.pt: Pre-computed ensemble of text embeddings
• cluster_to_labels.pkl & cluster_centroids.pkl: Outputs from clustering analysis
• clip_train_embeddings.pkl: Stored embeddings for efficient inference

📈 Results and Performance

The solution achieves state-of-the-art performance on zero-shot image classification tasks by effectively leveraging:

• Novel prompt engineering techniques
• Multi-modal embedding alignment strategies
• Cluster-based class discovery
• Ensemble methods to improve robustness

graph TD
    A[Performance Metrics] --> B[Accuracy]
    A --> C[Precision]
    A --> D[Recall]
    A --> E[F1 Score]
    A --> F[Confusion Matrix]
    A --> G[ROC Curve]
    A --> H[AUC]

Loading

Performance Metrics Across Datasets

🔑 Challenges and Solutions

• Domain Gap: Addressed through prompt engineering and ensemble methods
• Semantic Ambiguity: Reduced with improved text-image alignment techniques
• Computational Efficiency: Optimized through embedding pre-computation and efficient inference strategies
• Class Imbalance: Mitigated with clustering-based approaches

graph TD
    A[Challenges] --> B[Data Scarcity]
    A --> C[Domain Shift]
    A --> D[Computational Cost]
    A --> E[Generalization]
    
    B --> F[Data Augmentation]
    C --> G[Domain Adaptation]
    D --> H[Efficient Algorithms]
    E --> I[Robust Models]

Loading

Challenges and Solutions Overview

📝 Future Work

• Exploration of more advanced transformer architectures
• Integration of additional modalities beyond text and images
• Self-supervised fine-tuning approaches to improve domain adaptation
• Improved prompt engineering strategies for better zero-shot generalization

graph TD
    A[Future Work] --> B[Expand Dataset]
    A --> C[Improve Model Architecture]
    A --> D[Real-time Deployment]
    A --> E[Cross-modal Learning]
    A --> F[Explainability]
    A --> G[Robustness Testing]

Loading

Future Research Directions

🌐 References

• OpenAI's CLIP (Contrastive Language-Image Pre-training)
• CVPR 2025 Zero-Shot Image Classification Challenge guidelines
• Related work in zero-shot learning and vision-language models

📞 Connect & Collaborate

Let's build the future together! 🌟

---

💬 Feedback & Questions

Found this repository helpful? ⭐ Star it!

Have questions or suggestions? 💭 Open an issue!

Want to collaborate? 🤝 Let's connect!