An end-to-end Retrieval-Augmented Generation (RAG) system that transforms educational videos into a searchable AI knowledge assistant.
This project processes videos, extracts audio, transcribes speech using Whisper, generates semantic embeddings, retrieves context-aware information using vector similarity search, and produces grounded responses using Large Language Models (LLMs).
The system is designed to reduce hallucinations and improve factual consistency over custom video datasets.
The pipeline converts raw video content into semantically searchable knowledge and generates grounded responses using retrieval-augmented generation.
- Video-to-text knowledge extraction pipeline
- Whisper-based multilingual transcription
- Timestamp-aware transcript segmentation
- Chunk optimization and merging
- Semantic retrieval using vector embeddings
- Context-aware grounded response generation
- Hallucination reduction through RAG
- Video timestamp retrieval for precise navigation
- Modular and scalable AI pipeline architecture
where the batch learning is taught in this course?
Video 4: Batch Machine Learning - Offline Vs Online Learning - Machine Learning Types
Time:
2:17–2:35
3:17–3:28
3:28–3:40
7:04–7:14
11:06–11:16
The system retrieves semantically relevant transcript chunks and returns grounded responses along with timestamps from the original video.
Raw video files are stored inside:
data/videos/
Supported use cases:
- Lecture recordings
- Tutorials
- Podcasts
- Educational courses
- Meetings and discussions
Videos are converted into MP3 audio using:
video_processing.pyGenerated audio files are stored in:
data/audios/
This stage uses FFmpeg for efficient audio extraction.
Audio files are transcribed using OpenAI Whisper:
transcription.pyThe transcription pipeline:
- extracts spoken text
- preserves timestamps
- supports multilingual translation workflows
- generates structured JSON transcripts
Raw transcripts are stored in:
data/raw_transcripts/
Initial Whisper segments are often too small for effective retrieval.
To improve semantic coherence, an additional chunk-merging stage was introduced using:
chunking.pyThis stage:
- merges fragmented segments
- improves contextual continuity
- increases retrieval relevance
- reduces noisy retrieval outputs
Processed transcripts are stored in:
data/transcripts/
Processed transcript chunks are converted into vector embeddings using:
embeddings.pyEmbeddings are generated using the locally hosted:
bge-m3
embedding model through Ollama.
Generated embeddings are stored inside:
data/vector_store/
Current vector storage:
- Joblib serialized embeddings
Future support planned:
- FAISS
- ChromaDB
The RAG pipeline receives a user query and:
- generates semantic query embeddings
- computes cosine similarity
- retrieves relevant transcript chunks
- injects retrieved context into prompts
- generates grounded responses using LLMs
Pipeline execution:
rag_pipeline.pyThe current implementation uses:
- OpenRouter API
- GPT OSS 120B model
- cosine similarity retrieval
This improves:
- factual consistency
- contextual grounding
- retrieval relevance
- hallucination reduction
- Python
- OpenAI Whisper
- Ollama
- OpenRouter API
- NumPy
- Pandas
- Joblib
- FFmpeg
- Retrieval-Augmented Generation (RAG)
- Semantic Search
- Vector Embeddings
- Cosine Similarity Search
- Prompt Engineering
- Context Injection
- Multilingual Speech Transcription
LLMs often hallucinate when answering domain-specific questions.
RAG improves reliability by grounding generation on retrieved contextual information instead of relying solely on parametric memory.
The system leverages Whisper timestamp segments to preserve temporal context during transcript processing.
This improves:
- retrieval alignment
- semantic continuity
- context tracking across long videos
Initial transcript segmentation produced overly small chunks, reducing semantic coherence during retrieval.
To improve retrieval quality, an additional chunk-merging stage was introduced to:
- preserve contextual continuity
- improve semantic relevance
- reduce fragmented retrieval outputs
Instead of relying entirely on high-level frameworks, cosine similarity retrieval was implemented manually using NumPy and scikit-learn.
This provided:
- better control over retrieval logic
- easier experimentation
- transparent similarity scoring
- Handling noisy transcriptions
- Balancing chunk size vs retrieval quality
- Preserving semantic continuity
- Reducing hallucinated outputs
- Managing multilingual transcription workflows
- Optimizing retrieval relevance
- Multi-agent retrieval workflows
- Conversational memory
- Hybrid search (semantic + keyword)
- Multimodal retrieval using slides/screenshots
- Automatic response evaluation framework
- Real-time streaming transcription
- Temporal video segmentation
- FAISS-based indexing
- Reranking models for improved retrieval quality
AGENTIC-VIDEO-RAG/
│
├── README.md
├── requirements.txt
├── .gitignore
├── .env
├── main.py
│
├── data/
│ ├── audios/
│ ├── raw_transcripts/
│ ├── transcripts/
│ ├── vector_store/
│ │ └── embeddings.joblib
│ └── videos/
│
├── outputs/
│
├── screenshots/
│
├── src/
│ ├── prompts/
│ │ ├── prompt.txt
│ │ └── response.txt
│ │
│ ├── chunking.py
│ ├── embeddings.py
│ ├── rag_pipeline.py
│ ├── transcription.py
│ └── video_processing.py
│
├── requirements.txt
└── README.md
git clone https://github.com/your-username/AGENTIC-VIDEO-RAG.git
cd AGENTIC-VIDEO-RAGpip install -r requirements.txtpython main.pypython src/video_processing.py
python src/transcription.py
python src/chunking.py
python src/embeddings.py
python src/rag_pipeline.pyThrough this project, I explored:
- end-to-end AI pipeline development
- practical RAG system design
- vector embedding workflows
- retrieval optimization techniques
- grounding strategies for LLMs
- multilingual AI processing
- modular AI system architecture
Jaideep Attri Indian Institute of Technology Roorkee
Focused on AI systems, retrieval pipelines, and applied machine learning.