YouTube RAG Chat
A production-oriented RAG application for conversational interaction with YouTube video content, including transcript extraction, embeddings, vector search, asynchronous ingestion, and retrieval-scoped chat.
Project overview
Brief description
A production-oriented RAG application for conversational interaction with YouTube video content, including transcript extraction, embeddings, vector search, asynchronous ingestion, and retrieval-scoped chat.
Role
I designed and implemented this project end-to-end as a practical AI application, covering backend architecture, ingestion workflows, vector storage, retrieval logic, AI integration, frontend development, and reliability-focused engineering patterns.
Full description
I designed and implemented YouTube RAG Chat as a production-shaped application for conversational interaction with YouTube content.
The system allows a user to submit a YouTube URL, after which the backend extracts or transcribes the video content, splits it into chunks, generates embeddings, stores them in PostgreSQL using pgvector, and makes the processed video available for retrieval-scoped chat.
I built the backend using Python and FastAPI, with clear boundaries between API handlers, services, repositories, AI gateways, and ingestion workflows. The goal was not just to make a demo that talks to a transcript, but to structure the application in a way that could be maintained, tested, and extended.
For ingestion, I implemented asynchronous background processing with Celery and Redis. The workflow handles long-running operations such as transcript extraction, audio processing, transcription fallback, chunking, embedding generation, and vector persistence. I added retry-aware processing, idempotent ingestion keyed by YouTube video ID, stale job recovery, and cleanup logic for failed persistence scenarios.
The retrieval layer was designed to keep conversations scoped to the selected video, so answers are grounded in the relevant document context rather than mixing information across unrelated videos. I used LangChain and OpenAI integration for embeddings, retrieval, and conversational response generation.
On the frontend side, I built a React/Vite interface for managing processed videos, switching context between videos, and chatting with streaming responses.
I also added focused tests around high-risk parts of the system, including ingestion lifecycle behavior, retry handling, retrieval scoping, and persistence failure recovery.
This project demonstrates how I approach AI application development: not as a thin wrapper around an LLM API, but as a real software system with ingestion, persistence, background processing, retrieval boundaries, error handling, and clear architectural structure.