MultiVerse of Captions

Video Captioning Agent is an AI-powered system that automatically generates high-quality captions for short videos by combining visual understanding, audio transcription, and large language models. The pipeline first extracts representative video frames and transcribes speech using Whisper, then creates a structured scene description using a multimodal vision-language model. Based on this scene representation, captions are generated in four distinct styles: Formal, Sarcastic, Humorous Tech, and Humorous Non-Tech. To improve caption quality, every generated caption is evaluated by an LLM-based judge that checks factual accuracy, scene grounding, hallucinations, and style adherence. Failed captions are automatically refined using the judge's feedback, and if no caption passes within the refinement budget, a selector model chooses the strongest candidate from all generated attempts. The entire system is optimized for parallel execution, enabling multiple videos and caption styles to be processed concurrently while remaining robust through automatic retries and fallback mechanisms. The project is fully Dockerized and includes an interactive Hugging Face demo where users can provide a public video URL (up to two minutes long) and instantly receive captions in all four supported styles.