CAPTION AI

CaptionAI represents a paradigm shift in how we process, understand, and accessibility-enhance video content. Designed for the AMD Developer Hackathon ACT II, this robust pipeline seamlessly bridges the gap between raw video files and contextualized, multi-audience metadata. At its core, the system utilizes PySceneDetect and structural similarity indexing (SSIM) to intelligently slice uploaded video files into distinct, logically consistent visual scenes. Following scene boundary detection, OpenCV extracts high-fidelity representative frames from each segment. Rather than relying on traditional, single-tone transcriptions, CaptionAI unlocks the true potential of multimodal AI. By offloading inference to the cutting-edge MiniMax M3 model via the Fireworks AI API, the pipeline concurrently generates four distinct caption variations per scene: Formal, Sarcastic, Humorous (Tech), and Humorous (Non-Tech). The backend is powered by a high-performance FastAPI server featuring a custom threading daemon, ensuring non-blocking execution. As the multimodal LLM processes each frame, the results and pipeline progression are streamed in real-time to the client via native WebSockets. This backend feeds into a stunning, responsive React frontend. Featuring a modern, glassmorphic UI built with Tailwind CSS, the dashboard provides a premium user experience, complete with granular pipeline controls, live progress tracking, and instantaneous caption copying. Ultimately, CaptionAI makes video metadata generation dynamic, effortless, and instantly tailored to any target audience.