CaptionForge AI–Multimodal Video Captioning Agent

CaptionForge AI is a production-grade multimodal video captioning system developed for the AMD Developer Hackathon Video Captioning Track. Instead of directly prompting a language model with raw video frames, the system follows an evidence-first reasoning pipeline designed to maximize factual accuracy and style consistency on unseen videos. The pipeline automatically downloads videos from the provided task list, validates input, detects scene boundaries, extracts representative keyframes, transcribes speech using Whisper, extracts on-screen text using OCR, and constructs a structured temporal representation of the video's content. This multimodal evidence is then fused into a unified semantic context that is provided to Gemma 4 through the Google Gemini API to generate a canonical factual caption. The factual caption is subsequently rewritten into four required styles—Formal, Sarcastic, Humorous-Tech, and Humorous-Non-Tech—while preserving identical facts and chronology. A verification stage checks for factual consistency, hallucinations, and style adherence before generating the final output. The application is fully containerized using Docker and automatically processes /input/tasks.json, producing /output/results.json exactly as required by the competition. The architecture emphasizes modularity, scalability, asynchronous processing, robust error handling, structured logging, and production-ready engineering practices. By combining computer vision, speech understanding, OCR, temporal reasoning, and multimodal language models, CaptionForge AI delivers accurate, context-aware captions that generalize well across diverse video categories, including people, nature, sports, urban environments, food, weather, and technology.