FourVoice Captioner

FourVoice is an autonomous video captioning pipeline built for Track 2 of the AMD Act 2 x lablab.ai hackathon. Given any short video clip (30 seconds to 2 minutes), it generates four tonally distinct captions — formal, sarcastic, humorous-tech, and humorous-non-tech — while ensuring every caption stays factually grounded in what actually happens in the video. Unlike template-based caption generators that guess or hallucinate details, FourVoice uses an audio-first, vision-fallback approach: it transcribes spoken dialogue with Whisper when present, and falls back to frame-by-frame visual analysis when a clip is silent, musical, or static. This ensures captions never invent details that aren't actually in the source material — a real problem we identified and fixed during development, where an early version of the pipeline captioned the wrong on-screen detail entirely. All four style captions are generated from a single, shared, fact-checked description, so tone changes across styles without the underlying facts drifting between them. A self-QC stage checks each caption against the source facts and regenerates any caption that scores poorly, and the pipeline reports its own best-fit style recommendation with reasoning, plus a confidence score reflecting how much of the caption is backed by direct evidence versus inference. The project is fully containerized with Docker, using Fireworks AI-hosted models (Whisper, Llama 3.3 70B, Gemma, and Qwen3 32B) across its pipeline stages, and is deployed as a live web application with a Vercel frontend and Render backend for interactive demo access alongside the standalone Docker pipeline used for batch evaluation.