Captionforge ai

This project is a video captioning agent built for the Video Captioning track. Given a video clip, it generates captions in four required styles — formal, sarcastic, humorous_tech, and humorous_non_tech — derived from a single shared understanding of the clip's content, so caption accuracy is decoupled from stylistic variation rather than re-guessed for each style independently. The pipeline begins with scene detection to segment the clip into meaningful cuts rather than sampling at fixed intervals. Within each scene, candidate frames are sampled and encoded as CLIP embeddings, and a greedy farthest-point selection algorithm picks the subset that is most visually distinct from one another, capped at 15 keyframes per clip. This keeps the frame budget small while still covering the visual range of the scene, which matters for generalizing to unseen clips rather than overfitting to a handful of examples. Each selected keyframe is analyzed by a Vision-Language Model (Groq running Llama-4 Scout), producing a factual description of the scene. These per-frame outputs are reconciled through a consensus step, reducing the risk that a single misread frame propagates into the final caption. The reconciled description is then passed to an LLM call that generates all four styled captions together, with each style aware of what the others produced, preventing the four outputs from reading as the same sentence with different adjectives swapped in. Results are cached per scene, and VLM and LLM calls run concurrently, keeping the full pipeline within the 10-minute runtime limit per clip. The container reads tasks from /input/tasks.json and writes validated results to /output/results.json.