Framewise — Multi-Style Video Captioning Pipeline

Framewise is a headless video-captioning agent for AMD Developer Hackathon Act II, Track 2. Given any video clip, it writes four style-matched captions — formal, sarcastic, humorous-tech, humorous-non-tech — grounded in what the video actually shows. The pipeline splits perception from writing. A vision-language model (Kimi-K2 on Fireworks) reads sampled, downscaled keyframes into a structured, factual scene description plus an explicit list of uncertain details. A second independent look cross-checks the first to catch misreads before they reach the writer. A separate writer model (GLM on Fireworks) renders that single grounded description into each style via per-style prompts and few-shot exemplars, followed by an LLM judge-and-repair pass that catches and fixes captions inventing details the source doesn't support. The container implements the Track 2 contract exactly: it reads /input/tasks.json, processes clips concurrently under a soft time budget, and always writes valid /output/results.json — every requested style is guaranteed present via a fallback ladder, and the process exits 0 even on bad input. It's a slim linux/amd64 image (~226MB, under 1 second cold start) authenticating to Fireworks purely through environment variables injected at run time, with no credentials baked into any layer. Beyond prompting, we built a full synthetic fine-tuning pipeline: an LLM invents diverse scene descriptions across twelve visual categories, a teacher model writes candidate captions per style, and a two-judge filter keeps only examples scoring high on both accuracy and style. That produced about 1,170 supervised examples and 217 preference pairs, used to fine-tune Qwen2.5-7B with LoRA — first SFT, then DPO. We kept the tuned model as a benchmarked, deployable asset, but shipped GLM as the writer since its style score was already near ceiling on our verified two-judge leaderboard baseline of 0.865.