VISTA

VISTA is a multimodal video captioning system that generates accurate, context-aware captions by combining visual scene understanding with audio information. Instead of describing an entire video with a single generic caption, VISTA analyzes each scene individually to produce richer and more meaningful descriptions. The system first detects scene boundaries using histogram-based cut detection. For each scene, it extracts a representative key frame for fine visual detail, and a sprite sheet, a grid of frames to capture how the scene changes over time. In parallel, it transcribes spoken dialogue and identifies background sounds such as music or environmental noise, giving the captioning step additional context beyond what is visible. Each scene is described individually by a vision-language model, so its full attention stays on one scene at a time instead of splitting across the whole video at once. When a video has more than one scene, these per-scene descriptions are then combined into a single coherent video-level description. That description is passed to the Gemma model again, along with a detailed persona prompt for each of four tones formal, sarcastic, tech humor, and everyday humor instructing it how each style should sound. VISTA also runs on a fallback chain across three LLM backends, so a failure or empty response from one backend does not stop a caption from being generated.