MOSAIC
This project is an automated video captioning agent designed for the AMD Hackathon. The pipeline processes video content by separating the task into four straightforward steps: Ingestion: The system samples frames from the video and transcribes the audio track into text using a local transcription model. Extraction: The sampled frames and audio text are sent to a Vision-Language Model, which identifies the raw, objective facts of the scene (e.g., physical objects, setting, actions, and lighting). Compression: The raw facts are grouped and simplified into a clean timeline of events, removing redundant information. Synthesis: Finally, the timeline and key frames are fed into a large language model. The model is given strict persona instructions to apply the requested styles (Formal, Sarcastic, etc.) while staying firmly grounded in the factual timeline, ensuring the captions are both stylized and visually accurate.
Category tags:
For Innovators & Creators