Hacienda - Video Captioning AI Agent

Video content is exploding, making it nearly impossible for creators, developers, and platforms to manually caption and categorize every upload. Hacienda solves this by acting as a fully autonomous, multimodal video-captioning agent built specifically for Track 2 of the LabLab.ai × AMD Developer Hackathon. Hacienda completely automates the process of understanding video. When fed a video clip, our Python-based pipeline instantly extracts visual keyframes using adaptive chunking and rips the audio track for transcription via Groq's Whisper-large-v3. It then passes the visual evidence to the cutting-edge minimax-m3 vision model on Fireworks AI, which acts as the "eyes" of the operation to extract concrete, literal details about the setting, subjects, and actions. Once the visual and audio facts are collected, they are passed to the "brain" of the operation: a custom, fine-tuned Gemma-4-e4b model running on Fireworks AI. This fine-tuned Gemma model drafts four stylistically distinct captions—ranging from professional and formal to witty, tech-humor sarcasm. Finally, Hacienda employs a self-evaluating LLM loop that scores each caption for tone and accuracy, automatically repairing and rewriting any weak captions before returning the final JSON. We wrapped this entire robust pipeline in an interactive FastAPI web interface and a Docker container, providing an end-to-end, deterministic, and highly creative video-analysis solution!