Video Captioning Penguin

# AMD Developer Hackathon ACT II - Track 2: Video Captioning Agent A Dockerized video captioning agent that uses **Gemma-4-31B-it INT8** (lokeshe09/gemma-4-31B-it-INT8) running locally on AMD GPUs via ROCm to generate captions in four styles: **formal**, **sarcastic**, **humorous_tech**, and **humorous_non_tech**. ## Architecture ``` tasks.json -> download video -> extract keyframes (OpenCV) -> Gemma 4 31B INT8 (multimodal) -> 4 style captions -> results.json ``` ### Pipeline 1. Reads `/input/tasks.json` containing video clip URLs 2. Downloads each video clip 3. Extracts 4-8 quality-scored keyframes using OpenCV (sharpness + contrast scoring) 4. Sends keyframes to Gemma-4-31B-it INT8 model with a structured prompt requesting 4 caption styles 5. Parses JSON output and writes `/output/results.json` ### Model - **Model**: `lokeshe09/gemma-4-31B-it-INT8` (INT8 quantized Gemma 4 31B) - **Quantization**: compressed-tensors INT8 (vision encoder, lm_head, embeddings, and norm layers in original precision) - **VRAM**: ~33GB for model weights, ~38-42GB total with KV cache - **Inference**: HuggingFace transformers with AutoModelForMultimodalLM ## I/O Format **Input** (`/input/tasks.json`): ```json [ { "task_id": "v1", "video_url": "https://example.com/video.mp4", "styles": ["formal", "sarcastic", "humorous_tech", "humorous_non_tech"] } ] ``` **Output** (`/output/results.json`): ```json [ { "task_id": "v1", "captions": { "formal": "...", "sarcastic": "...", "humorous_tech": "...", "humorous_non_tech": "..." } } ]