ScreenSage: Local Vision-Grounded LLM Agent

ScreenSage is a fully local, vision-grounded AI agent that operates your desktop like a human: it screenshots, detects UI elements, decides one next action, executes it, and repeats — entirely on-device on AMD hardware. The perception stage runs YOLOv10 through PyTorch's ROCm build to detect buttons, icons, and text fields in real time, assigning each a stable ID and pixel-center coordinate. Rather than making the LLM guess raw pixel coordinates, the reasoning stage — a local Qwen2.5-7B-Instruct model served via vLLM's ROCm backend — receives a JSON list of visible elements plus the goal and action history, and returns exactly one next action as strict JSON: click, move, type, key press, scroll, wait, or done. An input controller resolves that action against the freshest detections and drives the mouse/keyboard. This capture → detect → reason → act loop repeats until the goal is complete or a step cap is hit, so the agent never chains actions without re-observing the screen. Because reasoning is grounded in generic detected elements rather than a game- or app-specific API, ScreenSage works on any visible window — a game, spreadsheet, form, or settings panel. Everything runs offline after model download: no cloud API keys, no telemetry. It ships with a live Flask/SocketIO dashboard streaming each step with latency and device info, a one-command Docker deployment on AMD's ROCm PyTorch image, and setup scripts that install ROCm PyTorch and vLLM automatically. Both models are verified at runtime to run on the AMD GPU, and every step issues a fresh inference against a live screenshot — nothing is hardcoded or cached.