GemmaJudge: Gemma Attacks and Judges Itself

Every model update can introduce a new failure. GemmaJudge turns that risk into a repeatable release check: Gemma generates targeted adversarial prompts, the target model responds, and Gemma scores each response from 1 to 5 with reasoning and an evidence span. The shipped CLI accepts ASR and failed-case thresholds, emits a machine-readable JSON summary, and exits non-zero when a CI regression gate fails. Our committed AMD proof runs the official google/gemma-3-4b-it model as attacker and judge and google/gemma-3-1b-it as the target through local OpenAI-compatible vLLM endpoints on an AMD Instinct MI300X VF using ROCm 6.4. In the recorded three-case demonstration, the Gemma judge assigned failure scores to 3/3 cases, producing 100% judge-assigned ASR over that small run. The full multi-stage pipeline took 55.31 seconds, while each individual model request was capped at 25 seconds. Repeated temperature-0 scores were stable, but this measures deterministic score stability, not judge correctness. The public Streamlit application is an evidence dashboard over committed real artifacts rather than a public live-inference endpoint. It lets judges inspect the AMD run, individual attacks, target responses, judge reasoning, evidence spans, fine-tuning results, and a recorded five-target demonstration. GemmaJudge does not claim that adversarial testing is new. Its differentiator is an auditable, open-weight attacker-plus-judge loop with a self-hosted AMD path, case-level evidence, and a release decision. A single model family can share blind spots, so high-stakes use still requires human calibration or an independent second judge. Our ROCm LoRA study is deliberately scoped. On one 56-example validation split, also used for checkpoint selection, the tuned Gemma-3-4B judge improved valid JSON from 89.3% to 100.0% and pass/fail accuracy from 66.1% to 75.0%. This is a small internal validation study, not an independent benchmark or a production-readiness claim.