judgebench SigLIP brand judge v3 (hardened)

Part of judgebench: which image-QC judges survive optimization pressure? A brand-fidelity judge for rhode (beauty brand). A judge = these SigLIP weights

• calibration.json (rhode train-split centroid + Platt scaling params) in this repo.

Base: google/siglip-so400m-patch14-384 (full SiglipModel, contrastively fine-tuned).

Training: the v1 recipe plus SRPO-hacked images (outputs of the gradient attack on v1) folded in as a third negative class.

Report card (judgebench Phase 1, 2,622-item test set): real-rhode brand AUC 0.997 (v1 0.99); real rhode scores 0.97 vs competitors 0.06.

The arms-race result: the seen attack is fully defeated (SRPO hack images: v1 scored 0.84 → v3 scores 0.00) while the unseen DPO attack is only dampened (0.47 → 0.28). Hardening against an observed exploit works; novel attacks still get partial traction.

Scoring

Score = Platt-calibrated cosine similarity between the image embedding and the rhode centroid:

import json, numpy as np, torch
from PIL import Image
from transformers import SiglipModel, SiglipImageProcessor
from huggingface_hub import hf_hub_download

repo = "Gupta28/judgebench-siglip-judge-v3-hardened"
model = SiglipModel.from_pretrained(repo)
proc = SiglipImageProcessor.from_pretrained(repo)
params = json.load(open(hf_hub_download(repo, "calibration.json")))
centroid = np.array(params["centroid"])  # L2-normalized rhode train centroid

img = Image.open("image.jpg")
with torch.no_grad():
    emb = model.get_image_features(**proc(images=img, return_tensors="pt"))
v = emb[0].numpy(); v /= np.linalg.norm(v)
cos = float(v @ centroid)
score = 1 / (1 + np.exp(-(params["platt_a"] * cos + params["platt_b"])))  # calibrated on-brand prob

Full evaluation protocol, test-set construction, and findings: https://github.com/amargupta0428/judgebench.