Posts in tags: "Spearman" (2 posts)
Most AI evaluation tools let LLMs judge LLMs without any human anchor. ScoredQA Calibration flips that — a domain expert rates 50 answers by hand, and we compute Spearman ρ between their ratings and each LLM judge to find which judge actually agrees with the human before we trust its scores.
Read MoreWe ran a 200-item RAG arena on the AskTheDoctor corpus across three models and two retrieval configurations. The headline (v2-atd ≈ Llama 4 Scout, both at ~0.58) is interesting. The methodology footnote is more interesting: we then re-judged 415 of those answers with two different LLM judges and got Spearman ρ = 0.55 between them. That number is the case for human calibration.
Read More
