LLM-as-a-Judge Paradigm and DeepAmbigQA Benchmark Highlight Biases in Automated Evaluation

A recent study outlines the "LLM-as-a-Judge" approach, arguing that treating large language models as infallible evaluators leads to systematic biases such as verbosity, position, and self‑enhancement effects. The authors recommend converting raw scoring into a calibrated "laboratory instrument" method, employing chain‑of‑thought prompting, reference‑guided prompts, structured output schemas, and statistical validation with Cohen's Kappa to achieve human‑level agreement.

Another paper introduces DeepAmbigQA, a dataset of 3,600 ambiguous multi‑hop questions designed to test LLMs' ability to resolve name ambiguity and gather complete answer sets. Experiments with state‑of‑the‑art models, including GPT‑5, show low exact‑match rates (13% on ambiguous questions, 21% on non‑ambiguous), underscoring the need for more robust QA systems that can handle complex, ambiguous queries.