Large Language Models (LLMs), such as GPT-4, have exhibited exceptional capabilities across a wide range of natural language processing (NLP) tasks, including applications in medicine, LLM-based agents, science, and data synthesis. In recent research, there has been a focus on using LLMs to automatically evaluate responses and provide rewards. This methodology is commonly known as LLM-as-a-Judge, which involves using LLMs to assess responses in two main ways: comparing pairs of answers to determine superiority or directly scoring individual answers based on specific criteria. This method has been primarily applied in scoring and pairwise comparison tasks, yielding notable achievements.

Despite the increasing adoption of LLM-as-a-Judge, concerns regarding its reliability have emerged due to potential biases within the models. These biases cast doubt on the trustworthiness of LLMs, both in their evaluation processes and in their alignment with principles of fairness and transparency. For instance, extensive experiments have examined positional preferences in LLM-as-a-Judge, while previous research revealed that popular opinions reflecting majority viewpoints may compromise the fairness of LLM evaluations. Furthermore, experiments demonstrated that fabricated citations could disrupt the judgment accuracy of LLMs.

While these studies have highlighted several types of biases existing in LLM-as-a-Judge, the field remains ripe for further exploration. Firstly, the existing analyses of bias are relatively narrow in scope, which limits the development of a comprehensive framework for evaluating the multifaceted biases affecting LLM-as-a-Judge. Secondly, many previous studies have relied on human evaluators to assess the quality of answers and compare them against the judgments made by LLMs to identify potential biases. This methodology incurs substantial costs and introduces human subjectivity, complicating the establishment of reliable ground truth and the reproducibility of findings. Additionally, the limited size and scope of test data increase the risk of random interference, potentially obscuring the true extent of bias in LLM judgments. A more detailed discussion of related work is available in the appendices.

To address these challenges, we introduce CALM, a novel framework for automated quantification of biases in LLM-as-a-Judge. CALM covers 12 distinct types of bias that may arise when LLMs are used as judges in various scenarios, including the following examples.

Correctness of Scientific Reasoning. When using LLMs to judge reasoning results in scientific QA or answers to math problems, bias often occurs in understanding the content. Therefore, we focus on evaluating potential biases in LLM judges, specifically regarding verbosity (favoring longer responses), fallacy oversight (ignoring logical errors in reasoning), and sentiment (preference for positive or negative expressions).

Improvement on Answer Refinement. Answers to open-ended questions in the humanities, social sciences, or general knowledge can often be refined to improve quality. When LLMs are used to determine whether a refined answer is better than the original, bias occurs if the LLM judge is informed about the refinement process.

Alignment to Human Feedback. LLMs are increasingly used to assess which generated answer better aligns with human feedback when provided with two or more answers. In such cases, alignment bias often occurs, e.g., the LLM judge may favor responses that superficially match the feedback rather than accurately reflecting deeper alignment, or favor answers based on their placement (position bias), or favor answers they generated themselves (self-preference). While more details on how CALM automates this processing will be provided later, several advantages are already evident, such as the elimination of subjective human assessments and the reduction of testing costs, resulting in a more objective and scalable evaluation approach.

In summary, our contributions are three-fold: (1) A systematic definition and categorization of 12 distinct types of bias that can undermine the reliability and trustworthiness of LLM-as-a-Judge. (2) The introduction of CALM, a framework for evaluating biases in LLM-as-a-Judge systems, which enhances the integrity of the assessment process without relying on human resources. (3) An extensive evaluation of six popular LLMs using the CALM framework reveals that while some LLMs demonstrate notable fairness in judgment, there remains significant room for improvement in achieving more robust decision-making across various types of bias.