Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI
  • OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
  • DailyAI Briefing: A Lean-checked Riemann proof, plus 4 extra
AI-trends.todayAI-trends.today
Home»Tech»Use the LLM Arena as a judge to evaluate large language model outputs

Use the LLM Arena as a judge to evaluate large language model outputs

Tech By Gavin Wallace25/08/20255 Mins Read
Facebook Twitter LinkedIn Email
LifelongAgentBench: A Benchmark for Evaluating Continuous Learning in LLM-Based Agents
LifelongAgentBench: A Benchmark for Evaluating Continuous Learning in LLM-Based Agents
Share
Facebook Twitter LinkedIn Email

We will learn how to apply the LLM Arena as a Judge to evaluate outputs from large language models. Instead of assigning isolated numerical scores to each response, this method performs head-to-head comparisons between outputs to determine which one is better — based on criteria you define, such as helpfulness, clarity, or tone. Visit the FULL CODES here.

OpenAI GPT-4.1, Gemini 2.5 and GPT-5 will be used to create responses and evaluate the outputs. In order to show how this works, we will use an email support case. The context for the scenario is:

Thank you for your support  
Last week I had ordered a mouse, but instead I got a keyboard.  
Could you please solve this problem as soon as possible.  
You're welcome  
John 

Installing Dependencies

pip install deepeval google-genai openai

You’ll also need the API keys for both OpenAI (Google) and Google. You can also check out the FULL CODES here.

OpenAI API keys are required since Deepeval is used for evaluation.

Import os
Getpass Import
os.environ["OPENAI_API_KEY"] = getpass('Enter OpenAI API Key: ')
os.environ['GOOGLE_API_KEY'] = getpass('Enter Google API Key: ')

Definition of the context

The next step is to define the test scenario. This example is a scenario in which a user has reported receiving the incorrect product. The context_email will contain the message sent by the user. We can then create a prompt that generates a response using this context. See the FULL CODES here.

From deepeval.test_case, import ArenaTestCase LLMTestCase LLMTestCaseParams
Importing deepeval.metrics into ArenaGEval

context_email = """
Thank you for your support
Last week I had ordered a mouse, but instead I got a keyboard. 
Please resolve the issue as quickly as possible.
You're welcome
John
"""

prompt = f"""
{context_email}
--------

A: Please respond to the email from the customer.
"""

OpenAI Model Response

OpenAI import OpenAI
Client = OpenAI()

def get_openai_response(prompt: str, model: str = "gpt-4.1") -> str:
    response = client.chat.completions.create(
        model=model,
        messages=[
            {"role": "user", "content": prompt}
        ]
    )
 Choose return response.[0].message.content

openAI_response = get_openai_response(prompt=prompt)

Gemini Model Response

From Google, import Genai
Client = client()

def get_gemini_response(prompt, model="gemini-2.5-pro"):
    response = client.models.generate_content(
        model=model,
        contents=prompt
    )
 Text return for response
geminiResponse = get_gemini_response(prompt=prompt)

Determining the Arena Test Case

Here, we set up the ArenaTestCase to compare the outputs of two models — GPT-4 and Gemini — for the same input prompt. The same context_email is sent to both models, and the generated responses in geminiResponse are then stored for evaluation. Visit the FULL CODES here.

a_test_case = ArenaTestCase(
    contestants={
        "GPT-4": LLMTestCase(
            input="Write a response to the customer email above.",
            context=[context_email],
            actual_output=openAI_response,
        ),
        "Gemini": LLMTestCase(
            input="Write a response to the customer email above.",
            context=[context_email],
            actual_output=geminiResponse,
        ),
    },
)

How to Set Up an Evaluation Metric

The ArenaGEval support email quality is defined here. The evaluation focuses on empathy, professionalism, and clarity — aiming to identify the response that is understanding, polite, and concise. GPT-5 is used as an evaluator, with verbose recording enabled to gain better insights. See the FULL CODES here.

Measurement = ArenaGEval
    name="Support Email Quality",
    criteria=(
        "Select the response that best balances empathy, professionalism, and clarity. "
        "It should sound understanding, polite, and be succinct."
    ),
    evaluation_params=[
        LLMTestCaseParams.CONTEXT,
        LLMTestCaseParams.INPUT,
        LLMTestCaseParams.ACTUAL_OUTPUT,
    ],
    model="gpt-5",  
    verbose_mode=True
)

Run the evaluation

metric.measure(a_test_case)
**************************************************
Support Email Quality [Arena GEval] Verbose Logs
**************************************************
Criteria:
Choose the answer that balances professionalism with clarity and empathy. This response should be empathetic. 
Please be polite. 
 
Steps to evaluate:
[
    "From the Context and Input, identify the user's intent, needs, tone, and any constraints or specifics to be 
addressed.",
    "Verify the Actual Output directly responds to the Input, uses relevant details from the Context, and remains 
consistent with any constraints.",
    "Evaluate empathy: check whether the Actual Output acknowledges the user's situation/feelings from the 
Context/Input in a polite, understanding way.",
    "Evaluate professionalism and clarity: ensure respectful, blame-free tone and concise, easy-to-understand 
wording; choose the response that best balances empathy, professionalism, and succinct clarity."
] 
 
Winner: GPT-4
 
GPT-4 is a professional, single-sentence email which addresses context directly (acknowledgements). 
It apologizes for the mistake and details the next step (sending the wireless mouse). 
Then, a polite step of verification (such as requesting a photograph) is added. This is the best way to do it. 
The software balances empathy with clarity. In contrast, Gemini includes multiple 
While empathetic, and 
Detail (e.g. acknowledging frustration, offering pre-paid labels), the multiple-options format and an assertion that you have already located the order are all factors which reduce professionalism.
======================================================================

GPT-4 was rated higher than the other model for generating an email support that balances empathy, professionalism and clarity. GPT-4’s response was notable because it addressed the situation directly, was polite and action-oriented. It apologized for the mistake, confirmed the problem, and explained clearly the next steps in order to fix it. This included sending the right item, as well as providing instructions on how to return it. It was the perfect tone to match with user needs for an empathetic and clear response. Gemini’s response was also empathetic but included a number of response options, as well as unnecessary comments, which diminished its professionalism and clarity. GPT-4 has demonstrated its ability to communicate in a professional, considerate manner while being focused on the customer.


Click here to find out more FULL CODES here. Check out our website to learn more. GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Don’t forget about our 100k+ ML SubReddit Subscribe Now our Newsletter.


I’m a Civil Engineering graduate (2022) at Jamia Millia Islamia in New Delhi. I have an interest in Data Science and in particular, Neural Networks, as well as their applications in different areas.

large language model
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026
Top News

Pope Leo XIV declares AI a threat to human dignity and workers’ rights

China’s AI Boyfriend Business is Taking on a Life Of Its Own

The Rogue AI Agents aren’t evil. The Rogue AI Agents Aren’t Evil.

Hackers are finding new ways to hide malware in DNS records

I Wasn’t Sure I Wanted Anthropic to Pay Me for My Books—I Do Now

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

You can also “Safe AI” Can Companies survive in an AI landscape that is unrestrained? • AI Blog

27/05/2025

The Prompt Guide – A Statue of Ahmed el Ahmed

16/12/2025
Latest News

Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI

22/11/2026

OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.