We will learn how to apply the LLM Arena as a Judge to evaluate outputs from large language models. Instead of assigning isolated numerical scores to each response, this method performs head-to-head comparisons between outputs to determine which one is better — based on criteria you define, such as helpfulness, clarity, or tone. Visit the FULL CODES here.
OpenAI GPT-4.1, Gemini 2.5 and GPT-5 will be used to create responses and evaluate the outputs. In order to show how this works, we will use an email support case. The context for the scenario is:
Thank you for your support
Last week I had ordered a mouse, but instead I got a keyboard.
Could you please solve this problem as soon as possible.
You're welcome
John
Installing Dependencies
pip install deepeval google-genai openai
You’ll also need the API keys for both OpenAI (Google) and Google. You can also check out the FULL CODES here.
OpenAI API keys are required since Deepeval is used for evaluation.
Import os
Getpass Import
os.environ["OPENAI_API_KEY"] = getpass('Enter OpenAI API Key: ')
os.environ['GOOGLE_API_KEY'] = getpass('Enter Google API Key: ')
Definition of the context
The next step is to define the test scenario. This example is a scenario in which a user has reported receiving the incorrect product. The context_email will contain the message sent by the user. We can then create a prompt that generates a response using this context. See the FULL CODES here.
From deepeval.test_case, import ArenaTestCase LLMTestCase LLMTestCaseParams
Importing deepeval.metrics into ArenaGEval
context_email = """
Thank you for your support
Last week I had ordered a mouse, but instead I got a keyboard.
Please resolve the issue as quickly as possible.
You're welcome
John
"""
prompt = f"""
{context_email}
--------
A: Please respond to the email from the customer.
"""
OpenAI Model Response
OpenAI import OpenAI
Client = OpenAI()
def get_openai_response(prompt: str, model: str = "gpt-4.1") -> str:
response = client.chat.completions.create(
model=model,
messages=[
{"role": "user", "content": prompt}
]
)
Choose return response.[0].message.content
openAI_response = get_openai_response(prompt=prompt)
Gemini Model Response
From Google, import Genai
Client = client()
def get_gemini_response(prompt, model="gemini-2.5-pro"):
response = client.models.generate_content(
model=model,
contents=prompt
)
Text return for response
geminiResponse = get_gemini_response(prompt=prompt)
Determining the Arena Test Case
Here, we set up the ArenaTestCase to compare the outputs of two models — GPT-4 and Gemini — for the same input prompt. The same context_email is sent to both models, and the generated responses in geminiResponse are then stored for evaluation. Visit the FULL CODES here.
a_test_case = ArenaTestCase(
contestants={
"GPT-4": LLMTestCase(
input="Write a response to the customer email above.",
context=[context_email],
actual_output=openAI_response,
),
"Gemini": LLMTestCase(
input="Write a response to the customer email above.",
context=[context_email],
actual_output=geminiResponse,
),
},
)
How to Set Up an Evaluation Metric
The ArenaGEval support email quality is defined here. The evaluation focuses on empathy, professionalism, and clarity — aiming to identify the response that is understanding, polite, and concise. GPT-5 is used as an evaluator, with verbose recording enabled to gain better insights. See the FULL CODES here.
Measurement = ArenaGEval
name="Support Email Quality",
criteria=(
"Select the response that best balances empathy, professionalism, and clarity. "
"It should sound understanding, polite, and be succinct."
),
evaluation_params=[
LLMTestCaseParams.CONTEXT,
LLMTestCaseParams.INPUT,
LLMTestCaseParams.ACTUAL_OUTPUT,
],
model="gpt-5",
verbose_mode=True
)
Run the evaluation
metric.measure(a_test_case)
**************************************************
Support Email Quality [Arena GEval] Verbose Logs
**************************************************
Criteria:
Choose the answer that balances professionalism with clarity and empathy. This response should be empathetic.
Please be polite.
Steps to evaluate:
[
"From the Context and Input, identify the user's intent, needs, tone, and any constraints or specifics to be
addressed.",
"Verify the Actual Output directly responds to the Input, uses relevant details from the Context, and remains
consistent with any constraints.",
"Evaluate empathy: check whether the Actual Output acknowledges the user's situation/feelings from the
Context/Input in a polite, understanding way.",
"Evaluate professionalism and clarity: ensure respectful, blame-free tone and concise, easy-to-understand
wording; choose the response that best balances empathy, professionalism, and succinct clarity."
]
Winner: GPT-4
GPT-4 is a professional, single-sentence email which addresses context directly (acknowledgements).
It apologizes for the mistake and details the next step (sending the wireless mouse).
Then, a polite step of verification (such as requesting a photograph) is added. This is the best way to do it.
The software balances empathy with clarity. In contrast, Gemini includes multiple
While empathetic, and
Detail (e.g. acknowledging frustration, offering pre-paid labels), the multiple-options format and an assertion that you have already located the order are all factors which reduce professionalism.
======================================================================
GPT-4 was rated higher than the other model for generating an email support that balances empathy, professionalism and clarity. GPT-4’s response was notable because it addressed the situation directly, was polite and action-oriented. It apologized for the mistake, confirmed the problem, and explained clearly the next steps in order to fix it. This included sending the right item, as well as providing instructions on how to return it. It was the perfect tone to match with user needs for an empathetic and clear response. Gemini’s response was also empathetic but included a number of response options, as well as unnecessary comments, which diminished its professionalism and clarity. GPT-4 has demonstrated its ability to communicate in a professional, considerate manner while being focused on the customer.
Click here to find out more FULL CODES here. Check out our website to learn more. GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Don’t forget about our 100k+ ML SubReddit Subscribe Now our Newsletter.

