LLMs and AI agents are a promising combination for complex business tasks. This is especially true in CRM. Due to the absence of realistic, publicly-available business data, it is difficult to evaluate their effectiveness in real life. The benchmarks that are currently available focus mainly on one-step interactions, or on narrow applications such as customer support. They tend to ignore broader areas, like sales, CPQ, and B2B. Also, they do not test the ability of agents to manage sensitive data. This makes it difficult to understand how LLM agents respond in a wide range of business situations and communication styles.
The focus of previous benchmarks has been largely customer service in B2C scenarios. This neglects key business functions, like sales and CPQ process, along with the unique challenges that B2B interaction presents, including long sales cycles. Many benchmarks do not reflect reality, as they often ignore multi-turn conversations or skip expert validation for tasks and environments. The absence of a confidentiality assessment is another critical flaw, especially in the workplace where AI agents interact with highly sensitive customer data and business information. The benchmarks are not able to assess data awareness and therefore fail to tackle serious concerns such as privacy risks, trust or legal issues.
CRMArena-Pro is a benchmark developed by Salesforce AI Research to evaluate LLM agents such as Gemini 2.5 Pro realistically in business environments. The benchmark includes expert-validated customer service, sales and CPQ tasks, encompassing both B2B contexts and B2C. The benchmark measures confidentiality and multi-turn conversational skills. Even top models like Gemini 2.5 Pro only achieve 58% accuracy for single-turn tasks. Performance drops to 35% when dealing with multi-turn situations. Gemini 2.5 Pro excels in the Workflow Execution task, with a score of 83%. Confidentiality handling, however, remains a problem across all tested models.
The CRMArena Pro benchmark was created in order to test LLM agents rigorously, using realistic business scenarios, such as customer service, sales and CPQ. The benchmark was built using GPT-4 to generate synthetic enterprise data that is structurally accurate. It then used Salesforce Schemas and Sandboxed Salesforce Organizations to simulate business environments. The benchmark includes 19 tasks that are grouped into four main skills: textual reasoning and workflow execution. The CRMArena Pro also features multi-turn conversation with simulated users, and it tests the confidentiality awareness. Expert evaluations confirm the realistic data and the surrounding environment. This ensures a reliable testing platform for LLM agent performances.
In the evaluation, top LLM agents were compared across 19 different business tasks. The focus was on completing each task and being aware of confidentiality. Metrics varied by task type—exact match was used for structured outputs, and F1 score for generative responses. LLM judge based on GPT-4o assessed whether the models refused to disclose sensitive information in a reasonable manner. Gemini 2.5-Pro, o1 and other models that have advanced reasoning clearly perform better than lighter or less-reasoned versions. This is especially true for complex tasks. The performance of the models was comparable in B2B as well as B2C, but there were subtle differences depending on their strength. Confidentiality-aware prompts improved refusal rates but sometimes reduced task accuracy, highlighting a trade-off between privacy and performance.
CRMArena Pro, as a benchmark for LLM agents to measure their ability in handling real world business situations within Customer Relationship Management. The benchmark includes 19 tasks that have been reviewed by experts, and are based on B2B or B2C scenarios. They cover sales, pricing, service and other operations. Although top agents were successful in single-turn task (approximately 58%), they dropped off dramatically in multiple-turn conversations, where their success rates fell to approximately 35%. Most other skills were difficult, but the execution of workflows was easy. The level of confidentiality awareness is low and often, prompting to improve it reduces task accuracy. This study reveals a gap between LLMs’ capabilities and enterprise needs.
Take a look at the Paper, GitHub Page, Hugging Face Page The following are some examples of how to get started: Technical Blog. This research is the work of researchers.
🆕 Did you Know? Marktechpost is the fastest-growing AI media platform—trusted by over 1 million monthly readers. Book a strategy call to discuss your campaign goals. Also, feel free to follow us on Twitter Join our Facebook group! 95k+ ML SubReddit Subscribe now our Newsletter.

