This tutorial provides a benchmarking framework that allows us to compare various AI agent systems against real-world software challenges. The challenges range from API integration and data transformation, to performance optimization and workflow automation. We can gain an understanding of the strengths and weaknesses of agents in enterprise environments by running benchmarks, visualizing performance metrics such as execution time and accuracy and comparing them to other agents. Visit the Full Codes here.
Import json
import time
Random Import
Typing import List, Dict or Callable
Import dataclasses from asdict
import pandas as pd
Import numpy as an np
Matplotlib.pyplot can be imported as a plt
Import Seaborn as Sns
@dataclass
class:
Id: str
Name: str
description: str
Category:
Complexity:
Expected_output:
@dataclass
class BenchmarkResult:
task_id: str
Agent_name : str
Success: bool
execution_time : float
Accuracy: float
error_message: str = ""
The EnterpriseTaskSuite class:
def __init__(self):
self.tasks = self._create_tasks()
def _create_tasks(self) -> List[Task]:
You can return to your original language by clicking here. [
Task("data_transform", "CSV Data Transformation",
"Transform customer data by aggregating sales", "data_processing", 3,
{"total_sales": 15000, "avg_order": 750}),
Task("api_integration", "REST API Integration",
"Parse API response and extract key metrics", "integration", 2,
{"status": "success", "active_users": 1250}),
Task("workflow_automation", "Multi-Step Workflow",
"Execute data validation -> processing -> reporting", "automation", 4,
{"validated": True, "processed": 100, "report_generated": True}),
Task("error_handling", "Error Recovery",
"Handle malformed data gracefully", "reliability", 3,
{"errors_caught": 5, "recovery_success": True}),
Task("optimization", "Query Optimization",
"Optimize database query performance", "performance", 5,
{"execution_time_ms": 45, "rows_scanned": 1000}),
Task("data_validation", "Schema Validation",
"Validate data against business rules", "validation", 2,
{"valid_records": 95, "invalid_records": 5}),
Task("reporting", "Executive Dashboard",
"Generate KPI summary report", "analytics", 3,
{"revenue": 125000, "growth": 0.15, "customer_count": 450}),
Task("integration_test", "System Integration",
"Test end-to-end integration flow", "testing", 4,
{"all_systems_connected": True, "latency_ms": 120}),
]
def get_task(self, task_id: str) -> Task:
Return next ((t for task_id in self.tasks, if the t.id is equal to task_id), No)
We create the Task and BenchmarkResult data classes. The Task and BenchmarkResult class data are created and the EnterpriseTaskSuite is initialized. It contains enterprise relevant tasks like data transformation, integration, and reporting. The foundation was laid for comparing different agents in these tasks. See the Full Codes here.
Class BaseAgent
def __init__(self, name: str):
The name of the self is also called "self"
def execute(self, task: Task) -> Dict[str, Any]:
Raising NotImplementedError
class RuleBasedAgent(BaseAgent):
def execute(self, task: Task) -> Dict[str, Any]:
time.sleep(random.uniform(0.1, 0.3))
If category ==Return "data_processing":
return {"total_sales": 15000 + random.randint(-500, 500),
"avg_order": 750 + random.randint(-50, 50)}
Task.category :=Return "integration":
return {"status": "success", "active_users": 1250}
Elif task.category==Return "automation":
return {"validated": True, "processed": 98, "report_generated": True}
else:
return task.expected_output
The base agent is introduced, and the RuleBasedAgent implemented. This mimics the traditional automation logic by using predefined rule sets. The simulations show how these agents perform tasks in a deterministic manner while maintaining reliability and speed. Visit the Full Codes here.
The class LLMAgent is a base agent.
def execute(self, task: Task) -> Dict[str, Any]:
time.sleep(random.uniform(0.2, 0.5))
accuracy_boost = 0.95 if task.complexity >= 4 else 0.90
result = {}
for key, value in task.expected_output.items():
If value is (int or float), you can use the following formula:
variation = value * (1 - accuracy_boost)
The result is:[key] = value + random.uniform(-variation, variation)
else:
The result is:[key] Include value
Return result
class HybridAgent(BaseAgent):
def execute(self, task: Task) -> Dict[str, Any]:
time.sleep(random.uniform(0.15, 0.35))
If the task.complexity
The LLMAgent represents reasoning-based AI, while the HybridAgent combines LLM adaptability with rule-based precision. These agents are designed to demonstrate how learning-based techniques improve accuracy of tasks, particularly for enterprise workflows. Take a look at the Full Codes here.
class BenchmarkEngine:
def __init__(self, task_suite: EnterpriseTaskSuite):
self.task_suite = task_suite
Self-results List[BenchmarkResult] = []
def run_benchmark(self, agent: BaseAgent, iterations: int = 3):
print(f"n{'='*60}")
print(f"Benchmarking Agent: {agent.name}")
print(f"{'='*60}")
for task in self.task_suite.tasks:
print(f"nTask: {task.name} (Complexity: {task.complexity}/5)")
for i in range(iterations):
result = self._execute_task(agent, task, i+1)
self.results.append(result)
Status "✓ PASS" If result.success then "✗ FAIL"
print(f" Run {i+1}: {status} | Time: {result.execution_time:.3f}s | Accuracy: {result.accuracy:.2%}")
We build our benchmarking engine here, and manage agent evaluation within the task set. We develop methods for running each agent more than once per task. The results are logged and key parameters, such as accuracy and execution time, can be measured. The benchmarking process is repeatable and systematic. Visit the Full Codes here.
def _execute_task(self, agent: BaseAgent, task: Task, run_num: int) -> BenchmarkResult:
start_time = time.time()
try:
Output = Agent.execute(task).
execution_time = time.time() - start_time
accuracy = self._calculate_accuracy(output, task.expected_output)
success = accuracy >= 0.85
return BenchmarkResult(task_id=task.id, agent_name=agent.name, success=success,
execution_time=execution_time, accuracy=accuracy)
Exception to the rule:
execution_time = time.time() - start_time
return BenchmarkResult(task_id=task.id, agent_name=agent.name, success=False,
execution_time=execution_time, accuracy=0.0, error_message=str(e))
def _calculate_accuracy(self, output: Dict, expected: Dict) -> float:
If output is not available:
Return 0.0
scores = []
For key, expect_val in the expected.items():
If key is not output
scores.append(0.0)
Continue reading
Actual_val = output[key]
if isinstance(expected_val, bool):
Scores.append (1.0 if expected_val >= actual_val, else 0.0).
elif isinstance(expected_val, (int, float)):
diff = abs (actual_val + expected_val).
tolerance = abs(expected_val * 0.1)
Score = max(0 - 1 - diff / tolerance + 1e-9)
scores.append(score)
else:
Scores.append (1.0 if expected_val >= actual_val, else 0.0).
return np.mean(scores) if scores else 0.0
We specify the logic for task execution and accuracy calculation. A scoring system is used to compare each agent’s outputs with expected results. Our benchmarking is fair and quantitative. This gives us insight into the degree to which agents are aligned with expectations. See the Full Codes here.
def generate_report(self):
DataFrame = DataFrame (pd)[asdict(r) for r in self.results])
print(f"n{'='*60}")
print("BENCHMARK REPORT")
print(f"{'='*60}n")
Agent_name is a df['agent_name'].unique():
Agent_df = "df"[df['agent_name'] == agent_name]
print(f"{agent_name}:")
print(f" Success Rate: {agent_df['success'].mean():.1%}")
print(f" Avg Execution Time: {agent_df['execution_time'].mean():.3f}s")
print(f" Avg Accuracy: {agent_df['accuracy'].mean():.2%}n")
return df
def visualize_results(self, df: pd.DataFrame):
Axes = plt.subplots (2, 2, figsize=(14.10, 10)).
fig.suptitle('Enterprise Agent Benchmarking Results', fontsize=16, fontweight="bold")
success_rate = df.groupby('agent_name')['success'].mean()
The axes[0, 0].bar(success_rate.index, success_rate.values, color=['#3498db', '#e74c3c', '#2ecc71'])
The axes[0, 0].set_title() ('Success Ratio by Agent'), fontweight="bold")
The axes[0, 0].set_ylabel('Success Rate')
The axes[0, 0].set_ylim(0, 1.1)
for i, v in enumerate(success_rate.values):
Axes[0, 0].text(i, v + 0.02, f'{v:.1%}', ha="center", fontweight="bold")
time_data = df.groupby('agent_name')['execution_time'].mean()
The axes[0, 1].bar(time_data.index, time_data.values, color=['#3498db', '#e74c3c', '#2ecc71'])
The axes[0, 1].set_title('Average Execution Time', fontweight="bold")
The axes[0, 1].set_ylabel('Time (seconds)')
for i, v in enumerate(time_data.values):
Axes[0, 1].text(i, v + 0.01, f'{v:.3f}s', ha="center", fontweight="bold")
df.boxplot(column='accuracy', by='agent_name', ax=axes[1, 0])
The axes[1, 0].set_title('Accuracy Distribution', fontweight="bold")
The axes[1, 0].set_xlabel('Agent')
The axes[1, 0].set_ylabel('Accuracy')
plt.sca(axes[1, 0])
plt.xticks(rotation=15)
task_complexity = {t.id: t.complexity for t in self.task_suite.tasks}
df['complexity'] The df['task_id'].map(task_complexity)
complexity_perf = df.groupby(['agent_name', 'complexity'])['accuracy'].mean().unstack()
complexity_perf.plot(kind='line', ax=axes[1, 1], marker="o", linewidth=2)
The axes[1, 1]Set_title ('Accuracy Based on Task Complexity.' Fontweight="bold")
The axes[1, 1].set_xlabel('Task Complexity')
The axes[1, 1].set_ylabel('Accuracy')
The axes[1, 1].legend(title="Agent", loc="best")
The axes[1, 1].grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
If __name__ is equal to "__main__":
print("Enterprise Software Benchmarking for Agentic Agents")
print("="*60)
task_suite = EnterpriseTaskSuite()
benchmark = BenchmarkEngine(task_suite)
Agents [RuleBasedAgent("Rule-Based Agent"), LLMAgent("LLM Agent"), HybridAgent("Hybrid Agent")]
For agent in agents
benchmark.run_benchmark(agent, iterations=3)
results_df = benchmark.generate_report()
benchmark.visualize_results(results_df)
results_df.to_csv('agent_benchmark_results.csv', index=False)
print("nResults exported to: agent_benchmark_results.csv")
Our reports are detailed and we create visual analyses for comparisons. We examine metrics, such as execution time and accuracy, across all agents, to compare performance. We export all the results into a CSV, which completes a comprehensive enterprise evaluation workflow.
As a conclusion, we developed a flexible, robust system of benchmarking that allows us to evaluate and compare multiple AI agent approaches in terms of their efficiency, accuracy and adaptability. We were able to observe how various architectures performed better for different task levels and visual analytics revealed performance trends. The process allows us to assess existing AI agents. It also provides the foundation needed for future enterprise AI agents that are optimized for intelligence and reliability.
Take a look at the Full Codes here. Please feel free to browse our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Don’t forget about our 100k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.
Asif Razzaq serves as the CEO at Marktechpost Media Inc. As an entrepreneur, Asif has a passion for harnessing Artificial Intelligence’s potential to benefit society. Marktechpost is his latest venture, a media platform that focuses on Artificial Intelligence. It is known for providing in-depth news coverage about machine learning, deep learning, and other topics. The content is technically accurate and easy to understand by an audience of all backgrounds. This platform has over 2,000,000 monthly views which shows its popularity.

