Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI
  • OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
  • DailyAI Briefing: A Lean-checked Riemann proof, plus 4 extra
AI-trends.todayAI-trends.today
Home»Tech»How does an AI agent choose what to do under token, latency and tool-call budget constraints?

How does an AI agent choose what to do under token, latency and tool-call budget constraints?

Tech By Gavin Wallace23/01/20268 Mins Read
Facebook Twitter LinkedIn Email
This AI Paper Introduces MMaDA: A Unified Multimodal Diffusion Model
This AI Paper Introduces MMaDA: A Unified Multimodal Diffusion Model
Share
Facebook Twitter LinkedIn Email

We build in this tutorial a cost aware planning agent which balances out the quality of output against actual constraints, like token usage, latencies, and budgets on tool-calls. The agent is designed to create multiple candidates actions, calculate their costs and benefits and select the execution plan which maximizes value within strict budget constraints. We demonstrate that agentic systems are capable of moving beyond “always use the LLM” It is important to avoid a reactive behavior by focusing on trade-offs and resource awareness. Click here to see the FULL CODES here.

Copy codeCopiesUse a different Browser
import os, time, math, json, random
Dataclasses can be imported to fields
Typing import List, Dict Optional, Tuple or Any
From getpass import Getpass


USE_OPENAI is True


If USE_OPENAI is true, then the following will be displayed:
 If not, os.getenv ("OPENAI_API_KEY"):
       os.environ["OPENAI_API_KEY"] = getpass("Enter OPENAI_API_KEY (hidden): ").strip()
   try:
 OpenAI import OpenAI
 OpenAI client()
 Except Exception As e.
       print("OpenAI SDK import failed. Falling back to offline mode.nError:", e)
 USE_OPENAI is False

We configure the environment for execution and load the OpenAI API keys securely at runtime, without hardcoding them. Also, we initialize the client to ensure that the agent will gracefully switch over to offline mode when the API becomes unavailable. Visit the FULL CODES here.

Copy codeCopiesUse a different Browser
def approx_tokens(text: str) -> int:
   return max(1, math.ceil(len(text) / 4))


@dataclass
A Budget for Class:
   max_tokens: int
   max_latency_ms: int
 Maximum number of tool calls: Int


@dataclass
Spending on class:
   tokens: int = 0
   latency_ms: int = 0
 Integer = 0 for tool_calls


   def within(self, b: Budget) -> bool:
       return (self.tokens <= b.max_tokens and
               self.latency_ms <= b.max_latency_ms and
               self.tool_calls <= b.max_tool_calls)


 Def add (self, other) "Spend") -> "Spend":
 Return Spending
           tokens=self.tokens + other.tokens,
           latency_ms=self.latency_ms + other.latency_ms,
           tool_calls=self.tool_calls + other.tool_calls
       )

We provide the budgeting abstractions to enable agents to think explicitly about costs. Modeling token usage, tool calls, latency and other quantities as first class quantities, we provide methods for accumulating and validating spend. We can enforce the constraints in planning and implementation. Click here to see the FULL CODES here.

Copy codeCopiesUse a different Browser
@dataclass
Step Option:
 name: str
   description: str
   est_spend: Spend
   est_value: float
   executor: str
 Payload: dict[str, Any] = field(default_factory=dict)


@dataclass
Class PlanCandidate
 Steps:[StepOption]
 Spend:
 Value: float
 Hash: str ""


def llm_text(prompt: str, *, model: str = "gpt-5"The effort is: "low") -> str:
 If not USE_OPENAI
 Return to the Homepage ""
   t0 = time.time()
   resp = client.responses.create(
       model=model,
       reasoning={"effort": effort},
       input=prompt,
   )
   _ = (time.time() - t0)
 If you want to return the output text, please click on this link. ""

The data structures represent both individual actions and complete plan options. Also, we define a lightweight LLM wrapping that standardizes the way text is measured and generated. This allows planners to think about action abstractly, without becoming too tied down to the execution details. Take a look at the FULL CODES here.

Copy codeCopiesUse a different Browser
def generate_step_options(task: str) -> List[StepOption]:
 Base = [
       StepOption(
           name="Clarify deliverables (local)",
           description="Extract deliverable checklist + acceptance criteria from the task.",
           est_spend=Spend(tokens=60, latency_ms=20, tool_calls=0),
           est_value=6.0,
           executor="local",
       ),
       StepOption(
           name="Outline plan (LLM)",
           description="Create a structured outline with sections, constraints, and assumptions.",
           est_spend=Spend(tokens=600, latency_ms=1200, tool_calls=1),
           est_value=10.0,
           executor="llm",
           payload={"prompt_kind":"outline"}
       ),
       StepOption(
           name="Outline plan (local)",
           description="Create a rough outline using templates (no LLM).",
           est_spend=Spend(tokens=120, latency_ms=40, tool_calls=0),
           est_value=5.5,
           executor="local",
       ),
       StepOption(
           name="Risk register (LLM)",
           description="Generate risks, mitigations, owners, and severity.",
           est_spend=Spend(tokens=700, latency_ms=1400, tool_calls=1),
           est_value=9.0,
           executor="llm",
           payload={"prompt_kind":"risks"}
       ),
       StepOption(
           name="Risk register (local)",
           description="Generate a standard risk register from a reusable template.",
           est_spend=Spend(tokens=160, latency_ms=60, tool_calls=0),
           est_value=5.0,
           executor="local",
       ),
       StepOption(
           name="Timeline (LLM)",
           description="Draft a realistic milestone timeline with dependencies.",
           est_spend=Spend(tokens=650, latency_ms=1300, tool_calls=1),
           est_value=8.5,
           executor="llm",
           payload={"prompt_kind":"timeline"}
       ),
       StepOption(
           name="Timeline (local)",
           description="Draft a simple timeline from a generic milestone template.",
           est_spend=Spend(tokens=150, latency_ms=60, tool_calls=0),
           est_value=4.8,
           executor="local",
       ),
       StepOption(
           name="Quality pass (LLM)",
           description="Rewrite for clarity, consistency, and formatting.",
           est_spend=Spend(tokens=900, latency_ms=1600, tool_calls=1),
           est_value=8.0,
           executor="llm",
           payload={"prompt_kind":"polish"}
       ),
       StepOption(
           name="Quality pass (local)",
           description="Light formatting + consistency checks without LLM.",
           est_spend=Spend(tokens=120, latency_ms=50, tool_calls=0),
           est_value=3.5,
           executor="local",
       ),
   ]


 Use Open if USE_OPENAI
       meta_prompt = f"""
You're a planning assistant. You are a planning assistant.
Like checks, validations, stakeholder tailored or tailoring. Shorten each step.


TASK:
{task}


List of JSON values with the following fields: Name, Description, Est_Value(1-10).
"""
       txt = llm_text(meta_prompt, model="gpt-5", effort="low")
       try:
 Items = json.loads (txt.strip())
 It is available in all items[:5]:
               base.append(
                   StepOption(
                       name=str(it.get("name","Extra step (local)"))[:60],
                       description=str(it.get("description",""))[:200],
                       est_spend=Spend(tokens=120, latency_ms=60, tool_calls=0),
                       est_value=float(it.get("est_value", 5.0)),
                       executor="local",
                   )
               )
 Except Exception
           pass


 Return base

We focus on generating a diverse set of candidate steps, including both LLM-based and local alternatives with different cost–quality trade-offs. We can optionally use our model itself to suggest low-cost additional improvements, while still controlling its impact on budget. We can enrich our action area without compromising efficiency. Take a look at the FULL CODES here.

Copy codeCopiesUse a different Browser
Def plan_under_budget
 Choose from the list[StepOption],
   budget: Budget,
   *,
 Maximum number of steps: 6
 The beam width is int 12
 Diversity_penalty float = 0.
) -> PlanCandidate:
   def redundancy_cost(chosen: List[StepOption], new: StepOption) -> float:
       key_new = new.name.split("(")[0].strip().lower()
 Overlap = 0.
 For s, choose:
           key_s = s.name.split("(")[0].strip().lower()
 If key_s >= key_new
 1 + overlap
       return overlap * diversity_penalty


 Beams[PlanCandidate] = [PlanCandidate(steps=[], spend=Spend(), value=0.0, rationale="")]


 For _ within range (max_steps),:
 Expanded List[PlanCandidate] = []
 Cand in beams
 For opt-in options:
 If you opt-in cand.steps
 Continue reading
               new_spend = cand.spend.add(opt.est_spend)
               if not new_spend.within(budget):
 Continue reading
               new_value = cand.value + opt.est_value - redundancy_cost(cand.steps, opt)
               expanded.append(
                   PlanCandidate(
                       steps=cand.steps + [opt],
                       spend=new_spend,
                       value=new_value,
                       rationale=cand.rationale
                   )
               )
 If not expanded
 Breaking News
       expanded.sort(key=lambda c: c.value, reverse=True)
 Beams = Expanded[:beam_width]


 Best = max(beams; key=lambda C: c.value);
   return best

Budget-constrained logic is implemented to find the best combination of actions within strict limitations. In order to prevent wasteful actions, we use a beam search style and redundancy penalty. The agent is now cost-aware, as they are optimizing the value of a task within constraints. Look at the FULL CODES here.

Copy codeCopiesUse a different Browser
Run_local_step (task: str; step: StepOption and working: Dict)[str, Any]) -> str:
 Name = lowercase name()
 If you want to know more about if "clarify deliverables" In name:
 Return (
           "Deliverables checklist:n"
           "- Executive summaryn- Scope & assumptionsn- Workplan + milestonesn"
           "- Risk register (risk, impact, likelihood, mitigation, owner)n"
           "- Next steps + data neededn"
       )
 If you want to know more about if "outline plan" In name:
 Return (
           "Outline:n1) Context & objectiven2) Scopen3) Approachn4) Timelinen5) Risksn6) Next stepsn"
       )
 If you want to know more about if "risk register" In name:
 Return (
           "Risk register (template):n"
           "1) Data access delays | High | Mitigation: agree data list + ownersn"
           "2) Stakeholder alignment | Med | Mitigation: weekly reviewn"
           "3) Tooling constraints | Med | Mitigation: phased rolloutn"
       )
 If you want to know more about if "timeline" In name:
 Return (
           "Timeline (template):n"
           "Week 1: discovery + requirementsnWeek 2: prototype + feedbackn"
           "Week 3: pilot + metricsnWeek 4: rollout + handovern"
       )
 If you want to know more about if "quality pass" In name:
       draft = working.get("draft", "")
 Return to the Homepage "Light quality pass done (headings normalized, bullets aligned).n" Draft
 The return of f"Completed: {step.name}n"


Run_llm_step (task: str; step: StepOption; working: Dict[str, Any]) -> str:
 Get the kind of step.value.get ("prompt_kind", "generic")
   context = working.get("draft", "")
   prompts = {
       "outline"The f"Create a crisp, structured outline for the task below.nTASK:n{task}nReturn a numbered outline.",
       "risks"The f"Create a risk register for the task below. Include: Risk | Impact | Likelihood | Mitigation | Owner.nTASK:n{task}",
       "timeline"The f"Create a realistic milestone timeline with dependencies for the task below.nTASK:n{task}",
       "polish"""Rewrite and polish the following draft for clarity and consistency.nDRAFT:n{context}",
       "generic"The f"Help with this step: {step.description}nTASK:n{task}nCURRENT:n{context}",
   }
   return llm_text(prompts.get(kind, prompts["generic"]), model="gpt-5", effort="low")


def execute_plan(task: str, plan: PlanCandidate) -> Tuple[str, Spend]:
   working = {"draft": ""}
 Real = Spend()


   for i, step in enumerate(plan.steps, 1):
       t0 = time.time()
       if step.executor == "llm" USE_OPENAI
 Run_llm_step (task, step and working)
           tool_calls = 1
       else:
 Out = run_local_step() (task, Step, Working)
           tool_calls = 0


       dt_ms = int((time.time() - t0) * 1000)
       tok = approx_tokens(out)


       actual = actual.add(Spend(tokens=tok, latency_ms=dt_ms, tool_calls=tool_calls))
       working["draft"] += = f"nn### Step {i}: {step.name}n{out}n"


   return working["draft"].strip()Current


Task = "Draft a 1-page project proposal for a logistics dashboard + fleet optimization pilot, including scope, timeline, and risks."
BUDGET = Budget(
   max_tokens=2200,
   max_latency_ms=3500,
   max_tool_calls=2
)


options = generate_step_options(TASK)
best_plan = plan_under_budget(options, BUDGET, max_steps=6, beam_width=14)


print("=== SELECTED PLAN (budget-aware) ===")
For s, see best_plan.steps
   print(f"- {s.name} | est_spend={s.est_spend} | est_value={s.est_value}")
print("nEstimated spend:", best_plan.spend)
print("Budget:", BUDGET)


print("n=== EXECUTING PLAN ===")
Draft = actual + execute_plan (TASK, best_plan).


print("n=== OUTPUT DRAFT ===n")
print(draft[:6000])


print("n=== ACTUAL SPEND (approx) ===")
print(actual)
print("nWithin budget?", actual.within(BUDGET))

Step by step, we execute and monitor the chosen plan. Dynamically, we choose the local or LLM execution path and combine it into one coherent document. Comparing estimated and actual expenditure, we show how planning assumptions can validated and fine-tuned in practice.

We concluded by showing how an agent that is aware of its cost can adapt in real-time to its consumption. The planning assumptions were validated by tracking actual expenditures. We only executed the steps within the budget that was pre-defined. This closed the loop between estimation, and execution. We also showed how AI agent systems could become more controllable and scalable if they treated cost, tool use, and latency as primary decision variables, rather than secondary ones.


Take a look at the FULL CODES here. Also, feel free to follow us on Twitter Join our Facebook group! 100k+ ML SubReddit Subscribe Now our Newsletter. Wait! Are you using Telegram? now you can join us on telegram as well.

Post How an AI Agent Chooses What to Do Under Tokens, Latency, and Tool-Call Budget Constraints? First on MarkTechPost.

AI
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026
Top News

Trump Administration is lifting its export controls on Anthropic’s Mythos & Fable AI Models

Join the tech reporters who are using AI to write, edit and create their stories

Dr. ChatGPT Sees You Right Now

Google Plans a 2029 Launch of Rack-Scale AI Satellites

Silicon Valley’s Next IPO billionaires Are Here. They are coming.

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

How Great Product Teams Decide What to Build Next – Unite.AI

08/08/2026

Taalas has replaced programmable graphics cards with AI chips that are hardwired to reach 17,000 tokens/second.

23/02/2026
Latest News

Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI

22/11/2026

OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.