We build a Deep Reinforcement System that is advanced and agentic. It guides the agent in learning not just actions but how to pick its own strategies. Designing a Dueling Double DQN Learner. Introducing a curriculum that increases in difficulty. Integrating multiple exploration mode adapts to the training. The meta-agent we build plans, regulates and evaluates the whole learning process. It allows us to understand how agency turns reinforcement learning from a strategic, self-directed workflow into a self directed, autonomous one. Click here to see the FULL CODES here.
!pip install -q gymnasium[classic-control] The torch matplotlib
Import gymnasium as Gym
Numpy can be imported as np
Import torch as torch.nn.
Import deque defaultdict
Import matplotlib.pyplot, math random as Plt
random.seed(0); np.random.seed(0); torch.manual_seed(0)
class DuelingQNet(nn.Module):
def __init__(self, obs_dim, act_dim):
Supermarkets are a great way to buy goods and services.().__init__()
hidden = 128
self.feature = nn.Sequential(
nn.Linear(obs_dim, hidden),
nn.ReLU(),
)
self.value_head = nn.Sequential(
nn.Linear(hidden, hidden),
nn.ReLU(),
nn.Linear(hidden, 1),
)
self.adv_head = nn.Sequential(
nn.Linear(hidden, hidden),
nn.ReLU(),
nn.Linear(hidden, act_dim),
)
def forward(self, x):
Self.feature(x), h
v = self.value_head(h)
A = self.adv_head.h
return v + (a - a.mean(dim=1, keepdim=True))
class ReplayBuffer:
def __init__(self, capacity=100000):
self.buffer = deque(maxlen=capacity)
def push(self, s,a,r,ns,d):
self.buffer.append((s,a,r,ns,d))
def sample(self, batch_size):
batch = random.sample(self.buffer, batch_size)
s,a,r,ns,d = zip(*batch)
Define to_t (x,dt) and return the torch.tensor.
return to_t(s,torch.float32), to_t(a,torch.long), to_t(r,torch.float32), to_t(ns,torch.float32), to_t(d,torch.float32)
def __len__(self): return len(self.buffer)
Set up our core system of deep reinforcement learning. Initiate the environment. Create the Dueling Q-Network. Prepare the Replay Buffer to effectively store transitions. While we are establishing these foundations, our agent is equipped with everything it needs to learn. Click here to view the FULL CODES here.
Class DQNAgent
def __init__(self, obs_dim, act_dim, gamma=0.99, lr=1e-3, batch_size=64):
self.q = DuelingQNet(obs_dim, act_dim).to(device)
self.tgt = DuelingQNet(obs_dim, act_dim).to(device)
self.tgt.load_state_dict(self.q.state_dict())
self.buf = ReplayBuffer()
self.opt = optim.Adam(self.q.parameters(), lr=lr)
self.gamma = gamma
self.batch_size = batch_size
self.global_step = 0
def _eps_value(self, step, start=1.0, end=0.05, decay=8000):
return end + (start - end) * math.exp(-step/decay)
Select_action (self, mode, state, strategy, softmax_temp=1.0, def)
s = torch.tensor(state, dtype=torch.float32, device=device).unsqueeze(0)
No_grad. With torch():
q_vals = self.q(s).cpu().numpy()[0]
if mode == "eval":
return int(np.argmax(q_vals)), None
if strategy == "epsilon":
eps = self._eps_value(self.global_step)
If random()
The agent must be able to observe the surrounding environment and choose actions. It also needs a neural network that can update itself. Double DQN, gradient updates and exploration strategies are implemented to balance the learning and discovering of our agent. After we’ve finished this sample, our agent will have its low-level intelligence capabilities. Click here to view the FULL CODES here.
Class MetaAgent
def __init__(self, agent):
Self-agent = Agent
self.levels = {
"EASY": 100,
"MEDIUM": 300,
"HARD": 500,
}
self.plans = []
For diff. in self.levels.keys():
For mode ["train", "eval"]:
For expl. ["epsilon", "softmax"]:
self.plans.append((diff, mode, expl))
self.counts = defaultdict(int)
self.values = defaultdict(float)
self.t = 0
self.history = []
def _ucb_score(self, plan, c=2.0):
Self.counts[plan]
n == 0.
Return float ("inf")
Self-values[plan] + c * math.sqrt(math.log(self.t+1) / n)
def select_plan(self):
self.t += 1
scores = [self._ucb_score(p) for p in self.plans]
Return self-plans[int(np.argmax(scores))]
def make_env(self, diff):
max_steps = self.levels[diff]
return gym.make("CartPole-v1", max_episode_steps=max_steps)
def meta_reward_fn(self, diff, mode, avg_return):
r = avg_return
if diff == "MEDIUM"The following equation is equivalent to a 20-r
if diff == "HARD"R = 50
If mode == "eval" == "HARD"R = 50
Return r
def update_plan_value(self, plan, meta_reward):
self.counts[plan] += 1
n ='self.counts[plan]
m = values of oneself[plan]
self.values[plan] = (meta_reward-mu) + mu / n
def run(self, meta_rounds=30):
eval_log = {"EASY":[], "MEDIUM":[], "HARD":[]}
If k is in the range (meta_rounds+1), then:
diff. mode expl = self.select_plan()
Self.make_env()
avg_ret = self.agent.run_episodes(env, 5 if mode=="train" Otherwise 3, mode, if mode=="train" You can also find out more about "epsilon")
env.close()
if k % 3 == 0:
self.agent.update_target()
meta_r = self.meta_reward_fn(diff, mode, avg_ret)
self.update_plan_value((diff,mode,expl), meta_r)
self.history.append((k, diff, mode, expl, avg_ret, meta_r))
If mode == "eval":
eval_log[diff].append((k, avg_ret))
print(f"{k} {diff} {mode} {expl} {avg_ret:.1f} {meta_r:.1f}")
return eval_log
Design the agentic layers that determine how an agent will train. We then use a UCB to choose difficulty levels, modes and exploration style based on previous performance. The meta-agent is guiding our training by strategically selecting the options. See the FULL CODES here.
tmp_env = gym.make("CartPole-v1", max_episode_steps=100)
obs_dim, act_dim = tmp_env.observation_space.shape[0], tmp_env.action_space.n
tmp_env.close()
agent = DQNAgent(obs_dim, act_dim)
MetaAgent = Agent
eval_log = meta.run(meta_rounds=36)
final_scores = agent.evaluate_across_levels(meta.levels, episodes=10)
print("Final Evaluation")
For k, V in final_scores.items():
print(k, v)
The meta-rounds bring it all together. In these rounds, the DQN agent selects and executes plans that the meta-agent has selected. As the performance of our agent evolves, we can track its adaptation to more challenging tasks. This snippet shows the rise of self-directed long-term learning. See the FULL CODES here.
plt.figure(figsize=(9,4))
Colors are different. [("EASY","tab:blue"), ("MEDIUM","tab:orange"), ("HARD","tab:red")]:
if eval_log[diff]:
x, y = zip(*eval_log[diff])
plt.plot(x, y, marker="o", label=f"{diff}")
plt.xlabel("Meta-Round")
plt.ylabel("Avg Return")
plt.title("Agentic Meta-Control Evaluation")
plt.legend()
plt.grid(True, alpha=0.3)
plt.tight_layout()
plt.show()
In the graphs, we can see how an agent’s performance changes over time for Easy, Medium and Hard tasks. The curves allow us to observe trends in learning, progress, and agentic planning. We gain insights into the strategic choices that influence an agent’s progress as we examine these plots.
As a conclusion, we see our agent evolving into a multi-level system, one that refines policies, adapts exploration and chooses strategically how to train. We see the meta-agent improve its planning through UCB and lead the lower-level agent to more complex tasks. By understanding how agents can amplify the effects of reinforcement learning we are able to create systems which plan, adjust, and optimise their improvement over time.
Click here to find out more FULL CODES here. Please feel free to browse our GitHub Page for Tutorials, Codes and Notebooks. Also, feel free to follow us on Twitter Don’t forget about our 100k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.
Asif Razzaq, CEO of Marktechpost Media Inc. is a visionary engineer and entrepreneur who is dedicated to harnessing Artificial Intelligence’s potential for the social good. Marktechpost is his latest venture, a media platform that focuses on Artificial Intelligence. It is known for providing in-depth news coverage about machine learning, deep learning, and other topics. The content is technically accurate and easy to understand by an audience of all backgrounds. Over 2 million views per month are a testament to the platform’s popularity.

