Close Menu
  • AI
  • Content Creation
  • Tech
  • Robotics
AI-trends.todayAI-trends.today
  • AI
  • Content Creation
  • Tech
  • Robotics
Trending
  • Tesla Rolls Out FSD V15 with Actuality View and Imaginative and prescient Upgrades – Unite.AI
  • Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI
  • OpenRouter Cuts Three Suppliers After AI Immediate Publicity Findings – Unite.AI
  • A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels
  • Collectively AI Discloses Immediate Leak After Coding Agent Uncovered Credential – Unite.AI
  • Dragos Finds Immediate Leaks and Stolen Keys in AI Provide Chains
  • Lean and Mathlib Communities Certify Claude-Assisted Riemann Proof – Unite.AI
  • AWS Routes HyperPod Inference Throughout Areas From One Endpoint – Unite.AI
AI-trends.todayAI-trends.today
Home»Tech»An Implementation of Totally Traced and Evaluated Native LLM Pipeline Utilizing Opik for Clear, Measurable, and Reproducible AI Workflows

An Implementation of Totally Traced and Evaluated Native LLM Pipeline Utilizing Opik for Clear, Measurable, and Reproducible AI Workflows

Tech By Gavin Wallace21/11/20256 Mins Read
Facebook Twitter LinkedIn Email
LifelongAgentBench: A Benchmark for Evaluating Continuous Learning in LLM-Based Agents
LifelongAgentBench: A Benchmark for Evaluating Continuous Learning in LLM-Based Agents
Share
Facebook Twitter LinkedIn Email

On this tutorial, we implement a whole workflow for constructing, tracing, and evaluating an LLM pipeline utilizing Opik. We construction the system step-by-step, starting with a light-weight mannequin, including prompt-based planning, making a dataset, and eventually working automated evaluations. As we transfer by means of every snippet, we see how Opik helps us observe each perform span, visualize the pipeline’s conduct, and measure output high quality with clear, reproducible metrics. By the top, we’ve a completely instrumented QA system that we will prolong, examine, and monitor with ease. Take a look at the FULL CODES here.

!pip set up -q opik transformers speed up torch


import torch
from transformers import pipeline
import textwrap


import opik
from opik import Opik, Immediate, observe
from opik.analysis import consider
from opik.analysis.metrics import Equals, LevenshteinRatio


machine = 0 if torch.cuda.is_available() else -1
print("Using device:", "cuda" if machine == 0 else "cpu")


opik.configure()
PROJECT_NAME = "opik-hf-tutorial"

We arrange the environment by putting in the required libraries and initializing Opik. We load the core modules, detect the machine, and configure our mission so that each hint flows into the proper workspace. We lay the inspiration for the remainder of the tutorial. Take a look at the FULL CODES here.

llm = pipeline(
   "text-generation",
   mannequin="distilgpt2",
   machine=machine,
)


def hf_generate(immediate: str, max_new_tokens: int = 80) -> str:
   consequence = llm(
       immediate,
       max_new_tokens=max_new_tokens,
       do_sample=True,
       temperature=0.3,
       pad_token_id=llm.tokenizer.eos_token_id,
   )[0]["generated_text"]
   return consequence[len(prompt):].strip()

We load a light-weight Hugging Face mannequin and create a small helper perform to generate textual content cleanly. We put together the LLM to function regionally with out exterior APIs. This offers us a dependable and reproducible technology layer for the remainder of the pipeline. Take a look at the FULL CODES here.

plan_prompt = Immediate(
   identify="hf_plan_prompt",
   immediate=textwrap.dedent("""
       You might be an assistant that creates a plan to reply a query
       utilizing ONLY the given context.


       Context:
       {{context}}


       Query:
       {{query}}


       Return precisely 3 bullet factors as a plan.
   """).strip(),
)


answer_prompt = Immediate(
   identify="hf_answer_prompt",
   immediate=textwrap.dedent("""
       You reply based mostly solely on the given context.


       Context:
       {{context}}


       Query:
       {{query}}


       Plan:
       {{plan}}


       Reply the query in 2–4 concise sentences.
   """).strip(),
)

We outline two structured prompts utilizing Opik’s Immediate class. We management the planning section and answering section by means of clear templates. This helps us preserve consistency and observe how structured prompting impacts mannequin conduct. Take a look at the FULL CODES here.

DOCS = {
   "overview": """
       Opik is an open-source platform for debugging, evaluating,
       and monitoring LLM and RAG purposes. It supplies tracing,
       datasets, experiments, and analysis metrics.
   """,
   "tracing": """
       Tracing in Opik logs nested spans, LLM calls, token utilization,
       suggestions scores, and metadata to examine advanced LLM pipelines.
   """,
   "evaluation": """
       Opik evaluations are outlined by datasets, analysis duties,
       scoring metrics, and experiments that mixture scores,
       serving to detect regressions or points.
   """,
}


@observe(project_name=PROJECT_NAME, sort="tool", identify="retrieve_context")
def retrieve_context(query: str) -> str:
   q = query.decrease()
   if "trace" in q or "span" in q:
       return DOCS["tracing"]
   if "metric" in q or "dataset" in q or "evaluate" in q:
       return DOCS["evaluation"]
   return DOCS["overview"]

We assemble a tiny doc retailer and a retrieval perform that Opik tracks as a software. We let the pipeline choose context based mostly on the person’s query. This enables us to simulate a minimal RAG-style workflow with no need an precise vector database. Take a look at the FULL CODES here.

@observe(project_name=PROJECT_NAME, sort="llm", identify="plan_answer")
def plan_answer(context: str, query: str) -> str:
   rendered = plan_prompt.format(context=context, query=query)
   return hf_generate(rendered, max_new_tokens=80)


@observe(project_name=PROJECT_NAME, sort="llm", identify="answer_from_plan")
def answer_from_plan(context: str, query: str, plan: str) -> str:
   rendered = answer_prompt.format(
       context=context,
       query=query,
       plan=plan,
   )
   return hf_generate(rendered, max_new_tokens=120)


@observe(project_name=PROJECT_NAME, sort="general", identify="qa_pipeline")
def qa_pipeline(query: str) -> str:
   context = retrieve_context(query)
   plan = plan_answer(context, query)
   reply = answer_from_plan(context, query, plan)
   return reply


print("Sample answer:n", qa_pipeline("What does Opik help developers do?"))

We deliver collectively planning, reasoning, and answering in a completely traced LLM pipeline. We seize every step with Opik’s decorators so we will analyze spans within the dashboard. By testing the pipeline, we verify that every one parts combine easily. Take a look at the FULL CODES here.

consumer = Opik()


dataset = consumer.get_or_create_dataset(
   identify="HF_Opik_QA_Dataset",
   description="Small QA dataset for HF + Opik tutorial",
)


dataset.insert([
   {
       "question": "What kind of platform is Opik?",
       "context": DOCS["overview"],
       "reference": "Opik is an open-source platform for debugging, evaluating and monitoring LLM and RAG applications.",
   },
   {
       "question": "What does tracing in Opik log?",
       "context": DOCS["tracing"],
       "reference": "Tracing logs nested spans, LLM calls, token usage, feedback scores, and metadata.",
   },
   {
       "question": "What are the components of an Opik evaluation?",
       "context": DOCS["evaluation"],
       "reference": "An Opik evaluation uses datasets, evaluation tasks, scoring metrics and experiments that aggregate scores.",
   },
])

We create and populate a dataset inside Opik that our analysis will use. We insert a number of query–reply pairs that cowl totally different features of Opik. This dataset will function the bottom reality for our QA analysis later. Take a look at the FULL CODES here.

equals_metric = Equals()
lev_metric = LevenshteinRatio()


def evaluation_task(merchandise: dict) -> dict:
   output = qa_pipeline(merchandise["question"])
   return {
       "output": output,
       "reference": merchandise["reference"],
   }

We outline the analysis job and choose two metrics—Equals and LevenshteinRatio—to measure mannequin high quality. We guarantee the duty produces outputs within the actual format required for scoring. This connects our pipeline to Opik’s analysis engine. Take a look at the FULL CODES here.

evaluation_result = consider(
   dataset=dataset,
   job=evaluation_task,
   scoring_metrics=[equals_metric, lev_metric],
   experiment_name="HF_Opik_QA_Experiment",
   project_name=PROJECT_NAME,
   task_threads=1,
)


print("nExperiment URL:", evaluation_result.experiment_url)

We run the analysis experiment utilizing Opik’s consider perform. We hold the execution sequential for stability in Colab. As soon as full, we obtain a hyperlink to view the experiment particulars contained in the Opik dashboard. Take a look at the FULL CODES here.

agg = evaluation_result.aggregate_evaluation_scores()


print("nAggregated scores:")
for metric_name, stats in agg.aggregated_scores.objects():
   print(metric_name, "=>", stats)

We mixture and print the analysis scores to grasp how nicely our pipeline performs. We examine the metric outcomes to see the place outputs align with references and the place enhancements are wanted. This closes the loop on our totally instrumented LLM workflow.

In conclusion, we arrange a small however totally useful LLM analysis ecosystem powered fully by Opik and an area mannequin. We observe how traces, prompts, datasets, and metrics come collectively to provide us clear visibility into the mannequin’s reasoning course of. As we finalize our analysis and evaluation the aggregated scores, we recognize how Opik lets us iterate shortly, experiment systematically, and validate enhancements in a structured and dependable manner.


Take a look at the FULL CODES here. Be at liberty to take a look at our GitHub Page for Tutorials, Codes and Notebooks. Additionally, be happy to observe us on Twitter and don’t overlook to hitch our 100k+ ML SubReddit and Subscribe to our Newsletter. Wait! are you on telegram? now you can join us on telegram as well.


Asif Razzaq is the CEO of Marktechpost Media Inc.. As a visionary entrepreneur and engineer, Asif is dedicated to harnessing the potential of Synthetic Intelligence for social good. His most up-to-date endeavor is the launch of an Synthetic Intelligence Media Platform, Marktechpost, which stands out for its in-depth protection of machine studying and deep studying information that’s each technically sound and simply comprehensible by a large viewers. The platform boasts of over 2 million month-to-month views, illustrating its reputation amongst audiences.

🙌 Follow MARKTECHPOST: Add us as a preferred source on Google.

AI ar work
Share. Facebook Twitter LinkedIn Email
Avatar
Gavin Wallace

Related Posts

A Group of Researchers from Columbia and CertiK Introduce TensorLean for Formal Verification of GPU Kernels

22/11/2026

OpenAI Ships GPT-Stay-1.1 for Voice Brokers, Cuts Idle Value to $0.02/Min

21/11/2026

Meet Pondering Machines Inkling 2 Mini: 26B Looped MoE Runs on a 32GB Mac

21/11/2026

Baseten Releases Bison-1 and Groundhog-1, Topping Choice Mannequin Index

21/11/2026
Top News

The Blue Wave Has Tech Bros Scrambling for Knowledge Facilities in House

Meta Claims Downloaded Porn at Center of AI Lawsuit Was for ‘Personal Use’

Goose, A New Gay-Dating App Appears To Be Psyop

Instagram is Hacked and DOGE’s whistle-blower sues El Musk.

I Asked ChatGPT What WIRED’s Reviewers Recommend—Its Answers Were All Wrong

Load More
AI-Trends.Today

Your daily source of AI news and trends. Stay up to date with everything AI and automation!

X (Twitter) Instagram
Top Insights

NVIDIA AI Has Released A Streaming Diarization Tool: This Real-Time Speaker Diarization Identifies Who Is Speaking in Meetings or Calls Instantly

21/08/2025

How to add a link to your Instagram Story (3 steps + examples)

15/12/2025
Latest News

Tesla Rolls Out FSD V15 with Actuality View and Imaginative and prescient Upgrades – Unite.AI

22/11/2026

Potential Cognition AI Spherical Would Elevate $2B-Plus at $75B Valuation – Unite.AI

22/11/2026
X (Twitter) Instagram
  • Privacy Policy
  • Contact Us
  • Terms and Conditions
© 2026 AI-Trends.Today

Type above and press Enter to search. Press Esc to cancel.