We will develop a complete sentiment analysis workflow in this tutorial using the Stanford NLP IMDb Compare classical machine-learning with parameter-efficient transformator fine-tuning. We start by establishing a reproducible system and auditing datasets for class order, review length skew, leakage and preprocessing issues before constructing a solid TFIDF and Logistic regression baseline. Then, we fine-tune DistilBERT using LoRA via PEFT. We evaluate the model with accuracy, macro F1, ROC AUC, confusion matrixes and ROC curvatures, as well as reliability and Expected calibration error. We also investigate performance across different review lengths and word-level saliency. We use IMDb’s unlabeled split to perform confidence-based pseudolabeling. The resulting model is compared against the baseline and the transformed transformer saved for sentiment analysis.
Importlib.util import, warnings.
warnings.filterwarnings("ignore")
os.environ["TOKENIZERS_PARALLELISM"] = "false"
os.environ["WANDB_DISABLED"] = "true"
_REQUIRED = {
"transformers": "transformers",
"datasets": "datasets",
"peft": "peft",
"accelerate": "accelerate",
"sklearn": "scikit-learn",
}
_missing = [pkg for mod, pkg in _REQUIRED.items() if importlib.util.find_spec(mod) is None]
If _missing
print(f"Installing: {', '.join(_missing)} ...")
subprocess.run([sys.executable, "-m", "pip", "install", "-q", *_missing], check=True)
print("Done. (If imports fail below, restart the runtime and re-run.)n")
Numpy can be imported as np
import pandas as pd
Import torch
Matplotlib.pyplot can be imported as a plt
Load datasets from datasets
from sklearn.feature_extraction.text import TfidfVectorizer
From sklearn.linear_model, import LogisticRegression
From sklearn.pipeline, import make_pipeline
Import the metrics (accuracy_score, f1_score, roc_auc_score,
classification_report, confusion_matrix, roc_curve)
from transformers import (AutoTokenizer, AutoModelForSequenceClassification,
TrainingArguments, Trainer, DataCollatorWithPadding,
EarlyStoppingCallback, set_seed)
Get_peft_model from LoraConfig.
def _disable_torchao_probe():
Patched []
try:
Import peft.import_utils into _piu
_piu.is_torchao_available = lambda: False
patched.append("peft.import_utils")
The exception:
pass
for _name, _mod in list(sys.modules.items()):
If _name.startswith ("peft""" and _mod( "is_torchao_available"):
_mod.is_torchao_available = lambda: False
patched.append(_name)
Return Patches
try:
import torchao as _tao
_v = getattr(_tao, "__version__", "?")
If tuple (int(x), for x, in _v.split()".")[:2]Disabling the torchao probe of PEFT:
F"{', '.join(_disable_torchao_probe())}")
The exception:
_disable_torchao_probe()
SEED = 4
MODEL_NAME = "distilbert-base-uncased"
MAX_LEN = 256
N_TRAIN = 5000
N_EVAL = 2000
N_UNSUP = 3000
EPOCS = 2
BATCH = 16
The LR is 3e-4
FULL_RUN=False
If FULL_RUN
N_TRAIN = EPOCHS (25000), 25000, 3
set_seed(SEED); random.seed(SEED); np.random.seed(SEED); torch.manual_seed(SEED)
DEVICE = "cuda" if torch.cuda.is_available() The same applies to "cpu"
print("=" * 79)
print(f"device={DEVICE} | torch={torch.__version__} | "
F"gpu={torch.cuda.get_device_name(0) if DEVICE=='cuda' else 'n/a'}")
print("=" * 79)
t0 = time.time()
raw = load_dataset("stanfordnlp/imdb")
print(raw, f"nloaded in {time.time()-t0:.1f}sn")
print("--- example (truncated) ---")
print("label:", raw["train"][0]["label"], "|", raw["train"][0]["text"][:300], "...n")
first_labels = np.array(raw["train"]["label"][:5])
last_labels = np.array(raw["train"]["label"][-5:])
print(f"TRAP #1 - split ordering: first 5 labels {first_labels}, "
F"last 5 labels {last_labels} -> ALWAYS shuffle before subsampling.")
Train_full = Raw["train"].shuffle(seed=SEED)
Test_full => raw["test"].shuffle(seed=SEED)
train_ds = train_full.select(range(min(N_TRAIN, len(train_full))))
eval_ds = test_full.select(range(min(N_EVAL, len(test_full))))
print(f" after shuffle+subsample: train balance = "
F"{np.bincount(train_ds['label'])}, eval balance = {np.bincount(eval_ds['label'])}")
lens = np.array([len(t.split()) for t in train_full["text"]])
q = np.percentile(lens, [50, 75, 90, 95, 99])
print(f"nTRAP #2 - length (words): median={q[0]:.0f} p75={q[1]:.0f} p90={q[2]:.0f} "
F"p95={q[3]:.0f} p99={q[4]:.0f} max={lens.max()}")
print(f" ~{(lens > MAX_LEN*0.75).mean()*100:.1f}% of reviews exceed MAX_LEN={MAX_LEN} "
F"tokens (rough words->tokens factor 1.3). Section 9 measures what that costs.")
h_tr = {hashlib.md5(t.encode()).hexdigest() For t in raw["train"]["text"]}
h_te = {hashlib.md5(t.encode()).hexdigest() For t in raw["test"]["text"]}
print(f"nTRAP #3 - leakage: {len(h_tr & h_te)} exact duplicate reviews across "
F"train/test; {len(raw['train'])-len(h_tr)} dupes inside train itself.")
Define clean(t)
Return t.replace ("
", " ").replace("
", " ").strip()
plt.figure(figsize=(11, 3.2))
plt.subplot(1, 2, 1)
plt.hist(np.clip(lens, 0, 1000), bins=60)
plt.axvline(MAX_LEN, ls="--", color="k", label=f"MAX_LEN={MAX_LEN}")
plt.title("Review length (words, clipped at 1000)"); plt.legend()
plt.subplot(1, 2, 2)
plt.bar(["neg", "pos"], np.bincount(raw["train"]["label"]))
plt.title("Train class balance (perfectly balanced)")
plt.tight_layout(); plt.show()
We configure the Colab environment, install the required libraries, apply the PEFT–torchao compatibility fix, and set deterministic seeds for reproducible experiments. Loading the Stanford IMDb data, we shuffle, subsample, inspect the class balance, distribution of review-lengths, leakage and HTML artifacts. Before building models, we visualize the review lengths and frequency of labels.
print("n" + "=" * 79 + "n3. TF-IDF BASELINEn" + "=" * 79)
The Xtr is [clean(t) for t in train_ds["text"]Train_ds = array(np.array();["label"])
The Xte is [clean(t) for t in eval_ds["text"][]; yte=np.array()["label"])
t0 = time.time()
tfidf_clf = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=300_000,
sublinear_tf=True, strip_accents="unicode"),
LogisticRegression(C=8.0, max_iter=2000, n_jobs=-1),
)
tfidf_clf.fit(Xtr, ytr)
p_tfidf = tfidf_clf.predict_proba(Xte)[:, 1]
acc_tfidf = accuracy_score(yte, p_tfidf > 0.5)
auc_tfidf = roc_auc_score(yte, p_tfidf)
print(f"trained in {time.time()-t0:.1f}s -> acc={acc_tfidf:.4f} auc={auc_tfidf:.4f}")
vec, lr = tfidf_clf.steps[0][1], tfidf_clf.steps[1][1]
feats, coefs = np.array(vec.get_feature_names_out()), lr.coef_[0]
Order = order.argsort(coefs);
print("nmost NEGATIVE n-grams:", ", ".join(feats[order[:12]]))
print("most POSITIVE n-grams:", ", ".join(feats[order[-12:]][::-1]))
print("n" + "=" * 79 + "n4. LoRA FINE-TUNINGn" + "=" * 79)
tok = AutoTokenizer.from_pretrained(MODEL_NAME)
Def tokenize (batch).
The return tok (Return[clean(t) for t in batch["text"]], truncation=True, max_length=MAX_LEN)
tr_tok = (train_ds.map(tokenize, batched=True, remove_columns=["text"])
.rename_column("label", "labels"))
ev_tok = (eval_ds.map(tokenize, batched=True, remove_columns=["text"])
.rename_column("label", "labels"))
base = AutoModelForSequenceClassIf you want to know more about ification.from_pretrained(
MODEL_NAME, num_labels=2,
id2label={0: "NEGATIVE", 1: "POSITIVE"},
label2id={"NEGATIVE": 0, "POSITIVE": 1},
)
lora_cfg = LoraConfig(
task_type=TaskType.SEQ_CLS,
r=16, lora_alpha=32, lora_dropout=0.05,
target_modules=["q_lin", "v_lin"],
modules_to_save=["pre_classifier", "classifier"],
)
try:
model = get_peft_model(base, lora_cfg)
except ImportError as e:
_disable_torchao_probe()
print(f"[compat] retrying after backend probe failure: {e}")
model = get_peft_model(base, lora_cfg)
model.print_trainable_parameters()
def compute_metrics(eval_pred):
logits, labels = eval_pred
probs = torch.softmax(torch.tensor(logits), dim=-1).numpy()[:, 1]
preds = (probs > 0.5).astype(int)
return {"accuracy": accuracy_score(labels, preds),
"f1_macro": f1_score(labels, preds, average="macro"),
"roc_auc": roc_auc_score(labels, probs)}
_ta = inspect.signature(TrainingArguments.__init__).parameters
_eval_key = "eval_strategy" If you want to know more about if "eval_strategy" in _ta else "evaluation_strategy"
ta_kwargs = dict(
output_dir="./imdb_lora", learning_rate=LR,
per_device_train_batch_size=BATCH, per_device_eval_batch_size=BATCH * 2,
num_train_epochs=EPOCHS, weight_decay=0.01, warmup_ratio=0.06,
logging_steps=50, save_strategy="epoch", save_total_limit=1,
load_best_model_at_end=True, metric_for_best_model="accuracy",
fp16=(DEVICE == "cuda"), report_to="none", seed=SEED,
)
ta_kwargs[_eval_key] = "epoch"
_tr = inspect.signature(Trainer.__init__).parameters
_tok_key = "processing_class" if "processing_class" Other than _tr "tokenizer"
Trainer = trainer
model=model, args=TrainingArguments(**ta_kwargs),
train_dataset=tr_tok, eval_dataset=ev_tok,
data_collator=DataCollatorWithPadding(tok),
compute_metrics=compute_metrics,
callbacks=[EarlyStoppingCallback(early_stopping_patience=2)],
**{_tok_key: tok},
)
t0 = time.time()
trainer.train()
print(f"nfine-tuned in {(time.time()-t0)/60:.1f} min")
To establish a reference point that can be interpreted, we train a strong TFIDF baseline using Logistic Regression and examine the most important positive and negative ngrams. After tokenizing the IMDb ratings, we configure DistilBERT to use LoRA adapters which update only the most important model parameters. The Hugging Face Trainer is used with its dynamic padding and early stopping. It also has mixed precision. We can use multiple evaluation metrics.
print("n" + "=" * 79 + "n5. EVALUATIONn" + "=" * 79)
pred_out = trainer.predict(ev_tok)
p_lora = torch.softmax(torch.tensor(pred_out.predictions), dim=-1).numpy()[:, 1]
y_true = np.array(pred_out.label_ids)
yhat = (p_lora > 0.5).astype(int)
print(classification_report(y_true, yhat, target_names=["neg", "pos"], digits=4))
cm = confusion_matrix(y_true, yhat)
Figure, Ax = Plt.subplots (1, 2, figsize=(11, 4).
Ax[0].imshow(cm, cmap="Blues")
for i in range(2):
For j, in range(2)
ax[0]Text(j, I, Cm)[i, j], ha="center", va="center", fontsize=14)
Ax[0].set_xticks([0, 1], ["pred neg", "pred pos"])
Ax[0].set_yticks([0, 1], ["true neg", "true pos"]( ax[0].set_title("Confusion matrix")
Name, please p [("TF-IDF", p_tfidf), ("DistilBERT+LoRA", p_lora)]:
fpr, tpr, _ = roc_curve(y_true, p)
Ax[1].plot(fpr, tpr, label=f"{name} (AUC={roc_auc_score(y_true, p):.4f})")
Ax[1].plot([0, 1], [0, 1], "k--", lw=0.8)
Ax[1].set_xlabel("FPR"Ax[1].set_ylabel("TPR"( ; ax[1].set_title("ROC"( ; ax[1].legend()
plt.tight_layout(); plt.show()
print("n" + "=" * 79 + "n6. THRESHOLD & CALIBRATIONn" + "=" * 79)
ths = 0.95, 0.95, 91)
The accs is = [(y_true == (p_lora > t)).mean() for t in ths]
Best_t = Ths[int(np.argmax(accs))]
print(f"[email protected] = {accs[45]:.4f} | best threshold = {best_t:.2f} -> acc = {max(accs):.4f}")
def expected_calibration_error(probs, labels, n_bins=10):
"""ECE: |confidence - accuracy| averaged over confidence bins."""
Conf = max(probs, 1, - probs).
correct = (probs > 0.5).astype(int) == labels
Bins = np.linspace (0, 1, n_bins plus 1)
ece,xs,ys = 0. [], []
for lo, hi in zip(bins[:-1]Bins[1:]):
m = (conf > lo) & (conf The DistilBERT model is evaluated using classification metrics and ROCs, while comparing the ROC-AUC results with the TF/IDF baseline. The default probability threshold of 0.5 is compared to other classification thresholds in order to see which one gives best results on the evaluation set. Calculate the Expected calibration error and create a reliability chart to determine how close the predicted confidence of a model is to its true correctness.DataFrame = pd.Err(
print("n" + "=" * 79 + "n7. ERROR ANALYSISn" + "=" * 79)
Err = pd.DataFrame({
"text": eval_ds["text"], "y": y_true, "p_pos": p_lora,
"n_words": [len(t.split()) for t in eval_ds["text"]],
})
Err["pred"] = (Err.p_pos > 0.5).astype(int)
err["correct"] = err.pred == err.y
err["confidence"] = np.maximum(err.p_pos, 1 - err.p_pos)
print("--- 3 most CONFIDENT mistakes (where the model is confidently wrong) ---")
For _, r it err[~err.correct].nlargest(3, "confidence").itErrows():
print(f"n[true={'pos' if r.y else 'neg'} pred={'pos' if r.pred else 'neg'} "
f"conf={r.confidence:.3f} words={r.n_words}]")
print(clean(r.text)[:400].replace("n", " "), "...")
err["bucket"] = pd.qcut(err.n_words, 4, labels=["short", "med", "long", "v.long"])
by_len = err.groupby("bucket", observed=True).agg(acc=("correct", "mean"), n=("correct", "size"))
print("n--- accuracy by review length (truncation hurts long reviews) ---")
print(by_len.to_string())
print("n" + "=" * 79 + "n8. OCCLUSION SALIENCYn" + "=" * 79)
infer_model = model.merge_and_unload()
infer_model.to(DEVICE).eval()
@torch.no_grad()
def predict_proba(texts, bs=64):
Out = []
for i in range(0, len(texts), bs):
enc = tok([clean(t) for t in texts[i:i + bs]], truncation=True,
max_length=MAX_LEN, padding=True, return_tensors="pt").to(DEVICE)
out.append(torch.softmax(infer_model(**enc).logits, dim=-1)[:, 1].cpu().numpy())
return np.concatenate(out)
def occlusion(text, max_words=60):
words = clean(text).split()[:max_words]
base = predict_proba([" ".join(words)])[0]
[" ".join(words[:i] + words[i + 1:]) for i in range(len(words))]
dropped = predict_proba(variants)
return words, base - dropped, base
sample = err[err.correct].nlargest(1, "confidence").iloc[0]
words, contrib, base_p = occlusion(sample.text)
print(f"P(positive) for the full excerpt = {base_p:.3f} "
F"(true label = {'pos' if sample.y else 'neg'})n")
top = np.argsort(np.abs(contrib))[-15:]
plt.figure(figsize=(7, 5))
plt.barh(range(len(top)), contrib[top],
color=["tab:green" if contrib[i] > 0 else "tab:red" for i in top])
plt.yticks(range(len(top)), [words[i] for i in top])
plt.xlabel("Δ P(positive) when the word is removed")
plt.title("Occlusion saliency — green pushes POSITIVE, red pushes NEGATIVE")
plt.tight_layout(); plt.show()
print("n" + "=" * 79 + "n9. HEAD vs TAIL TRUNCATIONn" + "=" * 79)
Probe = err.nlargest (6, "n_words")
The W-180
head_txt = [" ".join(clean(t).split()[:W]"" for the text "text".
tail_txt = [" ".join(clean(t).split()[-W:]"" for the text "text".
Probe.y.values = yp
acc_head = ((predict_proba(head_txt) > 0.5).astype(int) == yp).mean()
acc_tail = ((predict_proba(tail_txt) > 0.5).astype(int) == yp).mean()
print(f"on the {len(probe)} longest reviews, using only {W} words:")
print(f" first {W} words -> acc {acc_head:.4f}")
print(f" last {W} words -> acc {acc_tail:.4f}")
print(" Practical takeaway: if the tail wins, feed head+tail to the model or "
"raise MAX_LEN, rather than blindly truncating from the left.")
The model’s confidently incorrect predictions are examined and grouped by review length in order to find truncation related failure patterns. The LoRA adapters are merged into the model’s underlying code and we apply a leave-one word-out occlusion in order to determine which words influence individual predictions towards positive or negative feelings. The strongest information about sentiment is found in the end and beginning portions of the long reviews.
print("n" + "=" * 79 + "n10. PSEUDO-LABELLINGn" + "=" * 79)
Unup = Raw["unsupervised"].shuffle(seed=SEED).select(range(N_UNSUP))
p_uns = predict_proba(unsup["text"])
keep = (p_uns > 0.95) | (p_uns 0.5).astype(int)
print(f"kept {keep.sum()}/{N_UNSUP} pseudo-labels at conf>0.95 "
The 'f"(balance: {np.bincount(pl_labels)})")
aug = make_pipeline(
TfidfVectorizer(ngram_range=(1, 2), min_df=2, max_features=300_000,
sublinear_tf=True, strip_accents="unicode"),
LogisticRegression(C=8.0, max_iter=2000, n_jobs=-1),
).fit(Xtr + pl_texts, np.concatenate([ytr, pl_labels]))
acc_aug = accuracy_score(yte, aug.predict(Xte))
print(f"TF-IDF baseline : {acc_tfidf:.4f}")
print(f"TF-IDF + pseudo-labels: {acc_aug:.4f} (Δ {acc_aug-acc_tfidf:+.4f})")
print("Caveat: gains are bounded by the teacher. Self-training also amplifies "
"the teacher's biases — always validate on clean, held-out data.")
print("n" + "=" * 79 + "n11. SAVE & INFERn" + "=" * 79)
SAVE_DIR = "./imdb-distilbert-lora-merged"
infer_model.save_pretrained(SAVE_DIR); tok.save_pretrained(SAVE_DIR)
print(f"saved merged model to {SAVE_DIR}/ (load with "
F"AutoModelForSequenceClassification.from_pretrained('{SAVE_DIR}'))")
demos = [
"A masterclass in tension. The final act left the whole theatre silent.",
"Two hours I will never get back. Wooden acting, incoherent plot.",
"It's not the disaster the trailer promised, but it never really lands either.",
]
for d, p in zip(demos, predict_proba(demos)):
print(f" P(pos)={p:.3f} -> {'POSITIVE' if p > 0.5 else 'NEGATIVE'} | {d}")
print("n" + "=" * 79)
print(f"SUMMARY (n_train={N_TRAIN}, n_eval={N_EVAL}, max_len={MAX_LEN})")
print("=" * 79)
print(pd.DataFrame([
{"model": "TF-IDF + LogReg", "accuracy": acc_tfidf, "roc_auc": auc_tfidf},
{"model": "TF-IDF + pseudo-labels", "accuracy": acc_aug, "roc_auc": float("nan")},
{"model": "DistilBERT + LoRA", "accuracy": accuracy_score(y_true, yhat),
"roc_auc": roc_auc_score(y_true, p_lora)},
]).to_string(index=False))
print("""
The next experiment is NEXT
- Set FULL_RUN to True for a real 25k/25k Benchmark (40 minutes on a T4).
Change MODEL_NAME from 'roberta' to 'roberta' base (target_modules=['query','value']" or
'answerdotai/ModernBERT-base' for an 8k context window — no truncation.
- The head+tail is truncated: the first 128 and last 128 tokens are based on section 9.
- Ablate LoRA rank r in {4, 8, 16, 64} and plot accuracy vs trainable params.
Replacing the teacher's pseudo-label with a group and self-training iterations.
- Push to the Hub: huggingface_hub.login() then infer_model.push_to_hub(...).
""")
The finely tuned transformer is used to create high-confidence, pseudo-labels from IMDb’s unlabeled examples and to add them to the TFIDF training corpus. To determine whether self-training with semi-supervised feedback improves the accuracy of prediction, we compare the enhanced classifier to the original baseline. Finaly, we combine the DistilBERT and tokenizer models, then run sentiment inferences on reviews. Finally, the model performance is summarized.
We developed an extensive sentiment classification pipeline, which goes beyond just fine-tuning the transformer or reporting accuracy. In order to evaluate the predictive quality of DistilBERT and its probability reliability, we established a competitive TFIDF baseline. We also trained DistilBERT with LoRA efficiently and evaluated both their probabilistic and statistical reliability. The occlusion saliency was used to interpret individual predictions. Sentiment information in long reviews is also tested for concentration near the beginning and end.
Click here to find out more FULL CODES here. Also, feel free to follow us on Twitter Don’t forget about our 150k+ML SubReddit Subscribe now our Newsletter. Wait! Are you using Telegram? now you can join us on telegram as well.
You can partner with us to promote your GitHub Repository OR Hugging Page OR New Product Launch OR Webinar, etc.? Connect with us
