ag-news-classifier

ag-news-classifier

A DeBERTa-v3-base model fine-tuned to classify English news text into three topics: Sports, Business, and Sci/Tech.

This is a 3-class model, not the standard 4-class AG News setup. The World category was removed from both the training and evaluation splits, and the remaining labels were remapped to 0, 1, 2.

Labels

Usage

from transformers import pipeline

clf = pipeline("text-classification", model="Bubunur/ag-news-classifier")
clf("The stock market rallied after the central bank's decision")

Or directly:

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_id = "Bubunur/ag-news-classifier"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSequenceClassification.from_pretrained(model_id).eval()

labels = ["Sports", "Business", "Sci/Tech"]
text = "Researchers unveiled a chip that runs models on-device"

inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=128)
with torch.no_grad():
    probs = model(**inputs).logits.softmax(-1)[0]

print(labels[probs.argmax()], probs.max().item())

Training data

AG News, filtered to remove the World class.

Classes are perfectly balanced: 30,000 training and 1,900 test examples each. Each example is a news title concatenated with a short description. Texts are short — truncation at 128 tokens affects almost no examples.

Training procedure

Fine-tuned from microsoft/deberta-v3-base using the HuggingFace Trainer.

Per-epoch results:

The epoch-3 checkpoint was selected on accuracy. Note that validation loss rose sharply at epoch 3 while accuracy continued to climb — a typical sign that the model is becoming overconfident on examples it already gets right. The accuracy gain over epoch 2 (+0.11 pp) is small enough to fall within run-to-run noise.

Implementation note: microsoft/deberta-v3-base ships fp16 weights. Loading it without forcing dtype=torch.float32 raises ValueError: Attempting to unscale FP16 gradients when fp16=True is enabled.

Evaluation

Measured on the 5,700-example AG News test split (World removed).

Because the classes are perfectly balanced, accuracy and macro F1 coincide.

An earlier run of the same architecture under a slightly different configuration reached 0.9560 accuracy, with this per-class breakdown:

              Sports  Business  Sci/Tech
Sports          1888        6        6
Business          11     1752      137
Sci/Tech          12       79     1809

The overall shape is expected to hold for the current checkpoint, but these per-class figures come from the earlier run and have not been recomputed.

Domain and period. AG News was collected from news wires in the early 2000s. Performance on social media text, long-form articles, non-news prose, or non-English text is untested and expected to be worse. Terminology that emerged after the collection period was never seen during training. The model has not been evaluated on any corpus other than AG News, so its ability to generalize to other news sources is unknown.

Inherited bias. Any bias present in microsoft/deberta-v3-base or in the AG News corpus carries over. No bias audit was performed.

Citation

@misc{zhang2015character,
  title={Character-level Convolutional Networks for Text Classification},
  author={Xiang Zhang and Junbo Zhao and Yann LeCun},
  year={2015},
  eprint={1509.01626},
  archivePrefix={arXiv}
}