Zero-Sentiment-0.2M

Zero-Sentiment-0.2M

Zero-Sentiment-0.2M is a 0.2M-parameter bidirectional sentiment classification model trained on 189k rows from four distinct datasets.

Architecture

Zero-Sentiment-0.2M uses a compact and parameter-efficient architecture, featuring GQA, RoPE, RMSNorm, Hadamard FFNs with SwiGLU intervals, and a small, custom 1024-vocab tokenizer.

• Hidden Size: 48
• Vocab Size: 1024
• Number of Layers: 12
• Number of Attention Heads: 4
• Number of KV Heads: 2
• Intermediate Size (for SwiGLU): 96
• SwiGLU Interval: 3 (every 3rd layer)
• RoPE Theta: 2500.0
• Max Position Embeddings: 96
• Tie Word Embeddings: true
• Use Engram: false

Training

Zero-Sentiment-0.2M was trained with the Muon-AdamW optimser and the WSD scheduler for 4 epochs.

Training datasets

• stanfordnlp/imdb
• nyu-mll/glue
• SetFit/sst5
• cardiffnlp/tweet_eval

Training Results

• Final Eval Acc: 72.49%
• Final Eval F1 Macro: 71.09%
• Final Eval F1 Weighted: 72.49%
• Final Eval Loss: 0.6587

Evaluation on fancyzhx/yelp_polarity

The model achieves an overall accuracy of 70.34% with only 0.2M parameters, while being exceptionally fast.

Generated Predictions

The model predicted correctly 9 out of 10 times on the sample test suite. Internal testing shows that it performs exceptionally well on clear positive and negative inputs, but can struggle with subtle, neutral, or ambiguous sentences where it tends to lean negative.

How to use

import torch
import torch.nn.functional as F
from transformers import AutoConfig, AutoModelForSequenceClassification, AutoTokenizer

MODEL_ID = "fromziro/Zero-Sentiment-0.2M"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(MODEL_ID, trust_remote_code=True)
config = AutoConfig.from_pretrained(MODEL_ID, trust_remote_code=True)
model = AutoModelForSequenceClassification.from_pretrained(MODEL_ID, config=config, trust_remote_code=True)
model.eval()

# 3-Class label mapping
LABEL_MAP = {0: "Negative", 1: "Neutral", 2: "Positive"}

text = "Replace with whatever you want"

# Tokenize input (max length 96 tokens)
inputs = tokenizer(text, max_length=96, padding="max_length", truncation=True, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    probs = F.softmax(outputs.logits, dim=-1)[0]
    pred_label = torch.argmax(probs).item()

print(f"Prediction: {LABEL_MAP[pred_label]}")
print(f"Confidence: {probs[pred_label] * 100:.1f}%")
print(f"Probabilities -> Neg: {probs[0]*100:.1f}% | Neu: {probs[1]*100:.1f}% | Pos: {probs[2]*100:.1f}%")

License

Apache 2.0.