edu-qurating-core-ed-gemma-3-4b

Edu-QuRating Core Educational Scorer (Gemma 3 4B)

https://arxiv.org/abs/2609.09425

This model is an Edu-QuRating sequence-classification scorer for educational text. It takes a text passage as input and outputs one scalar logit for each criterion in the core educational-quality rubric. It is intended for ranking, filtering, corpus curation, and reward-modeling experiments, not for text generation.

Scores

The model outputs six criterion-specific scores. Each score follows the corresponding pairwise judgement prompt: a higher score means the text is closer to the side that the judge would prefer for that criterion. Scores should be interpreted criterion by criterion, not as one general educational-quality label.

Training Procedure

This model was trained as a pairwise-preference distillation model, not as an absolute-rating regressor. The training data starts from a 500k-row FineWeb-Edu-Fortified sample and uses the core educational prompt set to construct 100k pairwise comparisons. The pairwise judgement excerpts are capped at 512 tokens.

Pairwise labels were generated with GPT-4.1-mini using token log-probabilities. For each criterion-specific prompt, the judge is asked to choose between two excerpts. The log-probabilities of the two answer labels are converted into soft preference probabilities. To reduce order effects, each pair is judged in both text orders, and the two directional probabilities are averaged into one soft preference label.

For a pair of texts (x_i, x_j), the scorer outputs a scalar score s_c(x) for each criterion c, and the model preference probability is computed as:

P_model(x_j preferred over x_i for criterion c) = sigmoid(s_c(x_j) - s_c(x_i))

The loss is binary cross-entropy between this model preference probability and the GPT-4.1-mini soft preference label. Missing labels and self-comparisons are masked out. Low-confidence labels are filtered with confidence_threshold = 0.5, excluding labels close to a tie from the loss.

Intended Use

This model is intended for educational-quality scoring and corpus curation. In the Edu-QuRating pipeline, the core educational scorer is applied to FineWeb-Edu-Fortified documents to produce document-level score columns. These scores can then be used for multi-dimensional filtering, for example selecting documents that score highly on factual accuracy, lesson engagement, and pedagogical structure.

The model can also be used as a reward component for generated educational responses, although the core corpus-filtering use case is document scoring.

For long documents, the project scoring pipeline tokenizes the full document, splits it into 512-token chunks, scores each chunk, and averages chunk scores back to document-level scores using token-count weights.

How to Use

Batch inference pipeline with edu-qurating package

Install package from github: https://github.com/AI-for-Education/edu-qurating.
Requires flash-attn>=2.8.3 to run the inference pipline below.

import json

from datasets import load_dataset, Dataset
from qurating.inference import ModelAnnotator, TokenizeAndChunk


model_name = "AI-for-Education/edu-qurating-core-ed-gemma-3-4b"

device_batch_size = 500
text_field = "text"

annotator = ModelAnnotator(
    model_name, labels=None, device_batch_size=device_batch_size
)
tokenizer = TokenizeAndChunk(model_name, text_field=text_field)

### Test with the the cosmopedia_v2 dataset.
### Load as streaming dataset because we only want to take a few rows
ds = load_dataset(
    "HuggingFaceTB/smollm-corpus", "cosmopedia-v2", split="train", streaming=True
)

# get the first 10 rows
ds_first10 = Dataset.from_list(ds.take(10).to_list())

#################
## The scoring process is split into 2 stages:
## 1. tokenize and chunk
## 2. annotate

# stage 1
tokenized = ds_first10.map(tokenizer, batched=True)

# stage 2
keep_columns = [text_field, "audience"]
remove_columns = [col for col in tokenized.column_names if col not in keep_columns]
scored = tokenized.map(
    annotator, batched=True, with_indices=True, remove_columns=remove_columns
)

# take a look at the items with scores
print(json.dumps(scored.to_list(), indent=2))

Call the model directly with pytorch (no auto chunk handling)

import torch
from transformers import AutoTokenizer, AutoModelForSequenceClassification

model_name = "AI-for-Education/edu-qurating-core-ed-gemma-3-4b"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

text = "A plant uses sunlight, water, and carbon dioxide to make sugar through photosynthesis."

inputs = tokenizer(
    text,
    return_tensors="pt",
    truncation=True,
    max_length=2048,
)
inputs = {key: value.to(next(model.parameters()).device) for key, value in inputs.items()}

with torch.no_grad():
    outputs = model(**inputs)
    scores = outputs.logits[0].float().cpu()

labels = [model.config.id2label[i] for i in range(model.config.num_labels)]

for label, score in zip(labels, scores):
    print(f"{label}: {score:.4f}")

The returned values are criterion-specific scalar logits. They are most useful for ranking, filtering, or comparing texts under the same criterion, rather than as calibrated absolute ratings.

Limitations

The scores are model-derived quality signals, not ground-truth educational labels. They should be interpreted as ranking, filtering, or reward signals rather than as definitive assessments of educational value. Performance may vary across domains, age levels, languages, and formats that differ from the training and validation data.