qwen3-4b-detect

qwen3-4b-detect

Predicts a continuous score in [0, 1] for how much AI contributed to a document, rather than a binary AI/human verdict. A QLoRA fine-tune of Qwen3-4B with an 11-bin soft ordinal head: targets are Gaussian-smoothed over the bins, trained with KL divergence, and decoded by expected value.

Contents

Pooling is last-token, not mean: under causal attention only the final position has seen the whole document.

Results

Trained on 50k bin-stratified examples of giga-editlens, then evaluated out-of-domain on three sets it never saw.

For reference, a DeBERTa-v3-base encoder trained identically scores 0.9324 / 0.3059 / 0.5889 / 0.6029 on the same four. This model ranks better on every set, at ~5.5x the inference cost (~10 vs ~55 samples/s).

Read Spearman, not thresholded accuracy. Spearman is threshold- and calibration-free. Fixed Human/Mixed/AI cut points are misleading here because the predicted-score distribution sits differently on each domain.

Known limits

• Out-of-domain scores are poorly calibrated even where ranking is good. Fit a calibration layer on held-out target-domain data before reading a raw score as a percentage.
• Light-touch assistance is underestimated. CoAuthor is the weakest case (0.39): its AI insertions average ~10-word spans, versus ~3-word edits in training.
• Unseen generators are under-detected. On BEEMO text from generators absent from training, genuinely AI-authored documents are frequently scored mid-range rather than high.
• Trained on English text of 50+ words. Shorter inputs are out of scope.

Provenance and licensing

Released for non-commercial research use (CC BY-NC 4.0). That reflects the training data's lineage rather than a preference:

• Trained on FineWeb, which is ODC-By 1.0 and requires attribution. Attribution is given here; whether trained weights constitute a "derivative database" under ODC-By is unsettled.
• Its ai_generated and ai_edited rows were produced by LLMs recorded in the dataset's model column. If any generator's terms of service restrict using its outputs to train competing models, that restriction flows through to these weights.

Usage

hf download Khushal2403/qwen3-4b-detect --local-dir ./qwen3-4b-detect
git clone https://github.com/darkengross/ai_detector && cd ai_detector/src
python eval/run_eval.py --arch decoder \
    --checkpoint-dir ../../qwen3-4b-detect --batch-size 32

Scoring a single document:

import torch, sys; sys.path.insert(0, "ai_detector/src")
from common.checkpoints import load_model
from common.binning import bin_centers
from common.loop import predict_scores

device = torch.device("cuda")
model, tok, k = load_model("decoder", "./qwen3-4b-detect", device)
scores, _ = predict_scores(model, [my_text], tok, bin_centers(k, device=device), device)
print(f"AI contribution: {scores[0]:.3f}")