openpangram-2b

OpenPangram-2B: Token-Level AI-Text Detection & Provenance

<p align="center"> <b>Replication of Pangram 4's Multi-Stage AI Detection Architecture on Qwen3.5-2B</b> </p> <p align="center"> <a href="https://huggingface.co/sandeshrajx/openpangram-2b"><img src="https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-Model-blue" alt="Hugging Face Model"></a> <a href="evals/stage2_evaluation_report.html"><img src="https://img.shields.io/badge/Evals-Interactive%20Reports-emerald" alt="Interactive Evals"></a> <a href="LICENSE"><img src="https://img.shields.io/badge/License-Apache%202.0-green" alt="License"></a> </p>

---

1. Overview & Key Capabilities

OpenPangram-2B is an open reproduction of the Pangram 4 AI-text detection architecture, trained on top of the Qwen3.5-2B hybrid linear-attention transformer backbone.

While conventional AI detectors produce only a single, blunt document-level binary score (often prone to false positives on human non-native writers), OpenPangram provides fine-grained, interpretable provenance:

1. Continuous Degree Calibration (0–100%): Predicts calibrated continuous AI involvement using an ordinal 15-bin regression head ($DOC\ \rho = 0.948$, Window $MAE = 6.7%$).
2. Token-Level Provenance Highlighting: Classifies every token into {Human, AI-Assisted, AI-Generated} using Repeat2 bidirectional causal context (83.23% token accuracy, 95.2% human recall, 90.6% AI recall).
3. Mixed-Authorship Window Detection: Identifies collaborative, human-AI co-authored passages (89.71% accuracy, 0.7097 binary F1).
4. Heuristic Span Smoothing: Employs contiguous span filtering to eliminate single-token argmax flicker without full CRF inference latency.

---

2. Training Methodology (Based on Pangram Technical Reports)

OpenPangram-2B was trained following the multi-stage training methodology formulated in the Pangram technical reports (Pangram 4 and Pangram 3 series), which replaces brittle perplexity heuristics with multi-task supervision:

[ Human & AI Reference Corpus (~60k Documents) ]
                    │
      Clause Alignment & Similarity
   (Lexical N-Gram Overlap + Embedding Cosine)
                    │
      Multi-Stage Curriculum Training
                    │
 ┌──────────────────┴──────────────────┐
 │                                     │
 ▼                                     ▼
Stage 1: Degree Calibration      Stage 2: Repeat2 Tokenwise
- Sequence-level supervision     - Duplicated sequence [x, x]
- 15-bin ordinal degree head     - Bidirectional causal context
- 4-bucket score head            - Multi-task token + mixed losses
- DOC rho = 0.948 achieved       - 972K validation tokens

Stage 1: Continuous Degree Calibration

• Supervision: Documents were supervised with continuous teacher degree targets derived from clause-level lexical and semantic edit distances against reference texts.
• Objective: Multi-task cross-entropy across a 15-bin ordinal degree head (segment_head) and a 4-ordered-bucket head (score_head).
• Results: Achieved a document-level Spearman correlation of DOC $\rho = 0.948$ against ground-truth AI degree on held-out validation sets.

Stage 2: Repeat2 Token-Level Provenance & Mixed Authorship

• Backbone Fusion: Stage 1 degree weights were permanently merged into the base backbone.
• Repeat2 Context Duplication: Each input window $x$ (up to 384 tokens) was duplicated into $[x, x]$ (length 768). Causal attention runs across the concatenated sequence, with supervision applied exclusively to the second copy. This grants every supervised token full bidirectional visibility over the entire passage within a causal model.
• Multi-Task Loss Balancing: $$\mathcal{L}{total} = 1.0 \times \mathcal{L}{token} + 0.5 \times \mathcal{L}{mixed} + 0.25 \times \mathcal{L}{segment} + 0.25 \times \mathcal{L}_{score}$$

Critical Engineering Fixes

• Head Persistence: Ensured all custom heads (score_head, segment_head, token_head, mixed_head) were registered under PEFT's modules_to_save, preventing head freezing during adapter training.
• Attention Mask Sanitization: Fixed unmasked mean pooling to prevent pad tokens from corrupting pooled representations.
• Native bfloat16 Dynamics: Bypassed PyTorch GradScaler to leverage bfloat16's native 8-bit dynamic exponent range.

---

3. Official Validation Benchmarks

Evaluated on the full held-out validation suite across 2,400 documents (3,711 windows, 972,065 tokens):

Benchmark Progression

Tokenwise Classification Breakdown (972,065 Tokens)

              precision    recall  f1-score   support

       Human      0.862     0.952     0.905    467,769
 AI-Assisted      0.470     0.126     0.198    119,710
AI-Generated      0.823     0.906     0.863    384,586

    accuracy                          0.832    972,065
   macro avg      0.718     0.661     0.655    972,065
weighted avg      0.798     0.832     0.801    972,065

---

4. Interactive Evaluation Dashboards

Interactive HTML evaluation reports and architectural walkthroughs are included directly within this repository under the evals/ directory:

• Stage 2 Full Benchmark Report (evals/stage2_evaluation_report.html): Complete interactive dashboard featuring benchmark progression, class confusion matrix, and a live toggle comparing raw argmax tokens with heuristic span smoothing.
• Training Approach Guide (evals/approach.html): Visual explanation of the multi-stage training pipeline, teacher signals, and validation gates.
• Label Distribution Audit (evals/stage1_full_label_audit.html): Audit of 60k training records, degree distributions, and clause-level alignment samples.
• Architectural Deep-Dive (evals/modelcard_with_diagrams.html): Visual diagrams illustrating Repeat2 sequence duplication, multi-head output routing, and decoding strategies.

---

5. How to Use

Installation

pip install torch transformers accelerate safetensors

Python Quickstart

You can load and query the model directly via transformers with trust_remote_code=True:

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "sandeshrajx/openpangram-2b"
device = "cuda" if torch.cuda.is_available() else "cpu"

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True, dtype=torch.bfloat16).to(device).eval()

text = "While artificial intelligence models can synthesize text rapidly, human prose carries distinct structural variance."

# Tokenize
inputs = tokenizer(text, return_tensors="pt").to(device)

with torch.no_grad():
    # Forward pass with Repeat2 context duplication
    out = model(inputs["input_ids"], inputs["attention_mask"], repeat2=True)
    
    # 1. Continuous Degree (0 to 100%)
    probs_seg = torch.softmax(out["logits_segment"][0], dim=-1)
    degree = float((probs_seg * torch.arange(15, device=device)).sum() / 14.0)
    
    # 2. Mixed Authorship
    probs_mixed = torch.softmax(out["logits_mixed"][0], dim=-1)
    is_mixed = bool(probs_mixed.argmax().item() == 1)
    
    # 3. Tokenwise Labels (0: Human, 1: AI-Assisted, 2: AI-Generated)
    token_classes = ["Human", "AI-Assisted", "AI-Generated"]
    token_preds = [token_classes[idx] for idx in out["logits_token"][0].argmax(dim=-1).tolist()]

print(f"Continuous AI Involvement: {degree * 100:.1f}%")
print(f"Mixed-Authorship Detected: {is_mixed} (p={probs_mixed[1].item():.3f})")
print(f"Analyzed {len(token_preds)} tokens.")

Command-Line Interface (CLI)

The included detect.py provides formatted terminal output and optional JSON export:

# Analyze text directly
python detect.py "The quick brown fox jumps over the lazy dog."

# Analyze a document with span smoothing (min length 3)
python detect.py --file paper.txt --smooth 3

# Output structured JSON
python detect.py --file essay.txt --json

Running the Interactive WebUI

An interactive Gradio WebUI with color-coded provenance heatmaps is provided in the repository:

python openpangram/app.py

Then navigate to http://127.0.0.1:7860 in your browser.

---

6. Model Architecture Specification

---

7. Limitations & Responsible Use

• Advisory Beta for High-Stakes Settings: While Document Degree correlation is high ($\rho = 0.948$), token-level AI-Assisted recall is currently 12.6%. The model is intended for triage, editorial assistance, and provenance exploration, not automated disciplinary sanctions (e.g. academic integrity violations) without human adjudication.
• Domain Shift & Non-Native English: Non-native English (ESL) text can exhibit statistical regularities that naive detectors penalize. We recommend calibrating operating thresholds to maintain $<0.1%$ false-positive rates on target domain baselines.
• Short Texts: Passages under 50 words contain limited statistical signal; scores on very short text should be interpreted cautiously.

---

Citation & Acknowledgments

Trained following the multi-stage detection formulation from the Pangram technical reports:

• Pangram 4 Technical Report (Pangram Labs, 2026).
• Repeat2: Efficient Bidirectional Attention via Prefix Repetition (Leviathan et al., 2025).