voight-kampff-pan2024-gte-en-v1.5

voight-kampff-pan2024-gte-en-v1.5

A fine-tuned sentence embedding model for AI-generated text detection, built for the Voight-Kampff Generative AI Shared Task (PAN @ CLEF 2024).

This model generates embeddings where human-written texts cluster together and are separated from AI-generated texts in the vector space. It is designed to be paired with a downstream classifier (we use a calibrated Linear SVM) for authorship verification.

Model Details

Training Data

Trained on the PAN 2024 competition dataset, augmented with texts rewritten by local LLMs via Ollama (llama 3.2 1b, qwen 2.5 1b, gemma 2 2b).

• Texts are chunked into ~64 token segments using the gte-base-en-v1.5 tokenizer
• Triplets formed as (human anchor, human positive, AI negative)
• 15% of training samples include leetspeak noise injection via pyleetspeak
• AI data from 18 different LLM sources (GPT-3.5, GPT-4, LLaMA, Mistral, Alpaca, Qwen, Gemma, etc.)

Usage

Generating Embeddings

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    'alejandroparbas/voight-kampff-pan2024-gte-en-v1.5',
    trust_remote_code=True
)

texts = [
    "A chunk of human-written text...",
    "A chunk of AI-generated text..."
]

embeddings = model.encode(texts)

Training a Classifier on the Embeddings

The embeddings can be used to train any classifier. We use a calibrated Linear SVM:

from sklearn.svm import LinearSVC
from sklearn.calibration import CalibratedClassifierCV

# Generate embeddings for your labeled data
train_embeddings = model.encode(train_texts, show_progress_bar=True)
test_embeddings = model.encode(test_texts, show_progress_bar=True)

# Train SVM
svm = LinearSVC(random_state=42, max_iter=10000)
svm.fit(train_embeddings, train_labels)  # labels: 0 = human, 1 = AI

# Calibrate for probability output
calibrated_svm = CalibratedClassifierCV(estimator=svm, cv='prefit')
calibrated_svm.fit(test_embeddings, test_labels)

Full Classification Pipeline

For the complete text-pair classification pipeline (chunking, embedding, SVM, scoring), see the GitHub repository.

A pre-trained SVM classifier is also available: alejandroparbas/voight-kampff-pan2024-classifier

Evaluation Results

When paired with a calibrated Linear SVM, the full pipeline achieves:

Chunk-Level Classification (~64 token chunks)

Full Text-Pair Classification (with chunk averaging)

On PAN 2024 test split (with noise):

On external Kaggle AI vs Human Text dataset:

Comparison with PAN 2024 Competition Leaderboard

Note: Our results are evaluated on our own test split and are not directly comparable to the official competition leaderboard.

Training Metrics

Limitations

• English only: The base model and training data are in English. Performance on other languages is not guaranteed.
• Short texts: Works best on texts long enough to produce multiple ~64 token chunks. With only 1-2 chunks, downstream classification accuracy is ~80%.
• Requires downstream classifier: This model produces embeddings, not classifications directly.
• Evaluation caveat: Results are on our own test split, not the official PAN 2024 evaluation set.

Authors

• Alejandro Pardo Bascuñana - Universidad Politécnica de Madrid
• Pedro Amaya Moreno - Universidad Politécnica de Madrid

Developed as part of the NLP course in the Master's program Aprendizaje Automático y Datos Masivos at UPM (2024-2025).

Citation

@misc{pardo2025voightkampff,
  title={Voight-Kampff: Contrastive Embedding Learning for AI-Generated Text Detection},
  author={Pardo-Bascu{\~n}ana, Alejandro and Amaya-Moreno, Pedro},
  year={2025},
  url={https://github.com/Alejandro-Pardo/voight-kampff-pan2024/}
}

Links

• GitHub: https://github.com/Alejandro-Pardo/voight-kampff-pan2024/
• Classifier Model: alejandroparbas/voight-kampff-pan2024-classifier
• Competition: PAN @ CLEF 2024 - Generative AI Authorship Verification
• Base Model: Alibaba-NLP/gte-base-en-v1.5