gs-ai-detector

gs-ai-detector

Domain-specific AI text detector for the GS content network of garden e-commerce sites (Greenhouse Stores, Garden Ornaments, Garden UK). Identifies AI-generated marketing copy in headers, footers, blog posts and product descriptions so it can be flagged and rewritten.

Why this exists

Off-the-shelf AI detectors (Hello-SimpleAI/chatgpt-detector-roberta, GPTZero, Originality.ai) lag behind current model releases by months. For our specific use case - detecting Anthropic Claude-generated content in the garden e-commerce domain - a small, fine-tuned model gives:

• Sharper precision on actual content patterns we care about
• Predictable cost (no per-call API fees)
• Self-improving via the flywheel pipeline that captures every (AI, human-cleaned) fix as a new training pair

Use cases

Designed for: flagging AI-generated content on UK garden / greenhouse / outdoor-living e-commerce pages (collection headers and footers, product descriptions, blog articles).

Not for: general-purpose AI text detection across arbitrary domains. Performance outside garden e-commerce vocabulary is untested and likely degrades.

How to use

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

tok = AutoTokenizer.from_pretrained("clinteastman/gs-ai-detector")
model = AutoModelForSequenceClassification.from_pretrained("clinteastman/gs-ai-detector")
clf = pipeline("text-classification", model=model, tokenizer=tok,
               truncation=True, max_length=512)

text = "Our comprehensive collection of intricate stone garden ornaments..."
result = clf(text)
# [{'label': 'ai', 'score': 0.94}]

For long pages, chunk into ~300-word segments and average the AI probability across chunks.

The full ensemble (classifier + lexical pattern detector + admin UI) lives in the gs-ai-detector repo. The pattern detector catches em-dashes, banned vocab ("intricate", "comprehensive", "robust"), emoji headings, "focal point" overuse and similar lexical fingerprints that don't shift between model releases. The classifier corroborates with a distributional signal.

Training data

Two sources, combined into clinteastman/gs-ai-detector-training:

1. Real before/after pairs (851 samples) captured from manual and auto-cleaner rewrites of AI-flagged pages on Garden Ornaments. The AI side comes from the original page, the human side from the humanizer-rules cleanup.
2. Synthetic samples (1,616 samples) generated via the Anthropic Claude API (Sonnet 4.6). Same garden e-commerce topic in two distinct styles: default LLM output (label=1 AI) and humanizer- rules applied (label=0 human). 79 topics covering greenhouses, garden ornaments, plants, outdoor living, weddings and aspirational retail tone.

Each page is chunked into overlapping 200-word segments with 100-word stride, giving the model many sub-page samples per source page.

Total: 2,467 chunks (1,143 AI / 1,324 human after splitting).

Training procedure

• Base model: distilbert/distilroberta-base (~82M params)
• Hardware: NVIDIA RTX 4090
• Epochs: 4
• Batch size: 16
• Learning rate: 2e-5
• Weight decay: 0.01
• Warmup ratio: 0.1
• Mixed precision: fp16

Train/validation split is by source page (not by chunk) so chunks of the same page never leak across the split.

Total training time: ~30 seconds.

Evaluation

On the held-out validation set (223 samples from 15% of source pages):

The 100% precision is the most valuable property: when the model flags content as AI, it is reliably AI. The lower recall (58%) means we miss some AI content - this improves with more training data via the flywheel.

Limitations

• Domain narrow. Trained almost exclusively on garden e-commerce text. Don't expect strong performance on other content types.
• Conservative. Errs on the side of false negatives over false positives. Pair with the pattern detector for higher recall.
• Anthropic-family bias. Synthetic training data is Sonnet 4.6. Detection accuracy on other models (GPT-4, Gemini) is untested.
• English only.

Versions

• v1 (this version): baseline trained on 2,467 samples. F1=0.7345, Precision=1.00, Recall=0.58.

The flywheel automatically promotes new versions only when F1 beats the current production model by at least +0.005, so version numbers correspond to genuine improvements.

Citation / contact

Internal model for the GS content network. Maintained by Clinteastman.