llm-judge-bench-tr-en

LLM Judge Benchmark: Turkish-English Subjective Text Evaluation

Dataset Description

This dataset contains 23,780 LLM-as-a-Judge evaluations from a systematic benchmark comparing 16 large language models on subjective text evaluation tasks across 7 datasets (3 Turkish, 4 English).

Each record includes the full prompt sent to the LLM, the raw LLM response, the parsed score, and the human-annotated ground truth — enabling detailed analysis of both scoring accuracy and response quality.

The benchmark was conducted as part of a PhD thesis at Anadolu University, investigating the reliability of open-source LLMs as automated evaluators for subjective texts.

Key Features

• 16 LLMs tested: 15 open-source + 1 closed-source (GPT-4o-mini)
• 7 datasets: offensive detection, sentiment analysis, essay scoring, emotion intensity, response quality
• Bilingual: 3 Turkish + 4 English datasets
• Controlled variables: 2 prompt strategies × 3 temperatures × 2 scoring modes
• Full transparency: system prompt, user prompt, and raw LLM output included
• Two formats: JSONL (full data with prompts) + CSV (lightweight scores only)

Dataset Structure

Each row represents a single evaluation: one model scoring one text item under specific experimental conditions.

Files

• evaluations.jsonl (default): Full data including prompts and LLM responses (~56 MB)
• evaluations.csv: Lightweight scores-only version for quick numerical analysis (~2 MB)

Columns (JSONL — full)

Models

Datasets

N denotes the number of successful evaluations per dataset (API failures excluded). Each dataset was stratified-sampled to 11–20 text items (18–20 for most sets; 11 for UltraFeedback), then evaluated under 16 models × 2 prompt strategies × 3 temperatures × 2 scoring modes.

Key Findings

• Open-source models are competitive: Command-A (NMAE: 0.140) outperforms GPT-4o-mini (0.160) by 12%.
• Discrete scoring outperforms continuous by ~10% across all models and temperatures.
• Temperature has minimal effect: <5% NMAE difference between T=0.0 and T=1.0.
• Turkish performance is close to English: TR NMAE 0.188 vs EN NMAE 0.179.

Usage

from datasets import load_dataset

# Full dataset with prompts and LLM responses
ds = load_dataset("Guvenc/llm-judge-bench-tr-en")
df = ds["test"].to_pandas()

# Or lightweight scores-only CSV
ds_light = load_dataset("Guvenc/llm-judge-bench-tr-en", "scores_only")

# Overall NMAE by model
print(df.groupby("model")["nmae"].mean().sort_values())

# Inspect a specific evaluation
row = df.iloc[0]
print(f"Model: {row['model']}")
print(f"Prompt: {row['user_prompt'][:200]}...")
print(f"LLM said: {row['llm_response']}")
print(f"Score: {row['predicted_score']} (truth: {row['ground_truth']})")

# Turkish vs English
print(df.groupby("language")["nmae"].mean())

Citation

If you use this dataset, please cite:

@misc{usanmaz2026llmjudgebench,
  title={LLM Judge Benchmark: Turkish-English Subjective Text Evaluation Dataset},
  author={Usanmaz, Güvenç},
  year={2026},
  publisher={HuggingFace},
  url={https://huggingface.co/datasets/Guvenc/llm-judge-bench-tr-en}
}

License

This dataset is released under the Creative Commons Attribution 4.0 International License (CC BY 4.0).