adele-judge-qwen3-14B-cre

ADeLe Distilled Judge

This repository contains an ADeLe-suite-specific distilled judge. It scores a model response against a question and reference answer with an ordinal score from 1 to 5, then derives binary correctness with the ADeLe threshold.

The repository root contains a merged Transformers model for standard loading. The original LoRA adapter is also included under adapter/ for provenance and reuse.

Intended Use

Use this model to score ADeLe-style examples where a question, reference answer, and model response are available. It is intended for out-of-model evaluation within the ADeLe benchmark suite, not as a general-purpose evaluator.

Input Format

The recommended helper accepts:

• question
• reference_answer or ground_truth
• model_response

Score Rubric

Allowed scores: 1, 2, 3, 4, 5

• 1: surely incorrect
• 2: likely incorrect
• 3: minimally correct or sufficient
• 4: likely correct
• 5: surely correct

Binary label: scores greater than or equal to 3 are CORRECT; lower scores are INCORRECT.

Training And Validation Data

• train models: DK-R1-Dist-Qwen-1.5B, DK-R1-Dist-Qwen-32B, DK-R1-Dist-Qwen-7B, gemini-2.5-flash, gemini-3.1-pro, gpt-35-turbo, gpt-5.2, gpt4o, llama3d1-405b, llama3d2-11b, llama3d2-1b, llama3d2-90b, llama4-17B-128E, o1-mini, o1_re=low, o3-mini
• validation models: DK-R1-Dist-Qwen-14B, gemini-3-flash, llama3d2-3b

Data Quality And Label Construction

Training labels are distilled from two proprietary judge scores used by the ADeLe evaluation pipeline to derive the official correctness signal. The configured source columns are score_gpt4o and score_sonnet.

• Ordinal target: floor(mean(score_gpt4o, score_sonnet)).
• Binary target: CORRECT when the ordinal target is >= 3.
• Judge-agreement filter: keep examples with abs(score_gpt4o - score_sonnet) <= 1.
• Response-length filter: keep responses with at most 4096 base-tokenizer tokens before prompt formatting.
• Sequence-length filter: keep full chat-formatted examples within max_seq_length=8192.

Validation Results

Source artifact: validation_trainer_metrics.json.

Recommended Inference

Do not use free-form generation as the primary prediction method. The recommended path scores the restricted continuations "1", "2", "3", "4", and "5".

from transformers import pipeline

judge = pipeline(
    "adele-judge",
    model="adgomant/adele-judge-qwen3-14-cre",
    trust_remote_code=True,
    device_map="auto",
)
result = judge(
    {"question": "...", "reference_answer": "...", "model_response": "..."}
)
print(result)

results = judge([
    {"question": "...", "reference_answer": "...", "model_response": "..."},
    {"question": "...", "ground_truth": "...", "model_response": "..."},
], batch_size=8)

The result has this shape:

{
    "score": 4,
    "label": "CORRECT",
    "probs": {"1": 0.01, "2": 0.02, "3": 0.08, "4": 0.70, "5": 0.19},
    "logprobs": {"1": -5.0, "2": -4.2, "3": -2.9, "4": -0.8, "5": -2.1},
    "confidence": 0.70,
    "margin": 1.3,
    "entropy": 0.82,
}

Standard Transformers Loading

from transformers import AutoModelForCausalLM, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("adgomant/adele-judge-qwen3-14-cre", trust_remote_code=True)

generation_config.json uses safe one-token defaults for debugging, but generate() is not the recommended scoring method.

Metadata

Training, filtering, split, tokenization, and metric artifacts available at packaging time are stored in adele_judge_metadata.json.

The model is trained on distilled judge targets. These targets are useful for reproducing the ADeLe paper-style correctness signal at lower inference cost, but they should not be interpreted as independent human annotations.

References

• ADeLe project page: ADeLe v1.0.
• ADeLe paper and official correctness definition: General scales unlock AI evaluation with explanatory and predictive power.
• Official ADeLe dataset: CFI-Kinds-of-Intelligence/ADeLe_battery_v1dot0.
• Official instance-level model-response data used for distillation: https://github.com/Kinds-of-Intelligence-CFI/ADeLe-AIEvaluation/tree/main/ADeLe_battery_data/subject_specific_instance_level_data.
• Training and Hub packaging implementation: https://github.com/adgomant/adele-judge.

Limitations

• ADeLe-specific judge; not a general-purpose evaluator.
• Distilled from proprietary judge labels and inherits their noise, calibration, and biases.
• Intended for scoring responses against a reference answer.
• It should not produce explanations; the expected output is a single score.
• Validation is out-of-model within the ADeLe suite, so transfer outside that suite should be measured before relying on it.