Downloads · 30 days
15
3% of all-time downloads
center-of-excellence/extract-prompt-quality-criteria
extract-prompt-quality-criteria is a machine learning model from center-of-excellence. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for sentence-transformers. The card lists the license as mit.
Automatic prompt quality scoring for IBM Bob, IBM's AI coding assistant. Scores user prompts on a 1–10 scale validated against manually annotated IBM Bob prompts and confirmed on additional unseen prompts.
Downloads · 30 days
15
3% of all-time downloads
All-time downloads
565
Public
Repo size
295 MB
Likes
1
Public
Click a slice to open those files.
.safetensors273 MB · 99%
From the Hugging Face model README
Automatic prompt quality scoring for IBM Bob, IBM's AI coding assistant. Scores user prompts on a 1–10 scale validated against manually annotated IBM Bob prompts and confirmed on additional unseen prompts.
Primary scorer (v3): agentlans/bge-small-en-v1.5-prompt-quality + regex
(90/10 blend). MIT license confirmed.
Evaluated on unseen IBM Bob prompts (honest zero-shot generalization):
| Model | Spearman r | MAE | Within 2pt % | ms/prompt |
|---|---|---|---|---|
| agentlans+regex (recommended) | 0.753 | 1.92 | 53.9% | ~12.8ms |
| agentlans-only | 0.738 | 1.95 | 55.0% | 12.7ms |
| Regex+SetFit (v2) | 0.679 | 1.76 | 69.6% | 35.1ms |
| Regex-only | 0.649 | 2.17 | 63.4% | 0.264ms |
| TinyLlama v1 (original) | 0.346 | 2.16 | 62.5% | 1,680ms |
The agentlans+regex blend improvement over agentlans-only is statistically significant (p=0.003, Bonferroni-corrected permutation test, n=191).
from src.agentlans_scorer import AgentlansScorer
scorer = AgentlansScorer(blend_regex=True) # 90% agentlans + 10% regex
result = scorer.score("Fix the null pointer in login.py.")
print(result['score']) # float 1.0–10.0
print(result['blend_ratio']) # '90% agentlans + 10% regex'
print(result['agentlans_raw']) # raw model output before calibration
from src.regex_detector import score
result = score("Fix the null pointer in login.py.")
print(result['score']) # float 1.0–10.0
print(result['explanation']) # human-readable breakdown
print(result['criteria']) # {'clear_task': True, 'examples_provided': False, ...}
from data_bob.prompt_quality_transformer import PromptQualityTransformer
import pandas as pd
df = pd.DataFrame({'text': ['Fix the bug.', '## Task\nImplement retry...']})
transformer = PromptQualityTransformer(prompt_column='text')
df_scored = transformer.fit_transform(df)
print(df_scored[['text', 'prompt_score', 'prompt_criteria', 'prompt_explanation']])
Six binary quality criteria:
| Criterion | Description |
|---|---|
clear_task | Explicit action verb and clear goal |
examples_provided | File paths, code refs, before/after, sample data |
structure_and_clarity | Headers, numbered lists, markdown sections |
output_format_specified | Specifies JSON, code, list, specific file, language |
constraints_defined | Technical or business constraints, versions, limits |
edge_cases_handled | Error handling, null checks, fallback behavior |
Score formula: score = 1 + (criteria_met / 6) × 9
Possible values: 1.0, 2.5, 4.0, 5.5, 7.0, 8.5, 10.0
├── src/
│ ├── agentlans_scorer.py # agentlans+regex blend scorer (v3, recommended)
│ ├── regex_detector.py # standalone regex scorer, zero dependencies
│ ├── hybrid_scorer.py # Regex+SetFit (v2, kept for reference)
│ └── evaluate.py # metrics, permutation test, comparison table
│
├── setfit_models/ # v2 SetFit classifiers (kept for reference)
│ ├── output_format_specified/
│ ├── constraints_defined/
│ └── edge_cases_handled/
│
└── prompt_quality_transformer.py # sklearn transformer (v3)
agentlans/bge-small-en-v1.5-prompt-quality — MIT license confirmed.
Safe for IBM commercial use.BAAI/bge-small-en-v1.5 (base model) — MIT license.@misc{extract-prompt-quality-criteria,
author = {Gregoire Cattan and Elif Uzun},
title = {Extract Prompt Quality Criteria},
year = {2026},
publisher = {Hugging Face},
url = {https://huggingface.co/center-of-excellence/extract-prompt-quality-criteria}
}