Eval Gap Finder

Eval Gap Finder

Automates the process of finding and closing the gap between Python and AILANG benchmark success rates. Identifies language limitations, prompt gaps, and missing stdlib functions.

Quick Start

Most common usage:

# User says: "Find eval gaps" or "Analyze benchmark failures"
# This skill will:
# 1. Run evals with dev models (gemini-3-flash, claude-haiku-4-5)
# 2. Compare Python vs AILANG success rates
# 3. Identify benchmarks where Python passes but AILANG fails
# 4. Analyze error patterns and categorize them
# 5. Check if gaps are documented in prompt
# 6. Test proposed examples and add to prompt
# 7. Create design docs for language limitations

When to Use This Skill

Invoke this skill when:

• User asks to "find eval gaps" or "close the Python-AILANG gap"
• User wants to analyze benchmark failures
• After running evals and seeing lower AILANG success
• User says "why is AILANG failing?" or "improve AILANG benchmarks"
• User wants to identify language limitations

Available Scripts

scripts/​run_gap_analysis.sh [eval_dir]

Run full gap analysis on eval results.

.claude/​skills/​eval-gap-finder/​scripts/​run_gap_analysis.sh eval_results/​v0.6.5

scripts/​identify_python_only.sh <eval_dir>

List benchmarks where Python passes but AILANG fails.

.claude/​skills/​eval-gap-finder/​scripts/​identify_python_only.sh eval_results/​v0.6.5

scripts/​categorize_errors.sh <eval_dir>

Categorize AILANG failures by error type.

.claude/​skills/​eval-gap-finder/​scripts/​categorize_errors.sh eval_results/​v0.6.5

scripts/​test_example.sh <code>

Test if an AILANG code example compiles and runs correctly.

.claude/​skills/​eval-gap-finder/​scripts/​test_example.sh /​tmp/​test.ail

Workflow

1. Run Evals with Dev Models

ailang eval-suite --models gemini-3-flash,claude-haiku-4-5 --output eval_results/​gap-analysis

Target: Run with cheap/​fast models first. If they succeed, larger models should too.

2. Generate Summary and Identify Gaps

ailang eval-summary eval_results/​gap-analysis
.claude/​skills/​eval-gap-finder/​scripts/​identify_python_only.sh eval_results/​gap-analysis

Key metrics:

• AILANG success rate (target: >70%)
• Python success rate (baseline)
• Gap (python_only benchmarks)

3. Analyze Error Patterns

For each Python-only pass, categorize the error:

4. Check Prompt Coverage

For each gap, check if the pattern is documented:

grep -n "pattern" prompts/​v0.6.5.md

If not documented, add:

• Working example to Quick Reference section
• Entry in "What AILANG Does NOT Have" table (if limitation)
• New section if pattern is complex

5. Test Examples Before Adding

CRITICAL: Always test examples before adding to prompt!

cat > /​tmp/​test.ail << 'EOF'
module benchmark/​solution
-- Your example code here
EOF
ailang run --caps IO --entry main /​tmp/​test.ail

If example fails, it reveals a language gap - create a design doc instead.

6. Create Design Docs for Language Gaps

If testing reveals a language limitation:

1. Create design doc: design_docs/​planned/​vX_Y_Z/​m-<feature>.md
2. Document:
   • Minimal reproduction
   • Error message
   • Workaround (for prompt)
   • Proposed fix
3. Add workaround to prompt with note

7. Track Improvement

After updates, re-run evals:

ailang eval-suite --models gemini-3-flash,claude-haiku-4-5 --output eval_results/​gap-analysis-v2

Compare:

• Success rate improvement
• Which benchmarks fixed
• Any regressions

Error Categories Reference

Resources

Gap Analysis Template

See resources/​gap_analysis_template.md for structured analysis format.

Common Patterns

See resources/​common_patterns.md for frequently encountered gaps.

Progressive Disclosure

This skill loads information progressively:

1. Always loaded: This SKILL.md file (workflow overview)
2. Execute as needed: Scripts in scripts/ directory
3. Load on demand: Resources for templates and patterns

Notes

• Always test examples before adding to prompt
• Prefer fixing language over prompt workarounds
• Track improvements with before/​after eval runs
• Create design docs for language limitations
• Update prompt hash in versions.json after changes