verantyx-hle-5

⚠️ WARNING: This score includes statistical biases

• Position Prior: Letter-frequency bias (B>D>C>A based on HLE training data stats)
• Fallback Prior: Default answer B→D→C→A when no reasoning path found
• General Detectors: Hardcoded answers for specific known problems

True bias-free score: ~3.80% (95/2500)
Clean implementation: https://github.com/Ag3497120/verantyx

---

Verantyx V6 — Rule-Based Symbolic Reasoning System

HLE Score: 6.84% (171/2500) — No GPU, No API, No LLM

⚠️ Please read the Limitations and Validity section before citing this result.

---

Model Overview

---

What is Verantyx?

Verantyx is a purely rule-based, symbolic reasoning pipeline — no neural network, no language model, no API calls. Every inference is deterministic and explainable.

The system decomposes a question into an Intermediate Representation (IR), searches a hand-crafted knowledge piece database (107 pieces), executes domain-specific functions, and assembles an answer — all via classical algorithms.

Question (text)
    ↓ Decomposer (domain/task classification)
Intermediate Representation (IR)
    ↓ Beam Search (piece retrieval from 107-piece DB)
Execution Path
    ↓ Executor (24 domain executors)
Structured Candidate
    ↓ Grammar Composer + Answer Matcher
Final Answer (string)

---

Quick Start

import sys
sys.path.insert(0, ".")

from pipeline_enhanced import VerantyxV6Enhanced

pipeline = VerantyxV6Enhanced(piece_db_path="pieces/piece_db.jsonl")

result = pipeline.solve("What is the length of the string 'hello world'?")
print(result["answer"])  # → "11"

result = pipeline.solve("Solve for x: 2*x + 3 = 7")
print(result["answer"])  # → "2.0"

No model weights to download. No API key needed. Just Python.

---

HLE Benchmark Result

Score

Category Breakdown

Phase 5H Improvements

---

⚠️ Important Limitations and Validity Disclosure

1. Test Set Contamination

Verantyx V6 was developed by directly analyzing the HLE 2500 questions.

The development process involved:

• Viewing HLE 2500 question texts and analyzing domain/type distributions
• Designing executors, domain classifiers, and piece databases based on that analysis
• Iterating by evaluating on the same 2500 questions after each change

This constitutes test set overfitting in ML terms. Generalization to unseen data is not guaranteed.

For academic or official evaluation, a held-out test set that was never referenced during development is required. This result does not meet that standard.

2. Nature of Correct Answers

The 6.84% breaks down approximately as:

• Multiple choice questions (~480 questions): Phase 5H fixes the _score_specificity weight (0.3→0.05), eliminating E-selection bias. Still heuristic-based, ≈ random (~20% accuracy). Most correct answers here are coincidental.
• Arithmetic/algebra/string operations: Genuine computation. Executor actually calculated the answer.
• Number theory/combinatorics: Genuine formula execution.

A significant portion of the 171 correct answers come from random multiple-choice selection, not genuine understanding.

3. Context vs. Frontier LLMs

HLE baseline scores for reference (2025):

The comparison with LLMs is not fair. LLMs are evaluated on unseen data; Verantyx was developed against this specific test set.

---

Architecture

Components

Supported Domains (Executors)

arithmetic · algebra · calculus · linear_algebra · number_theory · combinatorics · advanced_combinatorics · probability · advanced_probability · statistics · geometry · graph_theory · logic · advanced_logic · modular_arithmetic · equation_solver · string_operations · multiple_choice · modal_logic · propositional_logic · knowledge

---

Design Philosophy

Verantyx is an experiment: "How far can rule-based symbolic reasoning reach on HLE without any LLM?"

LLM:      question → statistical pattern completion → answer
Verantyx: question → structural analysis → axiom/theorem search → symbolic computation → answer

The 6.84% figure matters less than the discovery of what fails and why: PhD-level math (algebraic topology, moduli spaces, functional analysis) is essentially impossible for rule-based systems, while deterministic computations (string operations, basic equations) succeed reliably.

---

Known Limitations

1. Advanced mathematics (PhD level): Algebraic topology, moduli spaces, functional analysis — limited executor coverage (Phase 5H Math: 5.3%)
2. Natural language understanding: Context-dependent reasoning and social science problems are fundamentally difficult
3. Chess/game problems: No engine integration (36 chess questions in HLE, ~0% accuracy)
4. Multiple-choice accuracy: Heuristic-based; Phase 5H corrected E-bias but still near-random (~20%)

---

Reproduce

git clone https://huggingface.co/kofdai/verantyx-hle-5
cd verantyx-hle-5
pip install -r requirements.txt
# Place hle_2500_eval.jsonl (obtain per HLE terms of use)
python quick_eval_hle.py

---

Citation

@misc{verantyx2026,
  author = {kofdai},
  title  = {Verantyx V6: A Rule-Based Symbolic Reasoning System for HLE},
  year   = {2026},
  url    = {https://huggingface.co/kofdai/verantyx-hle-5},
  note   = {HLE score: 6.84\% — test set contamination applies, see model card}
}

---

Development History

---

This model card prioritizes accuracy and transparency. Honest disclosure of score limitations contributes to the health of benchmarking research.