coneml-348m-gamma

ConeML 348M Gamma

ConeML 348M Gamma is a 348M-parameter, scratch-trained small language model and the successor to ConeML 348M Beta. Gamma moves HumanEval from Beta's 0% to 13.4%, extends the context window from 512 to 4096 tokens, and — measured on the same harness alongside a local Qwen2.5-0.5B-Instruct reference — scores 2× on machine-verified execution, leads held-out multi-hop reasoning at every measured depth with per-suite margins of +34 to +74 points, and wins the epistemic-calibration battery, at 70% of the parameters.

Gamma is a research artifact. Every evaluation number in this card comes from a saved run record; the raw generations, batteries, and scorer code for both models ship in the eval/ folder of this repository.

Why ConeML Exists

ConeML is an independent research effort studying how much capability a compact model can reach through deliberate data and training design rather than scale — specifically, whether multi-step relational reasoning, calibrated abstention, and executable code synthesis are properties of what a model is trained on rather than of parameter count. Gamma is the current measurement of that question.

Parameter accounting: Gamma is 348M total parameters with a 32K vocabulary and tied embeddings — roughly 315M excluding token embeddings. Qwen2.5-0.5B is 494M total and roughly 360M non-embedding by Qwen's published figure. Gamma therefore runs at ~70% of the total parameters and ~88% of the non-embedding core of the model it is measured against below.

Training budget disclosure: Gamma's disclosed pretraining budget is approximately 6.9B tokens. Qwen2.5 reports an 18-trillion-token pretraining corpus at the series level; per-size splits are not published, but on any reading the results below are achieved with roughly three orders of magnitude less training data.

Results at a Glance

Same 418-item battery, same scorer code, deterministic decoding for both models. Qwen2.5-0.5B-Instruct was run locally (methodology and caveats below); it is a measured reference, not a leaderboard citation.

The pattern is consistent: Gamma leads wherever the scorer is unfakeable — code execution, exact answers, held-out multi-hop probes, abstention. The reference leads the breadth-and-fluency middle — general knowledge, everyday reasoning, systems/devops, polished prose — where a corpus three orders of magnitude larger pays off. Both results are reported in full below.

Public Benchmarks

Zero-shot, chat format, full 4096-token context, deterministic decoding with repetition penalty 1.15:

HumanEval 13.4% is measured synthesis, not a formatting artifact. The saved run records the 22 passing task IDs and all raw generations; passes include genuine algorithmic implementations, such as an iterative Euclidean GCD for HumanEval/13. Truncation/non-EOS stops account for only 8/164 (4.9%); of the 142 failures, 124 are runnable Python that fails the unit tests — wrong or incomplete algorithms, not degeneration. Qwen's published HumanEval result remains higher. What this measures is working code synthesis appearing at fixed 348M scale, from 0% to 13.4% in a single release, with a failure profile now dominated by algorithm choice rather than by producing code at all.

GSM8K remains weak. Gamma's result is on the full test set (Beta's was a 300-item sample, so the two are not perfectly comparable). The failure audit locates this precisely: multi-step word-problem composition. The model frequently sets up the first quantity correctly, then drops or mis-combines subsequent steps; arithmetic slips are secondary. It is not a blanket math failure — on the direct-computation slice below, Gamma scores 17/56 against the Qwen reference's 14/56.

Measured Comparison: Qwen2.5-0.5B-Instruct Reference

Qwen2.5-0.5B-Instruct was chosen as the reference because it is among the strongest openly available models near this size class — there are few credible yardsticks at sub-500M scale, and a comparison is only informative against a good one.

Reference methodology: the Qwen reference is qwen2.5:0.5b served through an Ollama /api/chat endpoint — one user message, no system prompt, temperature=0, repeat_penalty=1.15, num_predict=2048 — scored by the identical scorer code as Gamma, with all raw generations saved in eval/artifacts/qwen/. The Ollama version, model digest, and quantization level were not captured, so treat it as a careful local reference run rather than a reproducible public baseline, and read Qwen's own published benchmarks for its leaderboard standing. Nothing below claims Qwen cannot write code or reason; the claims are about this battery, this harness, both models measured the same way.

Machine-verified execution — Gamma 50/136, Qwen 25/136

These are the four categories where the scorer executes the generated code against unit tests or requires an exact numeric answer — categories where a fluent but wrong generation scores zero:

On the full-program and algorithms slices the reference's generations, visible in the saved artifacts, tend to stop at the function signature and docstring under this battery's "return only code" prompt surface — a genuine harness-sensitivity caveat, and one reason these slices should not be read as a general statement about Qwen's coding ability. The result that stands regardless: under identical prompts and identical scorers, Gamma produces implementations that pass the tests. A 2× margin on machine-verified output, held at 70% of the reference's parameters, is the headline result of this release.

The aggregate and the two large slices carry the claim: the 50-vs-25 total and the full-program (9/20 vs 0/20) and algorithms (11/20 vs 1/20) margins are each well beyond sampling noise on a two-proportion test. The math-computation (17/56 vs 14/56) and code-write (13/40 vs 10/40) margins are only three items each and are within noise — real in the aggregate, not individually decisive, and the card does not lean on them alone.

Epistemic calibration — Gamma 23/32, Qwen 17/32

Same 32-item battery, same scorer, both models:

Gamma takes direct abstention and contradiction detection 14/16 against the reference's 3/16; the reference is stronger on the framing-based tests (false premise, sycophancy). Abstention at 10/10 — a model that reliably says "I can't know that" rather than guessing — is the most direct demonstration of ConeML's design target, epistemic calibration, and it is not a capability that standard benchmarks measure.

Held-out multi-hop binding — the clearest separation

The probe generates fresh transitive chains ("A is taller than B. B is taller than C. …") from a held-out name pool never used in Gamma's training rows, then asks for the top or bottom of the ordering. Three suites vary the generalization axis: trained query wording, unseen query wording, and an unseen relation (older/younger). N=128 per depth per suite for Gamma; chance = 1/(depth+1), since the answer is one of the chain's names. Both models get identical chains, names, and queries.

Gamma, exact counts (chat surface, first-choice accuracy):

Depth-averaged comparison, with chance for context:

From depth 3 the reference degrades toward its chance floor, falling below it in several cells (10% at depth 3 on the unseen-relation suite, against 25% chance; 12% at depth 5 unseen-phrasing, against 17% chance). Gamma stays at least 37 points above chance in every cell of all three suites — including five-hop chains over names it never saw, in question wording it was never trained on. Per suite, Gamma's lead over the reference ranges from +34 to +74 points, at every depth, in all three suites. This is the capability ConeML is built around, and it is not one that standard leaderboards measure.

Full 418-item battery — Qwen 241, Gamma 219

Qwen wins the battery total, and the split is informative. All four machine-verified categories go to Gamma (above). The reference's lead comes from the breadth-and-fluency categories: everyday reasoning (19/20 vs 10/20), systems/devops (12/15 vs 4/15), SQL (12/15 vs 6/15), math explanation (14/20 vs 8/20), reasoning decomposition (28/30 vs 23/30), narrative, general explanation, instruction-following, and word problems (5/42 vs 1/42). Factual recall ties 24/30, and the in-distribution relational-binding slice is 27/30 vs 28/30 — the binding separation appears only when depth and held-out surfaces are probed, as above. Broad world knowledge and surface polish scale with corpus volume and instruct-tuning; they are outside what a 6.9B-token corpus targets, and the battery total reflects exactly that trade.

Evaluation Artifacts

The eval/ folder of this repository contains the full audit trail: the 418-item battery definition, the scorer and probe source code, and the raw saved runs behind every table on this card — Gamma's HumanEval, GSM8K, coverage, epistemic, and binding-probe records (with generations), plus all four Qwen reference artifacts. Checkpoint identifiers are scrubbed to the release name; the numbers are otherwise exactly as run.

Intended Format

Prompt the model with the role-marker chat format below. Raw completion without these markers is not the intended surface.

User:
<instruction>
Assistant:

The same format is included in chat_template.jinja and tokenizer_config.json, so tokenizer.apply_chat_template(...) works directly.

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "ConeML/coneml-348m-gamma"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = '''User:
Complete this Python function. Return only code.

def add_one(n):
    """Return n plus one."""
Assistant:
'''
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=128,
    do_sample=False,
    repetition_penalty=1.15,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Architecture

• Family: Llama-style decoder
• Parameters: approximately 348M total; ~315M non-embedding (tied embeddings)
• Layers: 30
• Hidden size: 1024
• Attention heads: 8
• KV heads: 2
• Vocab size: 32768
• Context length: 4096
• Tokenizer: custom 32K
• Text-only causal language model; use AutoModelForCausalLM or LlamaForCausalLM

Strengths

• Machine-verified execution at 2× the measured Qwen2.5-0.5B reference (50/136 vs 25/136) on identical prompts and scorers — code that passes tests and math that matches exactly.
• Deep held-out multi-hop binding: +34 to +74 points over the reference at every depth of all three suites, including unseen names, unseen query phrasing, and an unseen relation; a 62% average at five hops against the reference's 23% (chance floor 17%).
• Epistemic calibration: wins the battery 23/32 vs 17/32, with perfect 10/10 abstention against the instruct-tuned reference's 2/10.
• HumanEval 0% → 13.4% at fixed scale; failures are dominated by wrong algorithms, not degeneration or formatting.
• 4096-token context window, up from Beta's 512.
• Small enough to run locally on modest hardware; full eval audit trail published with the model.

Known Limitations

• Not a production assistant and not a replacement for larger general models.
• GSM8K is weak (2.7%): multi-step word-problem composition is not solved. Direct computation improved (17/56), but translating multi-quantity word problems into correct step sequences remains the top capability gap.
• HumanEval 13.4% is below Qwen's published number; most failures are wrong or incomplete algorithms.
• Qwen wins the broad battery (241 vs 219): general knowledge, everyday reasoning, systems/devops, SQL, polished prose, and instruction-following surfaces all favor the reference.
• Binding strength is name-based: on a non-name entity control suite (colored cards, in the shipped probe artifact), Gamma drops to 39–60% across depths — well above chance but far below its name-suite results.
• The Qwen comparison is asymmetric and is a local reference, not a reproducible public baseline: Gamma runs in fp32 through Transformers, while the reference runs through Ollama at an uncaptured quantization, digest, and server chat template. The internal battery is ConeML-designed, and the code slices are prompt-surface-sensitive. Qwen's leaderboard standing should be read from its own published results; the eval package documents every captured and uncaptured reference parameter.
• Training corpus, data mixture, and recipe are intentionally not disclosed (research IP): this card reports what the model does, not how it was built. Two facts bound the contamination question within that limit — the held-out binding probes are generated fresh at evaluation time, and the low public-benchmark scores (HumanEval 13.4%, GSM8K 2.7%) are inconsistent with benchmark-adjacent training.
• Cross-tokenizer perplexity comparisons are invalid across ConeML's 32K vocabulary and Qwen's much larger one; this card uses functional pass/fail evaluation throughout.
• Deterministic decoding works best with repetition penalty ≈1.15; bare greedy decoding understates quality.

License

Released for non-commercial use under CC BY-NC 4.0. Commercial use is not granted by this release.