coneml-348m-beta

ConeML 348M Beta

ConeML 348M Beta is the second public release in the ConeML research series — a 348M-parameter, scratch-trained small language model. It is the successor to ConeML 348M Alpha (polish900) and improves on it on the held-out reasoning, code, arithmetic, and calibration evaluations reported below. It is a research artifact and beta candidate, not a polished general assistant.

Why ConeML Exists

ConeML is an independent research effort exploring how much capability a compact language model can reach through deliberate data and curriculum design rather than scale alone. The clearest capability carried forward from Alpha is held-out transitive relation binding; Beta extends that capability while improving code generation and arithmetic in the same model.

Evaluations vs Alpha

All numbers are held-out probes (fresh entities disjoint from training), measured with the same protocol on both models.

Transitive inference, chat surface, first-choice accuracy, depths 1–5:

Other capabilities:

Standard public benchmarks (zero-shot, chat format) — reported for comparability, and modest as expected at this scale:

These two numbers measure full multi-step / algorithmic problem-solving, which is beyond a 348M model: GSM8K reflects the unsolved multi-digit arithmetic, and HumanEval requires complete algorithmic solutions (the 45% code figure above is held-out simple function-body completion — a different and easier task). They are published for transparency, not as strengths.

On these evaluations Beta improves over the Alpha on held-out reasoning, code execution, arithmetic, perplexity, and output formatting. On the older/younger relation suite it is higher at every depth; on unseen-query phrasing it is higher at most depths (the Alpha is slightly higher at depth 2). The Alpha's internal fixed-template probe saturated at 100% (depths 1–3); Beta's held-out template accuracy is 99 / 97 / 95 — effectively equal, on a harder probe.

Intended Format

Prompt the model in the chat format below, using the exact User: / Assistant: markers. Raw completion (without the markers) produces degraded output. The template also ships in chat_template.jinja / tokenizer_config.json, so tokenizer.apply_chat_template(...) works directly.

User:
<instruction>
Assistant:

Loading

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "ConeML/coneml-348m-beta"
tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(repo_id, torch_dtype=torch.float32, device_map="auto")

prompt = "User:\nMia is taller than Ben. Ben is taller than Zoe. Who is tallest? Return only the name.\nAssistant:\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=16, do_sample=False, eos_token_id=tokenizer.eos_token_id)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Architecture

• Family: Llama-style decoder · Parameters: ~348M · Layers: 30 · Hidden: 1024
• Attention heads: 8 · KV heads: 2 · Vocab: 32768 · Context length: 512
• Tokenizer: custom 32K

Strengths

• Scratch-trained 348M model that improves on its own alpha across the held-out evaluations reported here.
• Held-out transitive binding that generalizes across new names, new relations, and unseen query phrasing — higher than the alpha at every depth on the older/younger suite, and at most depths under unseen-query phrasing.
• Usable Python function-body generation with stable formatting (45% strict execution on the held-out evaluation reported here).
• Materially improved held-out arithmetic over the alpha.
• Well-calibrated on reasoning/code/agentic (ECE ≤ 0.04) — uncommon for models this size.

Known Limitations

• Multi-digit arithmetic is weak. Held-out 10-bucket arithmetic is 33% overall; reliable 3-digit and multiplication computation is not solved.
• Context length is 512 tokens; longer inputs are out of scope for this release.
• Transitive binding for non-name entities (e.g., objects) is near chance at depth — binding is still somewhat surface-shaped.
• All figures are research results from held-out probes and the standard benchmarks above — not production guarantees.
• Research release, not a replacement for larger general assistants.

License

Released for non-commercial use under CC BY-NC 4.0. Commercial use is not granted by this release.