cobol-jeeves-sml

🧅 cobol-jeeves-sml — a 14M from-scratch COBOL SML (research artifact)

⚠️ PROTOTYPE / research artifact — read the Honesty section before citing any number. This is a method demonstration, not a working COBOL author and not a generalization result. It was adversarially reviewed (Claude, Codex, GLM) and the framing below is the corrected, honest version after that review.

A 14-million-parameter, from-scratch GnuCOBOL model (MLX MicroBrain, ~55 MB) in the SML — Smallest Language Model family (TinkyBrain lineage). No internet data; word/symbol COBOL tokenizer where the vocab is the domain boundary.

What this actually shows

The model cannot write compiling COBOL on its own (0% single-shot). The experiment is about the calling strategy around a tiny component:

The 33.8% is pass@16 with a compiler oracle and symbolic repair — NOT a compile-rate in the usual (pass@1) sense. It is the rate at which the search procedure finds a compilable program among up to 16 candidates.

Honesty (this is the important part)

The headline number is not evidence the model generalizes, and not comparable to a larger model:

1. It's pass@16, not compile-rate. 8 branches × (raw + repaired) = up to 16 compiler-checked candidates; first that builds wins. Single-output honest numbers are 0% (greedy) / 6.2% (greedy+repair).
2. The eval set is contaminated. Of 65 val prompts, ~61 are ≥0.95 near-duplicates of training prompts. Success split (audited): 21/61 on near-duplicates, 1/4 on novel prompts — the novel set (n=4) is too small to mean anything. So the 33.8% is consistent with memorization + search scaffold, not generalization.
3. Compilation ≠ correctness. Exact-match to the reference program was 0%. The verifier proves the compiler accepts the program, not that it does what the prompt asked.
4. No fair large-model comparison exists here. A 7B finetune scored 28.8% single-shot; running it under this same 16-candidate + repair pipeline would almost certainly exceed 33.8%. This model does not "beat" a 7B — that earlier claim was withdrawn after review.

What it can legitimately claim: a branch + symbolic-repair + compiler-verify call raises an ~0% tiny model to 33.8% pass@16 on an in-distribution set. Whether 14M × 16 verified calls is a compute-competitive path vs one large-model call is an interesting open question this artifact does not settle.

To make any of the numbers mean more, the next step is a template-disjoint, deduplicated held-out set and a matched-protocol comparison (same branches + repair for every model).

Architecture & files

MicroBrain (MLX decoder-only): d_model 512 · 8 heads · 6 layers · d_ff 1024 · max_seq 512 · vocab 1431, ~14.3M params, greedy decode. Trained from scratch on 1,236 prompt→COBOL pairs (free-format, cobc -free -c as ground truth).

• model.safetensors, config.json, tokenizer.json — the component
• sml_call.py — the branch/repair/verify call (the studied variable)
• cobol_repair.py — mechanical repair (never invents logic)
• cobol_sml_mcp.py — exposes it as an MCP tool (cobol_draft); honest compiles flag

Requires MLX (Apple Silicon) and GnuCOBOL 3.x (cobc) for the verify step.

License

Apache-2.0. From-scratch weights; deterministic training data. SML / TinkyBrain family.

Paper

📄 Verified Program Synthesis with a Symbolic Knowledge Graph and an Un-gameable Compiler Oracle — included here as PAPER.pdf, and published at https://perslis.com/cobol-paper.html. Reports all rates with Wilson 95% confidence intervals, a formal soundness lemma for the verification gate, measured cost, and prominent limitations. Preprint / working draft.