Skylar-980M-Cobol-Base

Skylar-980M-Cobol-Base

🔬 Published as a transparent research for the Skylar framework, not as a general-purpose or SOTA model.

A 980M decoder-only language model trained 100% from scratch (random init, no warm-start, no external pretrained weights) on a broad multi-language code corpus, as the pretrain foundation of a COBOL-specialist line. Qwen3-style architecture, HuggingFace-native.

⚖️ Honest scope

• This is a base (pretrain-only) model: it completes text/code, it does not chat, follow instructions, or solve tasks. Those capabilities come from the post-training stages (SFT/DPO), which are not part of this checkpoint.
• It is small and token-budget-limited (980M params, 20.37B training tokens) → undertrained by modern code-LLM standards. Expect competent local syntax, not strong reasoning.
• The corpus is code-dominated (general code is the mass; COBOL is a weighted minority). COBOL specialization is intentionally deferred to post-training and is not demonstrated by this base.
• Use it as a research artifact / a from-scratch foundation, not as a product.

Architecture

Training

• From scratch, random init — no warm-start, no third-party pretrained weights.
• 20.37B tokens, 1 epoch, sequence length 8192, AdamW (β 0.9/0.95), cosine LR (peak 3e-4 → 3e-5), warmup 2000, on a single B200.
• Corpus mix (tokenized): ~88% general code (permissive, multi-language) · ~7.6% Italian · COBOL up-weighted minority · conversational.

Results (measured)

A base model has no instruction-following benchmark to report: functional COBOL ability (compile rate, pass@1) emerges only after post-training and is evaluated on the final model, not on this base.

Usage

Use the skylar library — no custom modeling code lives in this repo; the architecture is provided by the package.

pip install skylar

import skylar

m = skylar.load("Skyl4r-Ai/Skylar-980M-Cobol-Base")

# base model = text/code completion (no chat, no instructions)
print(m.generate("       IDENTIFICATION DIVISION.\n       PROGRAM-ID. HELLO.\n",
                 max_new_tokens=80))

<details> <summary>Alternative — plain 🤗 transformers</summary>

import skylar registers the nano-transformer architecture, so AutoModelForCausalLM works with no trust_remote_code and no modeling files in the repo:

import skylar                          # registers the architecture
from transformers import AutoModelForCausalLM
from tokenizers import Tokenizer
from huggingface_hub import hf_hub_download
import torch

repo = "Skyl4r-Ai/Skylar-980M-Cobol-Base"
model = AutoModelForCausalLM.from_pretrained(repo).eval()
tok = Tokenizer.from_file(hf_hub_download(repo, "tokenizer.json"))

prompt = "       IDENTIFICATION DIVISION.\n       PROGRAM-ID. HELLO.\n"
ids = torch.tensor([tok.encode(prompt).ids])
out = model.generate(ids, max_new_tokens=80)
print(tok.decode(out[0].tolist()))

</details>

Part of the Skylar framework — a from-scratch decoder-only LLM training stack. This is the pretrain base; the instruction-tuned / specialist models are released separately.

License & attribution

© A. Ivanovitch — Apache-2.0. Code & framework: github.com/2sophia/skylar