cobolx-1.5b

COBOLX 1.5B

A small model that reads a COBOL program and tells you what is in it. What files it opens, what copybooks it pulls in, what it calls, what the fields are, and which paragraphs never run.

It was built to answer the first questions anyone asks when they inherit a codebase nobody has read in twenty years.

What it is good at, with numbers

Tested on 90 questions about 90 COBOL programs it had never seen. Those programs come from X-COBOL, a research dataset of 84 GitHub repositories, and they are real enterprise code including CICS. None of them were in the training data. We checked that by comparing file contents, not by trusting that the sources were different, and threw out 95 programs that turned out to overlap.

The two columns say different things and both matter.

The first column is strict. If a program declares six fields and the model names five, that case counts as zero.

The second column counts the items. Look at the "list all the data items" row: only 47% of cases were perfect, but the model named 82% of the fields. It is not failing to read the layout. It runs out of sentence before it runs out of fields.

So a fair summary is: it finds about four out of five things correctly, and gets everything right about two thirds of the time.

A model that answers everything with a fixed reply scores 0.1% on this test, so none of these numbers can be reached by guessing.

What it is bad at

Dead code. 53%. Working out which paragraphs never run means comparing every label in the program against every PERFORM statement. That is reasoning, not looking things up, and it is the one thing this model does not do reliably. Do not use it to decide what to delete.

Long lists. The more items an answer needs, the more likely it drops one. If you need every field, ask and then check.

Making things up when it is unsure. On one test program where the only COPY statement was inside a comment, the model still mentioned a dependency that does not exist. It is confident when it is wrong.

How to use it

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "thinkingdbx/cobolx-1.5b"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16")

program = open("PAYROLL.cbl").read()
prompt = f"```cobol\n{program}\n```\n\nWhat files does this program read or write?"

msgs = [{"role": "user", "content": prompt}]
enc = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              return_tensors="pt", return_dict=True)
out = model.generate(**enc, max_new_tokens=200, do_sample=False)
print(tok.decode(out[0][enc["input_ids"].shape[1]:], skip_special_tokens=True))

Keep programs under about 6KB. Longer ones were not in the training data and the model has not seen them.

Read the first two sentences of an answer. It tends to be right at the start and drift afterwards.

What it was trained on

9,112 question and answer pairs, built three ways.

Real COBOL. 1,358 programs from The Stack, about six megabytes. For comparison, the same release of that dataset holds 28 gigabytes of Python. COBOL is scarce because it lives inside banks, not on GitHub.

An earlier version of this card said six megabytes was nearly all the permissively licensed COBOL in existence. That was too strong. The Stack v2 holds 49 megabytes of it across 4,359 files, so there is more we have not used. It is less useful than it sounds, for a reason in the summary below.

Generated COBOL. 3,500 programs written by a program generator. Every one was fed to the GnuCOBOL compiler and thrown away if it did not compile. The answers are correct because the generator chose the facts before writing the program, so nothing had to be read back out and guessed at.

Broken COBOL. 700 programs deliberately broken in one specific way, where the answer is the compiler's actual error message.

Every answer was computed from the source by a rule, never written by another language model. That means nothing in the training data is invented, no other model's licence follows into these weights, and you can check any example by opening the program it came from.

About 8% of the answers say "there are none", because an earlier version could not report an absence. It had never seen one, so it made something up instead.

20% of the training mix is ordinary non-COBOL instruction data. Without it the model answered a question about the weather by writing a COBOL program.

Things we tried that did not work

Worth writing down so nobody repeats them.

Training only on the answer, not the question. Standard practice for this kind of tuning. Here it made things worse. The model memorised seven answer shapes, reached a training loss of 0.09, and lost the ability to read the code.

A 7B model instead of 1.5B. Tied on the first test set and hallucinated a copybook that only appeared in a comment. A second attempt was cut short when the cloud machine was reclaimed.

Answering in lists instead of sentences. The idea was that a list makes it harder to stop early. It moved the score two points and made two tasks slightly worse. The gap is the model's limit, not the format.

More data was the only thing that helped. Going from 2,870 to 9,112 examples took the hand written test from 5 out of 9 to 7 out of 9.

Licence and provenance

Apache 2.0. The base model is Qwen2.5-Coder-1.5B-Instruct, also Apache 2.0. Training data is The Stack's COBOL files plus generated programs. No teacher model was used at any point, so no other model's terms apply to these weights.

Repository contents include the LoRA adapter, the merged weights, the tokenizer, the evaluation harness, and a manifest with a checksum for every file.

Honest summary

This is a narrow tool with a measured boundary. It reads COBOL dependencies well and reasons about control flow poorly. Use it to answer "what does this touch", not "what can I safely remove".

The reason it is not better is not the model size or the training recipe. It is the training data, and the shape of it matters more than the size.

There are 49 megabytes of permissively licensed COBOL in The Stack v2. But 28 of those megabytes are one repository, and that repository is the NIST COBOL-85 conformance suite: compiler test cases, written to exercise a parser rather than to run a business. Take it out and 20.7 megabytes of real-world COBOL is left, spread across 608 repositories, with the ten largest holding 40 percent of it. We trained on six megabytes of that. So there is perhaps three times more usable data available, not the eight times the headline number suggests.

The COBOL that would really make this good is sitting inside insurance companies and banks, which is the whole problem with building it in the open.