qwen-cobol-27b

qwen-cobol-27b — a COBOL-specialized DoRA adapter for Qwen3.6-27B

A DoRA (weight-decomposed LoRA) adapter that fine-tunes Qwen3.6-27B to generate COBOL. Trained on data where every example is compiled and executed against the real GnuCOBOL compiler before entering the dataset.

On the COBOLEval benchmark (146 problems, HumanEval translated to COBOL):

It beats the published open-source SOTA on compile rate, more than triples the vanilla base model's compile rate (same model, only the training data changed), and more than doubles GPT-4o's Pass@1, on a total training budget under $120. It's still below the SOTA on functional correctness — see Limitations.

Usage

The adapter keys are canonical PEFT format, so it loads with standard peft — no patching needed:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.6-27B", torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, "AlexThunder0/qwen-cobol-27b")
tok = AutoTokenizer.from_pretrained("AlexThunder0/qwen-cobol-27b")

prompt = '''Complete the following COBOL program by implementing the PROCEDURE DIVISION.
The code MUST compile with the open-source GnuCOBOL compiler. Declare every variable
in WORKING-STORAGE, do not use IBM-only intrinsics, end with exactly one END PROGRAM.
Provide the complete, compilable COBOL program inside a single ```cobol code block.

```cobol
<the COBOLEval-style skeleton with a LINKAGE SECTION and a RESULT field>
```'''

msg = tok.apply_chat_template([{"role": "user", "content": prompt}],
                              tokenize=False, add_generation_prompt=True, enable_thinking=False)
out = model.generate(**tok(msg, return_tensors="pt").to(model.device),
                     max_new_tokens=2048, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

The model expects the COBOLEval sub-program format: a skeleton with a LINKAGE SECTION and a RESULT field, which it completes with the WORKING-STORAGE SECTION and PROCEDURE DIVISION, ending in GOBACK / END PROGRAM. Greedy decoding is what the reported numbers use.

How it was made

• Base: Qwen3.6-27B (open weights)
• Adapter: DoRA, rank 128, α=256, rsLoRA, 7 target projections; 2 epochs, bf16, single H100
• Data: a runtime-verified pipeline — every training example is compiled and executed with GnuCOBOL and checked against expected outputs before inclusion; decontaminated against COBOLEval (program-id / entry-point / n-gram overlap). The dataset itself is not released.

Full write-up: https://github.com/AlexThunder01/qwenCobol

Limitations

• Below SOTA on functional correctness (Pass@1 36.3% vs 49.3%). The model reliably writes valid, compilable COBOL (88.4%); of the 146 COBOLEval problems, 76 compile but give the wrong answer — mostly genuinely-wrong algorithms (52), inverted boolean conditions (14), and off-by-one errors (7).
• Tuned for the COBOLEval sub-program format and GnuCOBOL dialect. It may not follow other COBOL conventions (mainframe/Enterprise dialects, file I/O heavy programs).
• Greedy decoding recommended for this task; in earlier testing on this pipeline, the base model's "thinking" mode gave only a marginal Pass@1 gain over greedy direct decoding, at much higher inference cost.

Citation / references

• Benchmark: COBOLEval (BloopAI)
• SOTA reference: COBOL-Coder (arXiv:2604.03986)
• Base model: Qwen/Qwen3.6-27B