gemma-4-E2B-it-crap

gemma-4-E2B-it-crap

A newer version is available: v1.1 supersedes this model, with stronger robustness on out of distribution strings, repeated characters and multi turn prompts. This card documents the original release.

TL;DR: a State of the Art Binary Programming Model™. Normally a language model writes source code and a compiler turns it into a program. CRAP skips all that crap (like, the compiler) and writes the machine code itself, byte for byte. This is not a joke.

A LoRA fine-tune of google/gemma-4-E2B-it trained to generate raw executable binaries from natural language descriptions. The model targets ELF (Linux x86_64).

Part of CRAP - Compiler Replacement for Assembling Programs.

Given a prompt like "Generate a Linux x86_64 binary that prints 'hello world'", the model outputs the complete ELF binary as a hex string.

Links

• Project page: blog.studiohaynes.com/go/crap
• Repo: github.com/matthewhaynesonline/crap-public
• Paper: CRAP technical report (PDF)
• GGUF: matthewhaynesonline/gemma-4-E2B-it-crap-gguf - See the GGUF card for quant choices and precision caveats.

Usage

This repo contains the merged weights: a standalone transformers model. Load it directly with AutoModelForCausalLM; PEFT is not required.

System prompt

The system prompt is optional, but prefer the canonical one below. It carried most of training, and it's the only variant the eval numbers on this card were measured under; the alternatives are trained but not separately scored.

Most training examples used the canonical prompt; the rest were split between paraphrases of it and no system prompt at all. So a paraphrase works, and the user message alone is usually enough to trigger the task.

You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else.

Generating a binary

from transformers import AutoModelForCausalLM, AutoProcessor

model = AutoModelForCausalLM.from_pretrained("matthewhaynesonline/gemma-4-E2B-it-crap")
processor = AutoProcessor.from_pretrained("matthewhaynesonline/gemma-4-E2B-it-crap")

messages = [
    {"role": "system", "content": "You are a binary code generator. Given a description of a program, output the complete binary as a lowercase hex string with no spaces, newlines, or other formatting. Output only the hex string and nothing else."},
    {"role": "user", "content": "Generate a Linux x86_64 binary that prints 'hello world'."},
]
inputs = processor.apply_chat_template(
    messages, add_generation_prompt=True, tokenize=True, return_tensors="pt"
).to(model.device)

out = model.generate(**inputs, max_new_tokens=2048, do_sample=False)
hex_str = processor.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True)
print(hex_str)

Saving and running the output

The model emits the binary as a lowercase hex string. Decode it to raw bytes, write it to a file, and set the executable bit (stdlib only):

import re, subprocess
from pathlib import Path

# Strip any whitespace, then decode the hex to raw bytes
binary = bytes.fromhex(re.sub(r"\s+", "", hex_str))

path = Path("crap.out")
path.write_bytes(binary)
path.chmod(path.stat().st_mode | 0o111)  # add the executable bit (chmod +x)

# Linux x86_64: run it directly and inspect the result
result = subprocess.run([f"./{path}"], capture_output=True)
print("stdout:    ", result.stdout.decode(errors="replace"))
print("exit code: ", result.returncode)

A Linux binary runs as soon as the executable bit is set.

Training details

Training environment

Recorded at train time (not at push time) by training_metadata.json.

Library versions

Eval results

The evaluation is execution based with no partial credit. An example counts as passed only if the emitted hex decodes, the bytes parse as a structurally valid ELF and the binary runs with the exact expected stdout and exit code. (The stdout comparison strips a single trailing newline from both sides and nothing else; stray NUL bytes still fail).

The base model scores ~0/23 on the same harness.

Training progression

Scored on the split above every 50 steps, the unmerged adapter reaches its peak of 20/23 (87.0%) by step 300 and holds it across steps 450 - 550, while training loss falls smoothly throughout (token loss is not the eval metric). What you download is the merged model, and it scores 20/23 (87.0%).

The full per-step progression is in Appendix B of the paper.

Known limitations

A compiler computes every offset and length deterministically. CRAP approximates them, and these are the places the approximation shows.

• Output carries a trailing newline. Generated binaries print the requested string followed by \n (the training convention). Account for it when comparing output against an expected string.
• Write length miscounting is the main failure mode. For arbitrary strings the write length field can be wrong in either direction. Over counting reads past the payload and appends stray NUL bytes (visible when the output is piped, not on a terminal); under counting drops the trailing newline. Short strings are reliable; the error grows with length and novelty.
• Repeated characters make the byte errors worse. When a target string repeats a character (the doubled d in daddy, the oo in gobbledygook) the model is likelier to scramble the payload: drop a byte, swap two or duplicate one. It aggravates the write length and synthesis errors above rather than being a separate failure. Treat repeated characters and other rare byte patterns as lower confidence.
• Exit code immediates can be off. Decimal to hex immediate encoding is a persistent failure in the published checkpoint: exit(223) exits with 222. It survives to the end of training rather than being a mid run artifact, and it's what the exit code half of the eval exists to measure.
• Best on in distribution strings. The eval strings are held out, but they're drawn from the training generator's distribution. Genuinely novel words pass far lower: a 12 word out of distribution probe passed 4/12 (33%), and at n=12 that figure carries roughly 14 points of standard error. Far enough out, the failures move earlier in the pipeline: odd length (undecodable) hex or a header only stub, not just wrong behavior. The model leans on memorized byte sequences and synthesizes less than the headline number suggests.
• Single turn only. Trained on single turn examples; multi turn conversations degrade. Treat each request as independent.
• Linux x86_64 only. The reference binary-gen builder constructs and structurally validates macOS Mach-O, but the model is neither trained on it nor asked to emit it. Behavioral correctness is Linux only; on a Mac, run the output in a Linux container.