Maq-1.0-4B-SWE

Maq 1.0 4B SWE

Post-trained Gemma 4 E4B (instruction-tuned) for software-engineering, coding-agent, and competitive-programming style chat. Fine-tuned with LoRA SFT on a mix of NVIDIA Nemotron code/SWE datasets plus Fable/OpenCode agent traces, then merged into full BF16 weights.

• Base: google/gemma-4-E4B-it (Gemma 4 E4B-IT; ~4.5B effective / ~8B with per-layer embeddings)
• Method: LoRA SFT, then manual merge into dense BF16 (W += (α/r) B A)
• This repo: merged Transformers checkpoint (model.safetensors, ~15 GB)

This is not a full-data, multi-epoch Nemotron-scale post-train. It is a LoRA run on a sampled mix (33,585 JSONL rows collected; 4,096 used for the actual 400-step train) on a single Intel Arc iGPU.

Intended use

• Coding assistants and SWE-style issue → plan → patch conversations
• Tool-using / agentic coding traces (bash, file edits, multi-turn)
• Competitive programming and code-reasoning prompts
• Local / on-device experiments on Gemma 4 E4B

Not intended as a general chatbot replacement, a safety-tuned model, or a drop-in for 70B+ SWE agents.

How to use

from transformers import AutoModelForCausalLM, AutoProcessor
import torch

model_id = "WaylonJBrown/Maq-1.0-4B-SWE"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)
tokenizer = getattr(processor, "tokenizer", processor)

messages = [
    {"role": "user", "content": "Write a Python function that returns the nth Fibonacci number."},
]
text = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))

Requires a recent transformers (Gemma 4 / Gemma4ClippableLinear support).

Training

Hardware

PLE embedding tables (embed_tokens, embed_tokens_per_layer, ~5.3 GB + 1.3 GB) stayed on CPU. Decoder weights + an untied lm_head copy ran on XPU. Vision/audio towers were dropped for this text SFT.

Recipe

Wall clock: 400/400 steps in 7h 22m (train_runtime 26540 s). ~2.98M tokens seen. ~63–85 s/step.

PEFT cannot re-inject this adapter into Gemma 4 Gemma4ClippableLinear on reload, so merge was done by applying LoRA deltas onto the base Linear weights (258/258 pairs, 0 missing, 0 shape mismatches).

Metrics (train only — no held-out eval)

Loss is noisy because sequences vary in length and source. No SWE-Bench / LiveCodeBench numbers for this checkpoint.

Data

Collected by streaming (not full Hub dumps) into a unified messages JSONL, then shuffled. Competitive-Programming-v2 is 97 GB on Hub; it was not downloaded in full.

Normalization: messages / ShareGPT conversations / Fable traces → {role, content} with system/user/assistant. Tool roles flattened into user/assistant text. Per-message cap 24k chars. Skip every 7th stream row for light diversity.

Safety note: Nexlab/fable5-agentic-coding-sft documents that refusals were filtered. That subset is in the mix; guardrails vs the base IT model may be weaker.

Files

Unmerged LoRA adapter (66.6 MB adapter_model.safetensors, r=8) was trained at adapter_config.json peft_type: LORA and merged here. PEFT from_pretrained on Gemma 4 ClippableLinear is currently unsupported; use these merged weights.

Limitations

• 1024 context during SFT (base supports much longer)
• 400 steps / ~0.78 epoch on 4096 examples — a short post-train, not a Nemotron production blend
• No benchmark suite reported
• Text-only SFT; multimodal towers were not trained
• Intel XPU mixed-device training path is research-grade
• Dataset licenses are mixed (NVIDIA CC-BY / subset BSD, MIT on Fable derivatives, Gemma Apache 2.0). Check each source before commercial use.

License

Weights derived from Gemma 4: Apache 2.0. Datasets retain their upstream licenses.

Citation

@misc{maq10-4b-swe,
  title        = {Maq 1.0 4B SWE},
  author       = {Waylon Brown},
  year         = {2026},
  howpublished = {Hugging Face},
  note         = {LoRA SFT of google/gemma-4-E4B-it on sampled Nemotron + Fable coding/SWE mixes}
}

Base model: Gemma Team, Gemma 4 Technical Report, arXiv:2607.02770.