bitnet-roman-numeral-expert

bitnet-roman-numeral-expert

LoRA adapter for microsoft/bitnet-b1.58-2B-4T-bf16. Specialist for decimal → Roman numeral conversion (1–3999). One routed expert in a ternary Mixture-of-Experts project.

Results (120 held-out, exact-match)

Base cannot perform the conversion at all; the adapter learns it partially. Errors concentrate on subtractive-pair cases (IV, IX, XL, XC, CD, CM) and multi-symbol numbers. This is a genuine capability gain over base, not a trophy-grade expert — shipped as a routing target for the MoE.

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "microsoft/bitnet-b1.58-2B-4T-bf16"
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "UlukaDev/bitnet-roman-numeral-expert")   # attach adapter, no merge
tok = AutoTokenizer.from_pretrained("UlukaDev/bitnet-roman-numeral-expert")

msgs = [
    {"role":"system","content":"You are a careful calculator. Work step by step, then end with exactly 'The answer is X'."},
    {"role":"user","content":"What is the number 1994 in Roman numerals?"},
]
text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=220)
print(tok.decode(out[0], skip_special_tokens=True))

Do not call merge_and_unload() — BitNet re-quantizes to ternary each forward pass, which corrupts a merged LoRA. Always load as a separate adapter.