qwen3-1.7b-binary-rp-b3-hold-r64

b3_binary_hold_r64 — Qwen3-1.7B LoRA-QAT binary (roleplay PT-BR)

Manual LoRA + binary QAT adapter for Qwen3-1.7B, trained for Brazilian Portuguese roleplay.

Results (vs Bonsai 1.7B baseline ~82.5)

Seeds: 42 → 91.2 · 123 → 89.5 · 2026 → 87.1
Eval: data/pt_rp_v3_test (42 scenes), script bench_v8_lora.py.

Training recipe

• Base: Qwen3-1.7B
• Quant: binary on MLP only (quant_targets=mlp)
• Schedule: warmup 300 → ramp 1200 → full α=1.0 from step 1500; saved at step 6600
• LoRA: r=64, alpha=128, residual=True, rsLoRA, soft-tanh STE, pre-RMSNorm
• KD: KL weight 2.0, temp 5.0
• Data: cpt_mix_v3/train.jsonl
• Early-stop: smoke_5fail@6700 (full eval still strong)

Files

Load (cluster / repo scripts)

import torch
from scripts.exp_v8_lora import inject_lora, load_v8_lora, set_quant_schedule
from scripts.bench_v8_lora import ...  # or train helpers

ckpt = torch.load("best_lora.pt", map_location="cpu", weights_only=False)
# inject_lora(..., quant_mode="binary", r=64, alpha=128, residual=True,
#             quant_targets="mlp", use_rmsnorm=True, rs_lora=True, soft_tanh=True)
# set_quant_schedule(warmup+ramp, warmup, ramp)  # force alpha=1.0
# load_v8_lora(model, "best_lora.pt")

Not a PEFT/peft adapter — custom LoRALinear + binary STE from this project.

License

Weights are a derivative of Qwen3-1.7B. Follow the base model license (Apache 2.0) and applicable Qwen terms.