SymQwen3-1.7B-BDLM-10k

SymQwen3-1.7B-BDLM-10k

Direct conversion baseline: causal mask removed (free bidirectional attention), trained 10k steps from stock Qwen3-1.7B.

Part of the DLM1B qwen3_sym campaign (2026-08-26/27): a matched-total-compute comparison of AR→DLM conversion recipes for Qwen3-1.7B — direct bidirectional (BDLM), a causal mask-tolerance stage (CDLM), staged adaptations with free (BDLM) vs hard-symmetrized (SDLM) attention, and a λ-sweep of blended rules (MIX: λ·symmetrized + (1−λ)·free). Every path totals 10k steps at GBS 256 × L2048 (~5.2B tokens), lr 1e-5, wd 0, WSD 100/*/500, ADLMC v3 data mixture + frozen aug, all-position clean-target supervision.

⚠ SDLM eval rule. config.attn_mode records the training-time attention semantics and the loaded model applies it automatically. Do NOT override it — in particular, running the CDLM-SDLM checkpoint through free bidirectional attention collapses it instantly (established in the DLM1B toy laboratory, Exp 4). All state_dicts are architecturally identical across modes; the mode is forward semantics, not weights.

Endpoint results (all arms, for context)

All pass@1, greedy, generation canvas 256 (dQwen house harness; __g256 cells — NOT comparable to gen-1024 numbers elsewhere). τ0.95 = adaptive low-confidence commit, threshold 0.95. MBPP-499 = MBPP test minus task_id 493 (its 4.3k-char prompt starves a 2048 canvas). ⚠ Single seed, single greedy decode: per-problem paired tests (McNemar) find NO statistically significant difference between any two models; the consistent-sign finding is that only the hard-mirror (λ=1) arm degrades under adaptive commit. λ-comparisons are valid only within the staged family (same init/data/ steps); BDLM-10k has 2× the bidirectional-mode steps and 2× unique tokens.

Headlines: (1) conversion quality is remarkably ROBUST to the bidirectional mixing rule — hard mirror, free, and every tested blend land within ~2.5 points, none pairwise-significant; (2) the one consistent-sign effect is calibration under adaptive decode — only the hard-mirror (λ=1) arm loses answers at τ0.95, any free component makes adaptive commit a pure ~25% speedup; (3) λ=0.5 posts the best aggregates (interior maximum; hypothesis-grade, n=1). Details in the DLM1B campaign notes.

Loading

from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained(
    "EER6/SymQwen3-1.7B-BDLM-10k", trust_remote_code=True, dtype="bfloat16")
tok = AutoTokenizer.from_pretrained("EER6/SymQwen3-1.7B-BDLM-10k")

Forward contract (from ADLMC's DiffuQwen3): model(input_ids) prepends BOS internally; logits has shape (B, L, vocab) aligned to the input positions. Special ids: MASK = 151660 (<|fim_middle|>), PAD = 151643, BOS = 151652. The sym forward assumes a full canvas (attention_mask None/all-ones) and uses no KV cache.

Provenance

• Training: Vista (TACC) job 943781, 10k steps, attn_mode bidir; full step-level log in training_log.jsonl, exact CLI args in argparse.json.
• Code: DLM1B/qwen3_sym @ a431778 (self-contained stack; validation gates in verify_port.py — aug byte-parity, bitwise loss parity on real weights, save→reload).