Holo-3.1-35B-A3B-lora-demo

Holo-3.1-35B-A3B — Atlas LoRA demo adapter

A deliberately-overfit demo LoRA for the 40-layer, 256-expert MoE Hcompany/Holo-3.1-35B-A3B, trained to prove out the LoRA runtime in the Atlas pure-Rust CUDA inference engine end-to-end on a single NVIDIA GB10 — and to serve on the NVFP4-quantized base (~23 GB) while the adapter itself is trained against the BF16 base.

Train on BF16, serve on NVFP4

PEFT can only LoRA-train against an unquantized base, so this adapter is trained against the BF16 Hcompany/Holo-3.1-35B-A3B (~70 GB, full-bf16 frozen base, no QLoRA/bitsandbytes). Atlas then serves the compact NVFP4 base (Hcompany/Holo-3.1-35B-A3B-NVFP4) and applies this BF16 adapter as a runtime delta at attention k/v/o on the 10 full-attention layers — never merged, never dequantized. The BF16-delta-on-NVFP4-base quantization gap is why r=32 is used (enough capacity that exact tokens mostly survive; occasional 4-digit drift under NVFP4 is expected).

Atlas serving constraints (why the config looks the way it does)

• target_modules = ["k_proj", "v_proj", "o_proj"] — q_proj is a hard reject (Holo gated Q|gate interleave); MoE MLP (mlp.experts.* / gate/up/down) and GDN linear-attention layers are also rejects.
• layers_to_transform = [3, 7, 11, 15, 19, 23, 27, 31, 35, 39] — the 40-layer model's 10 full-attention layers (full_attention_interval=4); the other 30 are Gated-DeltaNet/linear.
• r = 32, lora_alpha = 64 (scale = α/r = 2.0), use_rslora = false, use_dora = false, bias = "none".

Requires the Atlas F32→BF16 adapter-loader fix and the holo3_1_moe LoRA family allow-list.

Usage

spark serve Hcompany/Holo-3.1-35B-A3B-NVFP4 \
  --lora-adapter holo35b=MonumentalSystems/Holo-3.1-35B-A3B-lora-demo \
  --max-lora-rank 64

Training

peft + transformers Trainer, 10 epochs / 1200 steps on 6 overfit Q→A pairs, r=32, lr 2e-4, bf16, full-bf16 frozen base + gradient checkpointing, ~75 min on a single NVIDIA GB10. A demo artifact, not a general-purpose fine-tune.