Downloads · 30 days
29
32% of all-time downloads
DiogenesChen122/Qwen3.8-27B-Lora-20260826
Qwen3.8-27B-Lora-20260826 is a text generation model from DiogenesChen122. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
Sequence-level distillation of GPT-5.6 teacher trajectories into Qwen/Qwen3.8-27B for CUDA sparse kernel generation (SpMV / SpMM / SpGEMM). Part of Dr.Sparse — Phase 0 checkpoint used as the initialization for Phase 1…
Downloads · 30 days
29
32% of all-time downloads
All-time downloads
91
Public
Repo size
954 MB
Likes
1
Public
Click a slice to open those files.
.safetensors934 MB · 97%
From the Hugging Face model README
Sequence-level distillation of GPT-5.6 teacher trajectories into
Qwen/Qwen3.8-27B for CUDA sparse
kernel generation (SpMV / SpMM / SpGEMM). Part of
Dr.Sparse — Phase 0 checkpoint used as the
initialization for Phase 1 (DPO) and Phase 2 (multi-turn GRPO).
Task. Given a matrix profile, planner strategy, and a coding-agent prompt,
write a full kernel.cu implementing {op}_preprocess / {op}_compute / {op}_cleanup that compiles, verifies numerically, and beats cuSPARSE.
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (dense + MLP) and in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj (linear-attention layers of Qwen3.8's hybrid stack). Vision tower
is excluded — the adapter lands only on model.language_model.*.base_model_name_or_path: Qwen/Qwen3.8-27B.spmm_k8 · 810 spmm_k32 · 435 spmm_k128
· 329 spmm_k256 · 753 spgemm · 696 spmv. Much better balance than the
earlier 1.5× threshold — SpMV and SpGEMM are meaningfully represented now.| framework | verl 0.8.0, SFT trainer |
| precision | bfloat16 (frozen base + bf16 compute; LoRA in fp32) |
| optimizer | AdamW, LR 1e-5, cosine, 3 % warmup, weight decay 0 |
| epochs / steps | 3 / 2880 total optimizer steps |
| global batch | 4 (dynamic sequence packing) |
| max_length | 24 576 (samples above this dropped at build time) |
| loss mask | assistant-turn only (pre-flight asserted) |
| hardware | 2 × NVIDIA B200 (single node, FSDP2) |
| wall clock | ≈ 42 h total (17.9 h on resume from step 960) |
Val loss (best) = 0.02852 at the end of epoch 3. Training curves in the
Dr.Sparse W&B project drsparse_distill.
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "Qwen/Qwen3.8-27B"
lora_id = "DiogenesChen122/Qwen3.8-27B-Lora-20260826"
tok = AutoTokenizer.from_pretrained(lora_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
base_id, torch_dtype=torch.bfloat16,
device_map="auto", trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, lora_id)
Merge and serve via vLLM:
CUDA_VISIBLE_DEVICES=0,1 vllm serve <merged-dir> \
--served-model-name Qwen/Qwen3.8-27B \
--tensor-parallel-size 2 --dtype bfloat16 \
--language-model-only --trust-remote-code \
--max-model-len 32768 --port 8028
The adapter was trained with the exact system turn the Dr.Sparse coding agent sends at inference time:
You are a CUDA sparse kernel specialist.
Chat template is applied with enable_thinking=False, matching the way the
Dr.Sparse eval pipeline drives the model.