Downloads · 30 days
8
27% of all-time downloads
DiogenesChen122/Qwen3.6-27B-Lora-20260826
Qwen3.6-27B-Lora-20260826 is a text generation model from DiogenesChen122. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
Sequence-level distillation of GPT-5.6 teacher trajectories into Qwen/Qwen3.6-27B for CUDA sparse kernel generation (SpMV / SpMM / SpGEMM). Part of Dr.Sparse — Phase 0 checkpoint used as the initialization for Phase 1…
Downloads · 30 days
8
27% of all-time downloads
All-time downloads
30
Public
Repo size
954 MB
Likes
0
Public
Click a slice to open those files.
.safetensors934 MB · 97%
From the Hugging Face model README
Sequence-level distillation of GPT-5.6 teacher trajectories into
Qwen/Qwen3.6-27B for CUDA sparse
kernel generation (SpMV / SpMM / SpGEMM). Part of
Dr.Sparse — Phase 0 checkpoint used as the
initialization for Phase 1 (DPO) and Phase 2 (multi-turn GRPO).
Supersedes the 2026-08-17 checkpoint, which was trained on the smaller 172-sample 1.5× dataset. This one uses the full 4046-sample 1.05× corpus.
q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (dense + MLP) and in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj (linear-attention layers of Qwen3.6's hybrid stack). Vision tower
is excluded — the adapter lands only on model.language_model.*.base_model_name_or_path: Qwen/Qwen3.6-27B.spmm_k8 · 810 spmm_k32 · 435 spmm_k128
· 329 spmm_k256 · 753 spgemm · 696 spmv. Much better balance than the
earlier 1.5× threshold — SpMV and SpGEMM are meaningfully represented now.| framework | verl 0.8.0, SFT trainer |
| precision | bfloat16 (frozen base + bf16 compute; LoRA in fp32) |
| optimizer | AdamW, LR 1e-5, cosine, 3 % warmup, weight decay 0 |
| epochs / steps | 3 / 2880 total optimizer steps |
| global batch | 4 (dynamic sequence packing) |
| max_length | 24 576 (samples above this dropped at build time) |
| loss mask | assistant-turn only (pre-flight asserted) |
| hardware | 2 × NVIDIA B200 (single node, FSDP2) |
| wall clock | ≈ 42 h total (18.0 h on resume from step 960) |
Val loss (best) = 0.03070 at the end of epoch 3. Training curves in the
Dr.Sparse W&B project drsparse_distill.
import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base_id = "Qwen/Qwen3.6-27B"
lora_id = "DiogenesChen122/Qwen3.6-27B-Lora-20260826"
tok = AutoTokenizer.from_pretrained(lora_id, trust_remote_code=True)
base = AutoModelForCausalLM.from_pretrained(
base_id, torch_dtype=torch.bfloat16,
device_map="auto", trust_remote_code=True,
)
model = PeftModel.from_pretrained(base, lora_id)
Merge and serve via vLLM:
CUDA_VISIBLE_DEVICES=0,1 vllm serve <merged-dir> \
--served-model-name Qwen/Qwen3.6-27B \
--tensor-parallel-size 2 --dtype bfloat16 \
--language-model-only --trust-remote-code \
--max-model-len 32768 --port 8027
Trained with the exact system turn the Dr.Sparse coding agent uses at inference:
You are a CUDA sparse kernel specialist.
Chat template is applied with enable_thinking=False.