Downloads · 30 days
0
jiamingshan/qwen3-4b-dflash-official100k-recipe
qwen3-4b-dflash-official100k-recipe is a machine learning model from jiamingshan. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repo is meant to be runnable by an external user on a normal 8-GPU Linux machine. It does not assume the original lab Docker container or a /data/sjm/... checkout.
Downloads · 30 days
0
Access
Public
Updated Jun 19, 2026
Repo size
—
Likes
0
Public
Click a slice to open those files.
.py530 KB · 88%
From the Hugging Face model README
This repo is meant to be runnable by an external user on a normal 8-GPU Linux
machine. It does not assume the original lab Docker container or a
/data/sjm/... checkout.
It contains the training/eval wrapper code used for this comparison. The prepared 100K dataset is hosted separately at:
https://huggingface.co/datasets/jiamingshan/qwen3-4b-dflash-official100k-prepared
Requirements:
Qwen/Qwen3-4B and
z-lab/Qwen3-4B-DFlash-b16.git clone https://huggingface.co/jiamingshan/qwen3-4b-dflash-official100k-recipe
cd qwen3-4b-dflash-official100k-recipe
python3 -m venv .venv
source .venv/bin/activate
pip install -U pip
pip install -r requirements-train.txt
# Only needed if your machine is not already logged in.
huggingface-cli login
./00_check_env.sh
./01_download_data.sh
# Train Qwen E2E+DFlash.
./10_train_e2e.sh
# Train Qwen frozen+DFlash.
./20_train_frozen.sh
The two training scripts print the final checkpoint paths. They look like:
outputs/<run_id>/checkpoints/step_002000
To run the GSM8K serving eval, install vLLM in an eval environment and pass the two checkpoint paths:
python3 -m venv .venv-vllm
source .venv-vllm/bin/activate
pip install -U pip
pip install -r requirements-eval.txt
./30_eval_gsm8k_table.sh \
outputs/<frozen_run_id>/checkpoints/step_002000 \
outputs/<e2e_run_id>/checkpoints/step_002000
The requested table is written to:
requested_gsm8k_table.md
Common:
TARGET_NAME_OR_PATH=Qwen/Qwen3-4B
DRAFT_NAME_OR_PATH=z-lab/Qwen3-4B-DFlash-b16
DATA_PATH=data/qwen3_dflash_official100k_prepared
STEPS=2000
WORLD_SIZE=8
GRADIENT_ACCUMULATION_STEPS=3
EFFECTIVE_GLOBAL_BATCH=24
SAMPLES_SEEN=48000
MAX_SEQ_LEN=3072
MAX_ANCHORS=512
ANCHOR_BATCH_SIZE=512
GAMMA=7
LOSS_NORMALIZATION=weighted
FREEZE_TARGET_ALL=0
FREEZE_TARGET_UNUSED_TAIL=0
LR_DRAFT=1e-4
LR_TARGET=2e-6
DFLASH_LOSS_WEIGHT=1.0
LM_LOSS_WEIGHT=1.0
KL_LOSS_WEIGHT=0.0
FEATURE_ANCHOR_LOSS_WEIGHT=0.0
EOS_LOSS_WEIGHT=0.0
Total loss:
dflash_loss + lm_loss
FREEZE_TARGET_ALL=1
LR_DRAFT=1e-4
LR_TARGET=0
DFLASH_LOSS_WEIGHT=1.0
LM_LOSS_WEIGHT=0.0
KL_LOSS_WEIGHT=0.0
FEATURE_ANCHOR_LOSS_WEIGHT=0.0
EOS_LOSS_WEIGHT=0.0
Total loss:
dflash_loss
dflash_loss is the DFlash weighted cross-entropy objective over static draft
targets from the prepared data. It uses gamma=7 and
loss_normalization=weighted.
lm_loss is next-token cross entropy from the trainable target's own
lm_head, computed over loss_mask[:, 1:] from the same target forward pass
that supplies DFlash hidden features.
This is the observed full GSM8K no-thinking table from the 2026-06-19 run:
| Metric | Qwen target | Qwen target + zlab-official DFlash | Qwen frozen+DFlash | Qwen E2E+DFlash |
|---|---|---|---|---|
| Accuracy | 1220/1319 = 0.9249 | 1209/1319 = 0.9166 | 1211/1319 = 0.9181 | 1207/1319 = 0.9151 |
| Mean tok/s | 141.82 | 672.98 | 687.82 | 687.48 |
| Mean accept | n/a | 6.222 | 6.324 | 6.345 |
Mean tok/s is mean single-shard throughput across 8 one-GPU shards. The
aggregate 8-shard throughput is also stored in each eval run's
aggregate_summary.json, but it is not the value in this table.
src/train/train_joint_target_dflash.py: self-contained training script.third_party/dflash-official: DFlash draft model code needed by training.third_party/speculators: attention helpers used by the DFlash objective.src/eval: GSM8K prompt, scoring, vLLM benchmark, acceptance parsing, and
sharded aggregation scripts.recipe.json: machine-readable recipe summary.