qwen3-1.7b-eagle3-sharegpt-sw512

Qwen3-1.7B EAGLE3 Draft Model - SW512

This repository contains intermediate checkpoints from SpecForge EAGLE3 draft-model training against Qwen3-1.7B. The draft attention uses a causal sliding window of 512 tokens.

Checkpoints

Checkpoints are stored in epoch_<epoch>_step_<global_step>/ directories at 5,000-step intervals. Each directory contains:

• config.json: draft model configuration
• model.safetensors: draft model weights
• training_state.pt: epoch, global step, optimizer state, scheduler state, and saved training arguments

This upload contains 82 checkpoints from step 5,000 through step 410,000. The latest local checkpoint at upload time is epoch_8_step_410000.

The checkpoint weights exclude the frozen target embedding weights. SpecForge loads those weights from the target model when training or resuming.

Training Configuration

<details> <summary>Full saved training arguments</summary>

target_model_path: /data/HF_MODELS/Qwen3-1.7B
draft_model_config: <workspace-root>/code/test-spec/SpecForge/configs/qwen3-1.7b-eagle3.json
embedding_key: model.embed_tokens.weight
target_model_backend: sglang
train_data_path: /data/HF_MODELS/data/specforge_sharegpt/sharegpt_train_clean.jsonl
train_hidden_states_path: null
eval_data_path: null
eval_hidden_states_path: null
chat_template: qwen
is_preformatted: false
train_only_last_turn: false
build_dataset_num_proc: 16
dataloader_num_workers: 4
num_epochs: 10
max_num_steps: null
total_steps: 463620
batch_size: 1
learning_rate: 0.0001
max_length: 2048
draft_sliding_window: 512
warmup_ratio: 0.015
max_grad_norm: 0.5
ttt_length: 7
draft_accumulation_steps: 1
tp_size: 1
dp_size: 2
sp_ulysses_size: 1
sp_ring_size: 1
attention_backend: sdpa
sglang_attention_backend: flashinfer
sglang_mem_fraction_static: 0.4
sglang_context_length: null
sglang_enable_nccl_nvls: false
sglang_enable_symm_mem: false
sglang_enable_torch_compile: false
sglang_enable_dp_attention: false
sglang_enable_dp_lm_head: false
sglang_enable_piecewise_cuda_graph: false
sglang_piecewise_cuda_graph_max_tokens: 4096
sglang_piecewise_cuda_graph_tokens: null
sglang_ep_size: 1
save_interval: 5000
eval_interval: 5000
log_interval: 50
seed: 0
trust_remote_code: false
is_vlm: false
report_to: none

</details>

Usage

These are SpecForge EAGLE3 draft checkpoints, not standalone causal language models. Load them with the matching SpecForge draft-model implementation and the Qwen3-1.7B target model.