qwen3-1.7b-eagle3-sharegpt-sw1024

huluhuluu/qwen3-1.7b-eagle3-sharegpt-sw1024

qwen3-1.7b-eagle3-sharegpt-sw1024 is a machine learning model from huluhuluu. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for transformers. The card lists the license as apache-2.0.

This repository contains all 47 checkpoints from a SpecForge EAGLE3 draft-model training run for the Qwen/Qwen3-1.7B target model. The draft attention window was fixed to 1024 tokens. Checkpoints were saved every 5,00…

Downloads · 30 days

Access

Public

Updated Sep 19, 2026

Repo size

14 GB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.safetensors12.9 GB · 92%

At a glance

Library: transformers
License: apache-2.0
Access: Public
Created: Aug 18, 2026
Updated: Sep 19, 2026
SHA: 6fbccdf2

Base models

Qwen/Qwen3-1.7B

Model card

From the Hugging Face model README

Qwen3-1.7B EAGLE3 ShareGPT SW1024 Checkpoints

This repository contains all 47 checkpoints from a SpecForge EAGLE3 draft-model training run for the Qwen/Qwen3-1.7B target model. The draft attention window was fixed to 1024 tokens. Checkpoints were saved every 5,000 steps and once at the final step, 231,810.

This is a speculative-decoding draft model, not a standalone language model. Use it with the matching Qwen3-1.7B target model and SpecForge-compatible EAGLE3 inference code.

Checkpoint Layout

Each epoch_<epoch>_step_<step>/ directory contains:

model.safetensors: BF16 EAGLE3 draft weights
config.json: draft-model architecture and sliding-window configuration
training_state.pt: epoch, global step, scheduler state, optimizer metadata, and the original training arguments

Available steps are 5,000 through 230,000 at 5,000-step intervals, plus the final checkpoint at step 231,810. The recommended checkpoint is:

epoch_9_step_231810

Training Configuration

Parameter	Value
Target model	`Qwen/Qwen3-1.7B`
Method	SpecForge EAGLE3 online training
Training data	ShareGPT-style conversations (`sharegpt_train_clean.jsonl`)
Chat template	`qwen`
Epochs	10
Total steps	231,810
Batch size	1 per data-parallel worker
Data parallel size	4
Tensor parallel size	1
Initial learning rate	1e-4
Scheduler	Linear warmup followed by cosine annealing
Warmup ratio	0.015
Max gradient norm	0.5
Precision	BF16 model parameters with FP32 optimizer parameters
Max sequence length	2,048
Draft sliding window	1,024
TTT length	7
Draft accumulation steps	1
Save interval	5,000 steps
Eval interval	5,000 steps (no eval dataset configured)
Draft attention backend	`sdpa`
Target model backend	`sglang`
SGLang attention backend	`flashinfer`
SGLang static memory fraction	0.4
Dataset build processes	16
Dataloader workers	4
Random seed	0
Experiment reporting	Disabled (`report_to=none`)

The run used SpecForge commit 9fbbde8.

Resume History and Optimizer-State Limitation

Training was stopped after checkpoint step 15,000 and resumed on four GPUs from that checkpoint. The model weights, global step, epoch, and learning-rate scheduler were restored. Training then completed normally at step 231,810.

The FSDP checkpoint writer saved the rank-0 AdamW state without gathering the optimizer shards. Consequently, optimizer_state_dict["state"] is empty in all 47 training_state.pt files. These checkpoints can restore model weights, epoch, global step, and scheduler metadata, but they cannot restore Adam first- and second-moment tensors. A resumed training job will rebuild those moments from zero.

Download

Download the final checkpoint with the Hugging Face CLI:

hf download huluhuluu/qwen3-1.7b-eagle3-sharegpt-sw1024 \
  --include "epoch_9_step_231810/*" \
  --local-dir qwen3-1.7b-eagle3-sharegpt-sw1024

License

The model is released under the Apache 2.0 license, matching the Qwen3 base model license. Users remain responsible for complying with the target model, dataset, and SpecForge licenses and terms.

Library: transformers
License: apache-2.0
Created: Aug 18, 2026
Updated: Sep 19, 2026