Downloads · 30 days
18
46% of all-time downloads
huluhuluu/qwen3-1.7b-eagle3-sharegpt-sw128-epoch8-step200000
qwen3-1.7b-eagle3-sharegpt-sw128-epoch8-step200000 is a text generation model from huluhuluu. Use it when you need the model to write or continue text. It is set up for transformers.
This private repository contains one SpecForge EAGLE3 draft-model checkpoint.
Downloads · 30 days
18
46% of all-time downloads
All-time downloads
39
Public
Parameters
137M
274 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors274 MB · 100%
How the weights are stored.
BF16137M · 100%
From the Hugging Face model README
This private repository contains one SpecForge EAGLE3 draft-model checkpoint.
This checkpoint should be associated with the feat/sliding-window branch of https://github.com/huluhuluu/SpecForge.git.
Branch link: https://github.com/huluhuluu/SpecForge/tree/feat/sliding-window
Branch commit recorded at upload time: 9fbbde8
| Field | Value |
|---|---|
| Epoch | 8 |
| Global step | 200000 |
| Local checkpoint directory | <workspace-root>/code/test-spec/SpecForge/outputs/qwen3-1.7b-eagle3-sharegpt-sw128/epoch_8_step_200000 |
| Files | config.json, model.safetensors, training_state.pt, README.md |
| Parameter | Value |
|---|---|
target_model_path | /data/HUGGINGFACE/Qwen3-1.7B |
draft_model_config | <workspace-root>/code/test-spec/SpecForge/configs/qwen3-1.7b-eagle3.json |
train_data_path | /data/HUGGINGFACE/data/specforge_sharegpt/sharegpt_train.jsonl |
train_hidden_states_path | `` |
eval_hidden_states_path | `` |
eval_data_path | `` |
num_epochs | 10 |
max_num_steps | `` |
batch_size | 1 |
learning_rate | 0.0001 |
max_length | 2048 |
warmup_ratio | 0.015 |
total_steps | 233900 |
max_grad_norm | 0.5 |
ttt_length | 7 |
draft_sliding_window | 128 |
save_interval | 5000 |
log_interval | 50 |
seed | 0 |
draft_accumulation_steps | 1 |
tp_size | 1 |
dp_size | 4 |
target_batch_size | 1 |
sp_ulysses_size | 1 |
sp_ring_size | 1 |
attention_backend | sdpa |
target_model_backend | sglang |
chat_template | qwen |
build_dataset_num_proc | 16 |
embedding_key | model.embed_tokens.weight |
cache_dir | <workspace-root>/code/test-spec/SpecForge/cache |
output_dir | <workspace-root>/code/test-spec/SpecForge/outputs/qwen3-1.7b-eagle3-sharegpt-sw128 |
resume | False |
report_to | none |
sglang_attention_backend | flashinfer |
sglang_mem_fraction_static | 0.4 |
sglang_context_length | `` |
sglang_enable_dp_attention | False |
| Parameter | Value |
|---|---|
architectures | ['LlamaForCausalLMEagle3'] |
model_type | llama |
dtype | bfloat16 |
hidden_size | 2048 |
intermediate_size | 6144 |
num_hidden_layers | 1 |
num_attention_heads | 16 |
num_key_value_heads | 8 |
head_dim | 128 |
draft_vocab_size | 32000 |
vocab_size | 151936 |
max_position_embeddings | 40960 |
rope_theta | 1000000 |
attention_dropout | 0.0 |
rms_norm_eps | 1e-06 |
max_window_layers | 28 |
sliding_window | 128 |
use_sliding_window | True |
transformers_version | 4.57.1 |
#!/bin/bash
SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
ROOT_DIR=$(dirname $SCRIPT_DIR)
export TORCHINDUCTOR_CACHE_DIR=$ROOT_DIR/cache/compiled_kernels
export MASTER_ADDR=${MASTER_ADDR:-127.0.0.1}
if [ -z "${MASTER_PORT:-}" ]; then
MASTER_PORT=$(python - <<'PY'
import socket
s = socket.socket()
s.bind(("127.0.0.1", 0))
print(s.getsockname()[1])
s.close()
PY
)
fi
# Four-card default example:
# CUDA_VISIBLE_DEVICES=0,1,2,3 bash examples/run_qwen3_1.7b_eagle3_online_sw256.sh 4 1
NUM_GPUS=${1:-4}
TP_SIZE=${2:-1}
BUILD_DATASET_NUM_PROC=${BUILD_DATASET_NUM_PROC:-16}
TARGET_MODEL_PATH=${TARGET_MODEL_PATH:-/data/HUGGINGFACE/Qwen3-1.7B}
TRAIN_DATA_PATH=${TRAIN_DATA_PATH:-/data/HUGGINGFACE/data/specforge_sharegpt/sharegpt_train.jsonl}
OUTPUT_DIR=${OUTPUT_DIR:-$ROOT_DIR/outputs/qwen3-1.7b-eagle3-sharegpt-sw128}
MAX_NUM_STEPS=${MAX_NUM_STEPS:-}
EXTRA_ARGS=()
if [ -n "$MAX_NUM_STEPS" ]; then
EXTRA_ARGS+=(--max-num-steps "$MAX_NUM_STEPS")
fi
torchrun \
--nnodes 1 \
--node_rank 0 \
--master_addr $MASTER_ADDR \
--master_port $MASTER_PORT \
--nproc_per_node $NUM_GPUS \
$ROOT_DIR/scripts/train_eagle3.py \
--target-model-path $TARGET_MODEL_PATH \
--draft-model-config $ROOT_DIR/configs/qwen3-1.7b-eagle3.json \
--train-data-path $TRAIN_DATA_PATH \
--build-dataset-num-proc $BUILD_DATASET_NUM_PROC \
--output-dir $OUTPUT_DIR \
--num-epochs 10 \
--batch-size 1 \
--learning-rate 1e-4 \
--max-length 2048 \
--draft-sliding-window 128 \
--ttt-length 7 \
--chat-template qwen \
--cache-dir $ROOT_DIR/cache \
--embedding-key model.embed_tokens.weight \
--tp-size $TP_SIZE \
--attention-backend sdpa \
--target-model-backend sglang \
"${EXTRA_ARGS[@]}"
training_state.pt is included for checkpoint provenance and optimizer/scheduler state.