EAGLE3-Qwen3-8B

<div align="center"> <a href="https://mobilint.com"> <img src="https://raw.githubusercontent.com/mobilint/.github/main/assets/Mobilint_Logo_Primary.png" width="50%" alt="mobilint" /> </a> </div>

About

This repository provides a model compiled and optimized for Mobilint NPU hardware.
The model is packaged for deployment on Mobilint's acceleration stack and is intended to be used within that environment.

EAGLE-3 speculative decoding for Qwen/Qwen3-8B: a quantized target body plus a one-block draft head and FC layer, all compiled for ARIES.

Usage

Requires mblt-model-zoo with the transformers extra:

pip install "mblt_model_zoo[transformers]"

from transformers import AutoTokenizer
from mblt_model_zoo.hf_transformers import AutoModelForCausalLM

model_id = "mobilint/EAGLE3-Qwen3-8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

messages = [{"role": "user", "content": "Explain speculative decoding in two sentences."}]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
)
outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

Draft tree configuration

The draft-tree budget ships as eagle3_tree_top_k=8, eagle3_tree_depth=4 in config.json and num_assistant_tokens=26 in generation_config.json. These were selected by sweeping decode throughput on aries-rb rather than inherited from the 4B release.

Per decode step the draft head runs depth NPU calls and builds a candidate pool of top_k + (depth-1) * top_k² = 200 nodes, from which num_assistant_tokens - 1 are handed to the base model for verification.

Override per call if your workload differs:

outputs = model.generate(**inputs, max_new_tokens=256, num_assistant_tokens=16)

Notes from the sweep:

• top_k has almost no effect in the 5–8 range (±0.7%); depth and num_assistant_tokens are the parameters that matter.
• depth=6 (the 4B release's value) is 6–14% slower than depth=4 here.
• Thinking mode matters more than any tree parameter. With Qwen3's thinking enabled (the chat template default), throughput drops ~17% because chain-of-thought text is harder for the draft head to predict. If you run with thinking disabled, depth=4 remains the best choice; with thinking enabled, depth=3 with num_assistant_tokens=16 is ~4% faster than the shipped default.

Measured decode throughput on a single ARIES card, greedy, 256 output tokens, averaged over four prompts (English and Korean):

Acceptance (tokens_per_step, counting the base model's own token) ranges 2.6–3.8 depending on language and thinking mode. These figures come from one host and one prompt set; treat them as indicative rather than a specification.

Files