Flow-Drafter-9B-tr

Flow-Drafter-9B-tr

Speculative-decoding draft model for Qwen/Qwen3.5-9B (Chained-Flow, joint-VAE drafter), adapted to Turkish.

Lossless by construction: a proposed token is emitted only if the target itself would have emitted it. Outputs can still differ from unspeculated decode, because verifying K+1 positions at once changes the target's own fp16 reduction order — the observed divergence is comparable to the run-to-run difference between two unspeculated runs.

Warm-started from Flow-Drafter-9B-v2 (sha256 526efd6347... — see the repo for the full value), not trained from scratch. Identify the parent by that content hash, not by commit — the parent repo has since moved commit with byte-identical weights. These weights are the ones in this repo.

Usage

pip install chained-flow

# CF_DRAFTER_DIR is the only variable you need; a bare repo id works and pulls shortlist.pt with it.
# K comes from num_speculative_tokens, not an env var.
CF_DRAFTER_DIR=selimaktas/Flow-Drafter-9B-tr \
vllm serve Qwen/Qwen3.5-9B --async-scheduling \
  --speculative-config '{"method":"custom_class","model":"chained_flow.vllm_plugin.flow_proposer.FlowDrafterProposer","num_speculative_tokens":5}'

curl localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{
  "model": "Qwen/Qwen3.5-9B",
  "messages": [{"role": "user", "content": "Merhaba, nasılsın?"}],
  "chat_template_kwargs": {"enable_thinking": false},
  "temperature": 0
}'

chat_template_kwargs: {"enable_thinking": false} is required — this drafter was trained on prompts carrying the <think>\n\n</think>\n\n prefix.

Served speedup

Every number below was measured under one condition: natural EOS (no ignore_eos), concurrency 1, greedy, chain proposer at K=5, async scheduling on for every arm including the baseline, 2 repeats, through vllm serve. Prompts: the chained-flow Turkish holdouts (bench_data_tr), 4 domains (173.6 tokens/request, not a forced 256). Aggregation is token-weighted pooling across those domains — each arm's own tokens ÷ its own wall-time — not a mean of per-domain ratios. Under natural EOS the arms emit different token counts, so the rates are comparable but the token totals are not.

81.4 → 118.9 tok/s, +46% against unspeculated decode.

The English parent is a net regression on Turkish (0.848x): at accept 1.27 it does not earn back its own draft pass. This fine-tune turns speculation from a loss into a win, not a bigger win.

The candidate shortlist ships with the model and is loaded automatically; the loader refuses to run against a different one.

Caveats

Turkish prompt sets only; this card does not characterise English behaviour.

Files

• model.safetensors — the drafter (flow experts, Markov head, path head, jointly-trained VAE)
• chained_flow_tree_config.json — architecture + loss config
• shortlist.pt — English+Turkish candidate shortlist (see above)
• vae/ — the base VAE checkpoint