HyLo-Qwen-14MLA14M2-8K-SFT

HyLo-Qwen-14MLA14M2-8K-SFT

HyLo converts a pretrained Transformer into a hybrid instead of pretraining one from scratch. Each layer is converted to one of two types: attention layers become Multi-head Latent Attention (MLA), which caches a low-rank latent rather than full keys and values, while the remaining layers become linear blocks (Gated DeltaNet or Mamba-2) that hold a fixed-size recurrent state and no KV cache. The converted model is then trained for long context, retaining short-context accuracy while running on a small fraction of the original KV cache.

This checkpoint upcycles Qwen/Qwen3-1.7B into 14 MLA layers and 14 Mamba-2 layers (28 layers total, 2.07B parameters), and was supervised-fine-tuned at 8,192 tokens (8K) of context with Qwen/Qwen3-8B as the distillation teacher.

Introduced in Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling (arXiv:2604.24715), where it appears as HyLo-Qwen-14MLA14M2 in Table 4 (Qwen3-1.7B backbone).

Model at a glance

Architecture

Layer types are placed by index, not in a repeating pattern: the MLA layers sit where the base model is most sensitive to losing full attention.

• MLA layers (14): [0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20, 22, 24, 26]
• Mamba-2 layers (14): [1, 3, 5, 7, 9, 11, 13, 15, 17, 19, 21, 23, 25, 27]

Mamba-2 is a state-space block that keeps a fixed-size recurrent state instead of a growing KV cache, so those layers contribute no KV cache at all. The MLA layers keep attention but cache a low-rank latent instead of full keys and values, which is what brings the total cache to 7.8% of the base model's.

The hybrid layout lives in hybrid_config.json; config.json is the base model's configuration and is kept for reference only.

Context length

Use this model up to 8,192 tokens. That is the length it was trained and evaluated at.

The backbone's native RoPE window (32,768 tokens) already covers the trained length, so no position interpolation was applied (rope_scaling.factor: 1.0).

max_position_embeddings in the config is 40,960, inherited from the base model. It is not a supported context length for this checkpoint: quality past 8,192 tokens is not something the paper measures or claims. Serving stacks size their KV cache from this field, so set the maximum length explicitly (for example --max-model-len 8192).

The 8K and 64K models are separate releases trained from the same recipe at different lengths. If you need 16K or more, use the 64K sibling of this model; the RULER numbers below show what happens when an 8K-trained model is pushed past its training length.

Training

• Loss: KL divergence between student and teacher next-token distributions (kl_weight 1.0, ce_weight 0.0)
• Global batch size: 32 sequences; 1 epoch, cosine schedule with 0.01 warmup ratio
• Precision: bfloat16 mixed precision
• Hardware: 8x AMD Instinct MI300X, FSDP
• Memory-efficient distillation: a fused KL kernel that avoids materialising the full logit tensor

Training data

Training used AMD-processed variants of these datasets (subsetting, reformatting to the chat template, and decontamination against the evaluation suites).

Evaluation

All numbers are taken from Table 4 (Qwen3-1.7B backbone) of the paper, measured 0-shot with the EleutherAI lm-evaluation-harness.

Commonsense reasoning (accuracy, 0-shot)

Long context: RULER (all 13 tasks) at 8K, 16K, 32K and 64K

The columns past 8,192 tokens are reported for completeness and show the model outside its training length; they are not a supported operating point.

Math

Usage

The hybrid layer types are not part of transformers, so AutoModelForCausalLM cannot build this model. Install AMD's hybrid-model code first:

git clone https://github.com/AMD-AGI/AMD-Hybrid-Models.git
cd AMD-Hybrid-Models/HyLo

Then follow the installation instructions in HyLo/README.md.

import torch
from transformers import AutoTokenizer
from hybrid.hybrid_wrapper import HybridModelWrapper

checkpoint = "amd/HyLo-Qwen-14MLA14M2-8K-SFT"

model = HybridModelWrapper.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).cuda()
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model.eval()

messages = [{"role": "user", "content": "Summarise the document above in five bullets."}]
input_ids = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).cuda()

tokens = model.generate(
    input_ids,
    max_new_tokens=256,
    do_sample=True, temperature=0.6, top_p=0.95, top_k=20,
    eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(tokens[0], skip_special_tokens=True))

Notes:

• The checkpoint stores float32 weights, which is what training wrote out. Load it in bfloat16 as shown above: that is the precision the evaluation numbers were measured at, and float32 doubles the memory footprint.
• Use the chat template shipped with this repository. It is the template the model was trained with, and it is not identical to the base model's.
• Unlike Qwen3's own template, this one does not open a <think> block, yet every Qwen release here began its answer with a <think> ... </think> span when we checked: the behaviour comes from distilling Qwen/Qwen3-8B. Strip or handle those spans if your application does not expect them, and budget tokens for them.
• Keep prompts within 8,192 tokens.

The HyLo family

Intended use and limitations

This is a research artifact released to support the paper. It has not been safety aligned or evaluated for production use.

• Capabilities and biases are inherited from the base model and the distillation teacher.
• Outputs can be factually wrong, biased, or otherwise objectionable; add your own safeguards before exposing it to users.
• Quality past 8,192 tokens is neither measured nor claimed.
• The recipe depends on a strong teacher model being available, which is itself a cost.
• Evaluation covers the benchmarks listed above only: no multilingual, coding, safety or instruction-following-at-length evaluation is reported.

License

Metadata declares apache-2.0, matching the existing AMD hybrid-model releases, and the bundled LICENSE file is AMD's research-only RAIL-MS licence. Where they disagree, the LICENSE file is the one that describes AMD's intent for this artifact, and part of the training data is non-commercial, so treat the model as research-only.

Citation

@article{fashi2026hylo,
  title={Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scaling},
  author={Parsa Ashrafi Fashi and Utkarsh Saxena and Mehdi Rezagholizadeh and Aref Jafari and Akash Haridas and Mingyu Yang and Vansh Bhatia and Guihong Li and Vikram Appia and Emad Barsoum},
  journal={arXiv preprint arXiv:2604.24715},
  year={2026},
  url={https://arxiv.org/abs/2604.24715}
}

@article{yang2025zebra,
  title={Zebra-Llama: Towards Extremely Efficient Hybrid Models},
  author={Yang, Mingyu and Rezagholizadeh, Mehdi and Li, Guihong and Appia, Vikram and Barsoum, Emad},
  journal={arXiv preprint arXiv:2505.17272},
  year={2025}
}

@article{li2025xecomla,
  title={X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compression},
  author={Li, Guihong and Rezagholizadeh, Mehdi and Yang, Mingyu and Appia, Vikram and Barsoum, Emad},
  journal={arXiv preprint arXiv:2503.11132},
  year={2025}
}