Qwen3-8B-base-mla-topk8-rank256

Qwen3-8B-base-mla-topk8-rank256

This repository contains the attention-only incremental weights for an MLAfication conversion of Qwen/Qwen3-8B-Base. It is not a standalone full-model checkpoint.

Variant

The delta contains only q_proj, k_r_proj, low-rank kv_proj, and QK-Norm parameters. Embeddings, MLP/MoE, output projection, LM head, and other base-model parameters are intentionally omitted.

Loading

Use the MLAfication implementation to instantiate the MLAfication architecture from the base model, then load model.safetensors with strict=False:

from safetensors.torch import load_file

delta = load_file("model.safetensors")
load_result = mla_model.load_state_dict(delta, strict=False)

The architecture must be patched before loading the delta. Loading this repository directly with vanilla AutoModelForCausalLM.from_pretrained(...) is not supported. config.json includes the layer-wise RoPE indices and sanitized MLAfication metadata needed by the project integration.

Files

• model.safetensors: attention-only delta weights
• config.json: base architecture, RoPE indices, and MLAfication metadata
• manifest.json: provenance, byte size, tensor count, and checksum

License

The weights follow the Apache 2.0 license of the base Qwen3 model.