Parallax-0.6B

Parallax-0.6B

Parallax (Parameterized Local Linear Attention) is an experimental language model that reuses the Qwen3 architecture but replaces softmax self-attention with the parallax op (attn_type="parallax"). This is the 0.6B-parameter model.

Reference: https://arxiv.org/abs/2605.29157

Model details

Usage

The model ships custom modeling code, so load it with trust_remote_code=True (requires a recent transformers with Qwen3 support):

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "YifeiZuo/Parallax-0.6B"
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True)

prompt = "The quick brown fox"
ids = tok(prompt, return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=32)
print(tok.decode(out[0], skip_special_tokens=True))

License

Released under the Apache 2.0 license.