Qwen3.6-35B-A3B-Slovenian

Qwen3.6-35B-A3B — Slovenian (merged full model)

Trained by MediaAtlas — LLM fine-tuning on your own data, trained in the EU, weights delivered. Pricing · All our models

Na kratko: Slovenski jezikovni model (35B MoE, 3B aktivnih) za pogovor, znanje in prevajanje EN ⇄ SL, z razmišljanjem. Glede na neuglašeno osnovo: slovenski nabor acc_norm 0,623 → 0,654, prevajanje BLEU EN→SL 23,8 → 26,3 in SL→EN 30,9 → 35,0. Sprejme tudi slike.

Vision / slike: the vision encoder of the base Qwen3.6 model is kept in this checkpoint, so it also takes images (load it with the processor, as the base model). Our training was text-only: image understanding is the base model's and was not evaluated in Slovenian.

Merged full model (bf16) of Qwen/Qwen3.6-35B-A3B with Slovenian fluency, knowledge, and en↔sl translation baked in (continued-pretraining + SFT). This is the LoRA adapter texdata/Qwen3.6-35B-A3B-Slovenian-LoRA already merged into the base, so you can load it directly. MTP (multi-token-prediction) tensors are preserved (model-mtp.safetensors) for speculative decoding.

On held-out evals vs the untuned base: Slovenian-LLM-Eval acc_norm 0.623 → 0.654; translation BLEU en→sl 23.8 → 26.3, sl→en 30.9 → 35.0. Reasoning model.

💡 For plug-and-play chat, a GGUF build (LM Studio / llama.cpp) is at texdata/Qwen3.6-35B-A3B-Slovenian-GGUF; the LoRA adapter alone is at texdata/Qwen3.6-35B-A3B-Slovenian-LoRA.

⚠️ How to run — load in 4-bit (not bf16)

This qwen3_5_moe arch has a broken fp16/bf16 forward in current transformers (zero hidden state → constant logits → garbage). Even though the weights here are bf16, load in 4-bit (bitsandbytes nf4) — the quantization it was trained/served under — or use the GGUF build. (vLLM does not run this arch on Blackwell sm_120 GPUs.)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

repo = "texdata/Qwen3.6-35B-A3B-Slovenian"
bnb  = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                          bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True)
tok   = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(repo, quantization_config=bnb,
                                             trust_remote_code=True, device_map="auto")

# reasoning model: enable_thinking=False for direct answers/translation (else it spends tokens on <think>)
msg  = [{"role": "user", "content": "Prevedi v slovenščino: Good morning!"}]
text = tok.apply_chat_template(msg, add_generation_prompt=True, tokenize=False, enable_thinking=False)
out  = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=128)
print(tok.decode(out[0], skip_special_tokens=True))

Benchmarks

All numbers below use the 4-bit bnb (nf4) eval path — the quantization this model was trained/served under (see How to run; the native bf16 forward for this arch is broken). n=100/task, held-out. Random baseline ≈ 0.25–0.50.

Slovenian-LLM-Eval — acc_norm

Wins 6/7 tasks (+3.1 pts avg). Base at n=200 is consistent (0.614), so the gain is not a small-sample artifact; the piqa −1 is within n=100 noise (±~5 pts).

Translation — en↔sl BLEU / chrF (n=150 held-out OPUS, greedy, thinking off)

Model details

• Type: merged full model (bf16), MoE qwen3_5_moe (16 shards + model-mtp.safetensors).
• Base: Qwen/Qwen3.6-35B-A3B (Apache-2.0, official — not an uncensored variant).
• Adaptation: CPT + SFT LoRA (sft_qwen35a3b_sl) merged in; template: qwen. MTP preserved.

License & data provenance

Base Qwen/Qwen3.6-35B-A3B is Apache-2.0. CPT data is open-licensed (attribution applies) → this repo is license: other:

Attribute the above. Comply with each source's terms before commercial use.

Citation

Cite this work (Tadej Fius, MediaAtlas Ltd):

@misc{fius2026qwen35bsl,
  title        = {Qwen3.6-35B-A3B Slovenian (merged)},
  author       = {Fius, Tadej},
  year         = {2026},
  publisher    = {MediaAtlas Ltd},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/texdata/Qwen3.6-35B-A3B-Slovenian}
}

Upstream / source citations:

@misc{qwen3, title={Qwen3 Technical Report}, author={{Qwen Team}}, year={2025}, url={https://huggingface.co/Qwen}}
@misc{wikipedia_sl, title={Slovenian Wikipedia}, author={{Wikimedia Foundation}}, note={CC BY-SA 4.0}, url={https://sl.wikipedia.org}}
@misc{penedo2024fineweb2, title={FineWeb2: A sparkling update with 1000s of languages}, author={Penedo, G. and others}, year={2024}, url={https://huggingface.co/datasets/HuggingFaceFW/fineweb-2}}

CPT data: Slovenian Wikipedia (CC BY-SA 4.0) and FineWeb2-sl (ODC-BY 1.0).