Downloads · 30 days
0
ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA
Qwen2.5-1.5B-Nogai-LoRA is a text generation model from ansarzeinulla. Use it when you need the model to write or continue text. It is set up for mlx. The card lists the license as apache-2.0.
An mlx-lm LoRA adapter for Qwen2.5-1.5B-Instruct, trained on raw Nogai text (Nogai-Unified-Corpus-v1, 9.8 M Qwen2.5 tokens). This is Phase 1 of NogaiLLM. It learns Nogai spelling and morphology, but after this phase t…
Downloads · 30 days
0
Access
Public
Updated Sep 23, 2026
Repo size
21.1 MB
Likes
1
Public
Click a slice to open those files.
.safetensors21.1 MB · 100%
From the Hugging Face model README
An mlx-lm LoRA adapter for Qwen2.5-1.5B-Instruct, trained on raw Nogai text (Nogai-Unified-Corpus-v1, 9.8 M Qwen2.5 tokens). This is Phase 1 of NogaiLLM. It learns Nogai spelling and morphology, but after this phase the model no longer follows chat instructions: it continues text like a newspaper. For translation, use Phase 2 (Qwen2.5-1.5B-Nogai-SFT-Experimental) on top of this adapter.
adapter_config.json)| Base | MLX copy of Qwen2.5-1.5B-Instruct |
| LoRA | rank 8, scale 20, dropout 0 |
| Trained layers | q/k/v/o/gate/up/down projections of layers 12–27 (16 blocks), 5.28 M parameters |
| Batch / iterations / learning rate | 2 / 2,500 / 2e-4, seed 0 |
| Max sequence length | 512 tokens |
| Hardware | Apple M2 Pro, 16 GB, mlx-lm |
Reported in the first version of the paper, on the corpus validation split:
| Model | Per-token perplexity | TD (digraphs / 100 words) |
|---|---|---|
| Qwen2.5-1.5B-Instruct (base) | 191.13 | 17.16 |
| + this adapter | 12.14 | 15.05 |
| Human Nogai text (reference) | — | 22.7 |
Notes:
eval_bits_per_byte.py.The model is a text continuer after this phase, so skip the chat template:
pip install mlx-lm
mlx_lm.generate --model Qwen/Qwen2.5-1.5B-Instruct \
--adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
--prompt "Буьгуьнги куьн" --max-tokens 128 --temp 0.5 --ignore-chat-template
To build the base for Phase 2, fuse this adapter into the model:
mlx_lm.fuse --model Qwen/Qwen2.5-1.5B-Instruct \
--adapter-path ansarzeinulla/Qwen2.5-1.5B-Nogai-LoRA \
--save-path local_qwen_1.5B_Nogai_Base
@misc{zeinulla2026nogaillm,
title = {NogaiLLM: Parameter-Efficient Continued Pre-Training and Catastrophic Forgetting in Zero-Resource Turkic Languages},
author = {Zeinulla, Ansar},
year = {2026},
note = {Manuscript under revision. Code: https://github.com/ansarzeinulla/NogaiLLM-Apple-Silicon}
}