Downloads · 30 days
0
qorva-model-fast
abdallah400/qorva-model-fast
qorva-model-fast is a machine learning model from abdallah400. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Hybrid Mamba-2 + GQA Attention language model trained on TinyStories.
Access
Public
Updated Jun 9, 2026
Repo size
91.5 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt74.2 GB · 100%
At a glance
- Access
- Public
- Created
- May 31, 2026
- Updated
- Jun 9, 2026
- SHA
- ae60d7bc
Model card
From the Hugging Face model README
Qorva-Micro 32M
Hybrid Mamba-2 + GQA Attention language model trained on TinyStories.
Results
| Metric | Value |
|---|---|
| Parameters | 37.3M |
| Vocab size | 50257 |
| Tokens trained | 380.1M |
| Final loss | 2.053 |
| Best loss | 1.7397 |
| Best PPL | 5.7 |
| Eval PPL | 10.93 |
| Dead layers | 0/16 |
Architecture
- hidden_dim: 288 — num_layers: 16
- Hybrid ratio: 0.25 (Mamba + GQA Attention)
- Trained on: Kaggle T4 (free)
Load
import torch
from model.config import ModelConfig
from model.architecture import QorvaModel
ck = torch.load('qorva_micro_final.pt', map_location='cpu')
cfg = ModelConfig.from_dict(ck['config'])
model = QorvaModel(cfg)
model.load_state_dict(ck['model'])
model.eval()
- Languages
- en
- Created
- May 31, 2026
- Updated
- Jun 9, 2026