Downloads · 30 days
0
CAT-YOKO
AvrovaDonz/CAT-YOKO
CAT-YOKO is a text generation model from AvrovaDonz. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
YOCO-style causal encoder-decoder MoE, upcycled from openbmb/MiniCPM5-2B-Base (Apache-2.0, Llama GQA).
Access
Public
Updated Oct 7, 2026
Repo size
18.4 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.pt12.3 GB · 100%
At a glance
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Access
- Public
- Created
- Sep 18, 2026
- Updated
- Oct 7, 2026
- SHA
- fcc6b963
Try a prompt
Base models
Model card
From the Hugging Face model README
CAT-YOKO-12B
YOCO-style causal encoder-decoder MoE, upcycled from openbmb/MiniCPM5-2B-Base (Apache-2.0, Llama GQA).
This card is AvrovaDonz/CAT-YOKO. Training code: AvrovaDonz2026/CAT-YOKO. Operator implementations and evidence: AvrovaDonz/CAT-YOKO-KERNEL.
library_name: transformers only means tokenizer / shard convention follows the Hugging Face ecosystem. The graph is the in-repo cat_yoko PyTorch implementation, not an AutoModelForCausalLM architecture on Hub.
Spec
| Name | CAT-YOKO-12B |
| Architecture | YOCO-style causal encoder-decoder MoE |
| Base | MiniCPM5-2B (Llama GQA, untied) |
| (d) | 2048 |
| (V) | 130560 |
| (L) | 42 (16 encoder + 26 decoder) |
| Attention | 16 Q / 2 KV, head_dim=128 |
| FFN | SwiGLU intermediate 6144 |
| MoE | 1 shared + 20 routed |
| top-(k) | encoder 7 / decoder 10 |
| Stored params | 12,250,381,312 (≈12.25B) |
| Encoder active | ≈2.03B / input token |
| Decoder active | ≈4.33B / output token |
Curriculum C1
| Stage | tokens | Encoder | Trainable |
|---|---|---|---|
| B0 | 8B | frozen | new modules only |
| B1 | 27B | frozen | decoder + lm_head + final RMSNorm |
| B2 | 15B | trainable | all |
Wall-clock (published ledger)
Theoretical hours on a 50B-token envelope, not a measured run. On B200 / SM100, allowed linear GEMMs use TeNvfp4Linear (TE NVFP4BlockScaling; B0 frozen encoder is FPROP, WGRAD is B1/B2). Without TE or on sm_120, Nvfp4Linear E2M1/16 emulation. Attn softmax / SDPA stay fp32.
| Recipe | H100-h | Role |
|---|---|---|
| C1+NVFP4 | 571 | published wall-clock. B0 student bf16; B1/B2 allowed GEMMs NVFP4 |
| C1+FP8 | 729 | Hopper / Ada fallback |
| joint bf16 | 1325 | 100% baseline |
Data and tokenizer
| Published recipe | Ultra-FineWeb en 55% / zh 30% + UltraData-Math 10% + StarCoder 5%; planned mixture |
| Current real-text pilot | Ultra-FineWeb en 60% / zh 30% + UltraData-Math L2 10%; no code slice |
| Prepared pilot | 79,998,976 training tokens / 999,424 validation tokens, packed at 4096 |
| Tokenizer | MiniCPM5-2B-Base tokenizer files; identities recorded in the release manifest |
Current B0 snapshot
The latest completed B0 recovery snapshot is step 81864. Adam / absolute cursor: 47062; next corpus row: 8000. B0 remains in its original budget.
Use trainable.pt for optimizer/RNG/cursor recovery or weights-only.pt for the identical overlay without Adam. Both require MiniCPM5-2B-Base upcycling; no full 12B graph is uploaded. Snapshot instructions and verified manifest record base, data, runtime and archived deployed code. Older downloader pins select historical snapshots.
Real packed inputs: 192,765,952, including repeated rows; unique training tokens: 79,998,976.
<!-- /CAT-YOKO latest completed release -->Historical weights
| Path | Source | Notes |
|---|---|---|
checkpoints/b0-rocm-realtext/step-53307/trainable.pt | Historical B0 recovery snapshot | Step 53307; overlay plus CPU Adam, RNG and cursor; identities above |
checkpoints/b0-rocm-realtext/step-53307/weights-only.pt | Same snapshot, smaller overlay | Same 132 weights byte-for-byte; no Adam |
checkpoints/b0-rocm-realtext/step-52616/ | Historical ROCm real-text release | Step 52616, cursor/Adam 17814; preceding packed-attention/native CPU Adam backend; both original files and hashes retained |
checkpoints/b0-full/trainable.pt | Historical Vast B200 DummyStream B0 | Step 26940, 130,041,856 cumulative phase tokens; sha256 7eebc9a4da78d79be71bbe52881f2a0eaffd899f58ada3a3325f410eca181955; weights only |
checkpoints/b0-3090-bf16/trainable.pt | Historical RTX 3090 BF16 DummyStream B0 | Step 33800; sha256 2dc31406c240ee8631eb49c22908e41734c6558325b4c19270dd7ab95679e690; weights only |
checkpoints/b0/trainable.pt | Historical 6000D --try, 32 steps | sha256 9012e5ac55c2f59ef7cacc34d5769444413d070116dbff0696c7b258b9aa0636; smoke snapshot |
checkpoints/b0-nvfp4-try/trainable.pt | Historical 6000D NVFP4 --try, 2 steps | sha256 461b4ffc05fd46e2668448393789764ccf9dd673644040fe4527259b176a510e; smoke snapshot |
checkpoints/b1/, checkpoints/b2/ | Future stages | No B1/B2 trained snapshot is included in this release |
The historical paths are preserved. The complete 23 GiB graph is not included. Historical logs: B200, 6000D, 3090. ROCm reports: real-text training, operator checks.
Validation observation
At completed step 81864, fixed held-out rows 0–31 have NLL 7.29112438273621 (32 batches, 130,877.0 valid tokens). Rows 32–63 have NLL 7.387959246989646 (32 batches, 130,878.0 valid tokens). These are two slices of the same pilot validation corpus, not external benchmarks.
License
| Artifact | License |
|---|---|
| This repo's code and derived weights | Apache-2.0 |
| MiniCPM5-2B base | Apache-2.0 |
- Task
- Text Generation
- Library
- transformers
- License
- apache-2.0
- Languages
- en, zh
- Created
- Sep 18, 2026
- Updated
- Oct 7, 2026