maccy-96m-16k-base

Maccy 96M (61M active), 16K vocabulary

Maccy is a compact, from-scratch base language model trained on Apple Silicon. It has 96,359,193 total parameters and activates approximately 60,527,385 parameters per token through top-2 routing across 4 SwiGLU experts.

This is a base completion model, not a chat or instruction-following model.

Architecture

Maccy combines Kimi Delta Attention (KDA), Multi-head Latent Attention (MLA), and a sparse mixture of experts. Input and output embeddings are tied.

Usage

The repository includes a portable Transformers reference implementation built for Transformers 5.14 or newer. Because Maccy is a custom architecture, loading the model requires trust_remote_code=True.

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "bgub/maccy-96m-16k-base"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=torch.float32,
)

inputs = tokenizer("Once upon a time", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.8,
    top_k=50,
    use_cache=False,
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

For the optimized Apple-Silicon kernels and training code, see https://github.com/bgub/mokka.

Training

• Training data: Karpathy's shuffled ClimbMix repack, derived from NVIDIA Nemotron-ClimbMix
• Inherited pretraining: 2,120,089,600 tokens with the original vocabulary
• Vocabulary-recovery pretraining: 196,608,000 tokens
• Total parameter exposure: 2,316,697,600 tokens
• Recovery optimizer steps: 6,000
• Training context: 1,024 tokens
• Effective batch: 32 sequences / 32,768 tokens per optimizer step
• Precision: bfloat16 activations with float32 master weights

The 16K tokenizer retains the first 15,991 mergeable tokens from Maccy's original 32K byte-level BPE, then places the nine special tokens directly after them. This preserves all retained ordinary-token IDs and complete UTF-8 byte fallback. The resized model was recovered with continued pretraining on the same corpus.

NVIDIA's source dataset card designates ClimbMix for research and development under CC BY-NC 4.0. Users are responsible for reviewing both the source-dataset terms and this model's license before use.

Evaluation

The table below recomputes bits per byte (BPB) with the same harness and 256 KiB of UTF-8 text per corpus. Models use their native tokenizers, float32 inference, a common 1,024-token context, and a 512-token sliding stride. Lower is better.

ClimbMix favors Maccy. FineWeb-Edu favors NanoWhale and SmolLM2 and is not claimed to be held out from them. Possible WikiText-103 and enwik8 overlap for public reference models is unknown. Training budgets differ enormously, so this is a checkpoint comparison, not a controlled architecture comparison. Corpus hashes, model revisions, loss sums, and the reproduction script are in the source repository under benchmarks/results/reference-bpb-v1.

Limitations

• This checkpoint has not been post-trained for conversation or instruction following.
• The 1,024-token context is short by modern standards.
• Code, mathematics, factual reliability, and long-form coherence are limited.
• The portable Transformers implementation does not yet include a recurrent generation cache and is slower than Mokka's native Metal implementation.
• Training data may contain errors, biases, and objectionable material that the model can reproduce.