Ember-Ardent-350M

Ember-Ardent-350M

Ember Ardent is a roughly 351M-parameter decoder-only language model checkpoint.

Important status

This is not fully pretrained. It was trained for approximately 3.26B tokens, rather than the planned 10B-token run, and is published as a research checkpoint. It is intentionally left in this state unless someone wants to continue pretraining (CPT) from the included checkpoint.

The checkpoint is not presented as instruction-tuned, chat-tuned, or production-ready. Expect factual errors, repetition, and incomplete capabilities.

Architecture

• 24 transformer layers
• 1024 hidden size
• 16 attention heads / 4 KV heads (GQA)
• SwiGLU feed-forward layers
• RMSNorm and RoPE
• tied input/output embeddings
• 32,768-token vocabulary
• maximum sequence length 2,048
• DGM learning-rate schedule during pretraining

Files

• step-00014000.pt: native PyTorch checkpoint, including model state and training metadata
• tokenizer.json: tokenizer used during training
• tokenizer_metadata.json: tokenizer metadata
• config.json: architecture and checkpoint metadata

The .pt file uses the project's native ArdentForCausalLM implementation. It is not directly loadable with AutoModelForCausalLM without the Ember Ardent source code.

Continuing pretraining

Load step-00014000.pt with the Ember Ardent training code and resume the optimizer, scheduler, RNG, and token position. Do not treat this checkpoint as a final model.

Evaluation snapshot

On a short held-out validation sample at publication time:

• loss: 3.51
• perplexity: 33.45

These are development measurements, not standardized benchmark scores.

---

Part of North ML - open weights, honest numbers.