ForgePlex-M1-6M

ForgePlex-M1-6M

ForgePlex-M1-6M is a ~6.58M-parameter Llama style language model from ForgeWorks the first model in the ForgePlex-M series. It was trained on 12.5B tokens of Fineweb-Edu.

We would like to thank Axiomic Labs for allowing us to use their TrainWork framework to train this model.

Architecture

Stock Llama-layout GQA + RoPE + RMSNorm + SwiGLU.

Config

vocab_size              = 4096
num_hidden_layers       = 10
num_attention_heads     = 7
num_key_value_heads     = 1
hidden_size             = 224
head_dim                = 32
intermediate_size       = 672
max_position_embeddings = 512
rope_theta              = 5000.0
tie_word_embeddings     = true
unique params           = 6,584,928

Training

• Data: FineWeb-Edu
• Tokenizer: Custom ForgePlexM1 4k
• Context: 512

Usage

Stock Transformers — do not pass trust_remote_code.

from pathlib import Path
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_path = r"C:\slm\ForgePlexM1\ForgePlex-M1-6M"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(
    model_path,
    torch_dtype=torch.float32,
    device_map="auto",
)

prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=80, do_sample=False)
print(tokenizer.decode(out[0], skip_special_tokens=True))

Or run python usage.py from this folder.

Caveats

• Tiny model (~6.6M): greedy decode is usually clearer than heavy sampling.
• Context limit is 512 tokens.