Dizel-1.5

Dizel v1.5

A 456M-parameter causal Transformer trained from scratch on a curated corpus of ~378M tokens (code, math, general text, and safety data).

Architecture

Training

• Phase A.1: General pretraining (15K steps)

• Phase A.2: Reasoning/code emphasis (5K steps)

• Phase B.1: Supervised fine-tuning (10K steps)

• Phase C.1/2: LoRA distillation from Qwen2.5-0.5B-Instruct

• Training data: D4niel-dev/Dizel-Datasets

Checkpoints

Auto mode uploads every phase's best checkpoint. Use --phase <name> to upload a single phase.

• Architecture version: 1.5

Usage

import torch
from model.architecture import DizelLM
from training.dataset import Tokenizer

# Load config
config = {
    "vocab_size": 100000,
    "context_length": 2048,
    "d_model": 1024,
    "n_layers": 28,
    "n_heads": 16,
    "kv_heads": 8,
    "head_dim": 64,
    "intermediate_size": 3072,
    "norm_type": "rmsnorm",
    "norm_eps": 1e-06,
    "rope_theta": 10000.0,
    "bias": False,
    "weight_tying": True,
    "dropout": 0.0,
}
from config import ModelConfigV15
mc = ModelConfigV15(**{k: v for k, v in config.items() if k != 'model_type' and k != 'architectures' and k != 'format_version'})

# Load model
model = DizelLM(mc)
ckpt = torch.load("dizel-v15-distill-best.pt", map_location="cpu", weights_only=False)
model.load_state_dict(ckpt["model_state"], strict=False)
model.eval()

# Load tokenizer
tokenizer = Tokenizer(model_path="dizel_v15.model")

# Generate
prompt = "def fibonacci(n):"
tokens = tokenizer.encode(prompt, bos=True, eos=False)
x = torch.tensor([tokens])
with torch.no_grad():
    logits, _ = model(x)
    next_token = logits[0, -1].argmax()
print(tokenizer.decode([next_token.item()]))

Files