Downloads · 30 days
762
46% of all-time downloads
alea-institute/kl3m-007-500m-checkpoint-15500
kl3m-007-500m-checkpoint-15500 is a text generation model from alea-institute. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 500M parameter language model trained on multi-domain legal text using Gstack depth expansion (4x cyclic layer duplication from 170M baseline). This checkpoint represents 15,500 steps of continued training on the 12…
Downloads · 30 days
762
46% of all-time downloads
All-time downloads
1.6K
Public
Parameters
500M
2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2 GB · 99%
From the Hugging Face model README
A 500M parameter language model trained on multi-domain legal text using G_stack depth expansion (4x cyclic layer duplication from 170M baseline). This checkpoint represents 15,500 steps of continued training on the 120-layer architecture with Muon optimizer and Phase A improvements.
alea-institute/kl3m-007-500m-step0 (stacked from kl3m-006-170m-checkpoint-63000)| Step Range | Loss (100-avg) | Improvement |
|---|---|---|
| 1-100 | ~7.66 | Baseline (stacked initialization) |
| 5000-5100 | ~2.63 | Layers diverging, specializing |
| 11000-12000 | ~2.52 | Continued improvement |
| 14000-15000 | ~2.63 | Stabilizing |
| 15400-15600 | 2.62-2.81 | Current |
Overall improvement: ~65% reduction in loss from initialization to step 15500.
Based on "Stacking Your Transformers" (NeurIPS 2024, arXiv:2405.15319):
Stacking Pattern (G_stack = G_direct depthwise):
Source (30 layers): [0, 1, 2, ..., 28, 29]
Target (120 layers): [0-29, 0-29, 0-29, 0-29]
└─ Cyclic repetition 4 times (G_stack/G_direct)
Method: Direct duplication in depthwise manner (G_stack operator from paper)
Training Philosophy:
Muon Learning Rate: 0.000365 (depth-scaled)
Auxiliary Learning Rate: 0.0005
Muon Parameters:
Per-Layer Learning Rate Multipliers (Phase A):
self_attn.q_proj: 0.7× (slower for better conditioning)self_attn.o_proj: 0.7× (slower for better conditioning)self_attn.k_proj: 0.9×self_attn.v_proj: 0.9×mlp.*: 1.0× (baseline)lm_head: 0.85×Adjusted for 120-layer model with more frequent attention clamping:
Attention layers (q_proj, k_proj, v_proj, o_proj):
MLP layers (gate_proj, up_proj, down_proj):
LM head:
Source: alea-institute/kl3m-data-sample-004-balanced (streaming with buffer=32)
Document Distribution:
Data Characteristics:
Note: The training data is heavily weighted toward court documents (RECAP) which influences the model's generation style and domain expertise.
Analysis of Q-projection weight norms across checkpoints (Steps 14600-15000):
Divergence Timeline:
Steady increase indicates continued layer specialization and differentiation of the 4 duplicated stacks.
Fastest evolving (most specialization):
Slowest evolving (most stable):
Stack-Level Changes (14600 → 15000):
Interpretation: The 4 stacked copies are successfully diverging and specializing, with Stack 1 (early layers) showing the most evolution and middle Stack 3 being most stable.
temperature = 0.5
top_p = 0.9
repetition_penalty = 1.2
max_new_tokens = 128
Strengths:
Limitations:
Context Confusion Example:
Recommended Use Cases:
Not Recommended:
from transformers import AutoModelForCausalLM, AutoTokenizer
# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
"alea-institute/kl3m-007-500m-checkpoint-15500",
torch_dtype="auto",
device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained("alea-institute/kl3m-007-500m-checkpoint-15500")
# Generate with recommended parameters
inputs = tokenizer(
"<|start|>This Agreement is entered into as of",
return_tensors="pt"
).to(model.device)
outputs = model.generate(
**inputs,
max_new_tokens=128,
temperature=0.5,
top_p=0.9,
repetition_penalty=1.2,
do_sample=True
)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))
from transformers import pipeline
generator = pipeline(
"text-generation",
model="alea-institute/kl3m-007-500m-checkpoint-15500",
torch_dtype="auto",
device_map="auto"
)
outputs = generator(
"<|start|>WHEREAS, the parties desire to enter into",
max_new_tokens=128,
temperature=0.5,
top_p=0.9,
repetition_penalty=1.2
)
print(outputs[0]['generated_text'])
| Model | Layers | Params | Steps | Tokens | Loss | Use Case |
|---|---|---|---|---|---|---|
| kl3m-006-170m-checkpoint-63000 | 30 | 181.7M | 63K | 15.83B | 3.18 | Production 170M |
| kl3m-007-500m-step0 | 120 | 500.3M | 0 | 0 | ~8.0 | Stacked init |
| kl3m-007-500m-checkpoint-15500 | 120 | 500.3M | 15.5K | 2.16B | 2.62 | Current |
| kl3m-007-500m-checkpoint-* (future) | 120 | 500.3M | 100K+ | 14B+ | <2.5 | Production 500M target |
✓ Successful layer divergence (6.48% cross-stack variance) ✓ Stable gradient norms (mean 2.27, no clipping needed) ✓ 65% loss reduction from initialization ✓ Multi-domain legal text generation capability ✓ Proper spectral conditioning maintained
⚠️ Domain mixing issues (court vs contract language) ⚠️ Data distribution imbalance affecting generation style ↻ Continued layer specialization (stacks still differentiating) ↻ Loss descent toward production target
Target Performance (by step 50-100K):
G_stack Efficiency Gains (per NeurIPS 2024 paper):
Inherited from source checkpoint + Phase A regularization:
Attention Layers:
MLP Layers:
LM Head:
Training Stability: Zero gradient clipping events in recent window indicates healthy optimization landscape.
{
"stacking_method": "G_stack (G_direct depthwise)",
"implementation": "cyclic_duplication",
"paper": "Stacking Your Transformers (NeurIPS 2024, arXiv:2405.15319)",
"authors": "Du et al.",
"source_checkpoint": "checkpoints/muon_170m_phase2/step-00063000",
"growth_factor": 4,
"source_layers": 30,
"target_layers": 120,
"stacking_pattern": "[0-29] repeated 4 times",
"training_steps_post_stack": 15500,
"tokens_processed_post_stack": 2160107520,
"layer_divergence_pct": 6.48,
"expected_efficiency_gain": "54.6% fewer tokens (per paper)"
}
Alea Institute
If you use this model, please cite the G_stack paper:
@inproceedings{gstack2024,
title={Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training},
author={Du, Wenyu and Luo, Tongxu and Qiu, Zihan and Huang, Zeyu and Shen, Yikang and Cheng, Reynold and Guo, Yike and Fu, Jie},
booktitle={NeurIPS},
year={2024},
note={arXiv:2405.15319}
}
Apache 2.0