Downloads · 30 days
226
30% of all-time downloads
alea-institute/kl3m-006-170m-checkpoint-63000
kl3m-006-170m-checkpoint-63000 is a text generation model from alea-institute. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as apache-2.0.
A 170M parameter language model trained on multi-domain legal text using the Muon optimizer with Phase A improvements: layer-selective spectral clamping and per-layer learning rates. This is the final checkpoint befor…
Downloads · 30 days
226
30% of all-time downloads
All-time downloads
755
Public
Parameters
182M
739 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors727 MB · 98%
From the Hugging Face model README
A 170M parameter language model trained on multi-domain legal text using the Muon optimizer with Phase A improvements: layer-selective spectral clamping and per-layer learning rates. This is the final checkpoint before 4x depth stacking to 500M parameters.
alea-institute/kl3m-data-sample-004-balancedPer-Layer Learning Rate Multipliers:
self_attn.q_proj: 0.7× (slower for better conditioning)self_attn.o_proj: 0.7× (slower for better conditioning)self_attn.k_proj: 0.9×self_attn.v_proj: 0.9×mlp.*: 1.0× (baseline)lm_head: 0.85×Layer-Selective Spectral Clamping:
q_proj, k_proj, v_proj, o_proj):
gate_proj, up_proj, down_proj):
Analysis of weight matrix conditioning shows Phase A improvements achieved target:
| Metric | 57K (Before Phase A) | 63K (After Phase A) | Improvement |
|---|---|---|---|
| Max Condition | 3101 | 2501 | ↓ 19.4% ✓ |
| Q_PROJ Median | 2274 | 2168 | ↓ 4.6% ✓ |
| O_PROJ Mean | 2085 | 1940 | ↓ 7.0% ✓ |
| Overall Median | 420 | 383 | ↓ 8.7% ✓ |
| Generation Quality | Good | Excellent | ↑ Reduced repetition |
Key Success: Phase A layer-selective clamping locked max condition at 2500 (target ceiling) while per-layer LRs improved gradient flow through attention layers.
Generates coherent, fluent legal text across multiple domains with excellent repetition control:
Phase A improvements reduced catastrophic repetition by 2.5x compared to earlier checkpoints.
from transformers import pipeline
# Create text generation pipeline
generator = pipeline(
"text-generation",
model="alea-institute/kl3m-006-170m-checkpoint-63000",
torch_dtype="auto",
device_map="auto"
)
# Generate text
outputs = generator(
"This Agreement is entered into as of",
max_new_tokens=100,
temperature=0.7,
top_p=0.95,
repetition_penalty=1.15
)
print(outputs[0]['generated_text'])
This checkpoint serves as the source for 4x cyclic layer duplication using the G_stack method (NeurIPS 2024):
The stacked 500M model is available at alea-institute/kl3m-007-500m-step0.
Phase A demonstrates spectral health improvements through targeted interventions:
Result: 19.4% reduction in max condition number with no loss of training speed or quality.
Alea Institute
For technical details, see the paper: https://arxiv.org/abs/2504.07854
@misc{kl3m2025,
title={KL3M: Knowledge-Guided Language Model Training},
author={Alea Institute},
year={2025},
url={https://arxiv.org/abs/2504.07854},
note={Trained with Muon optimizer, Phase A spectral improvements, and G_stack ready}
}
Apache 2.0