steerling-8b

Steerling-8B:

An interpretable causal diffusion language model with concept steering.

Paper: Scaling Inherently Interpretable Language Models
Code: github.com/guidelabs/steerling
Project page: guidelabs.ai/papers/scaling-inherently-interpretable-language-models

Steerling-8B is an 8 billion parameter language model that combines masked diffusion with interpretable concept decomposition. Unlike standard autoregressive LLMs, Steerling generates text by iteratively unmasking tokens in order of confidence, and decomposes its internal representations into human-interpretable concepts that can be inspected and steered.

Quick Start

pip install steerling

from steerling import SteerlingGenerator, GenerationConfig

generator = SteerlingGenerator.from_pretrained("guidelabs/steerling-8b")

text = generator.generate(
    "The key to understanding neural networks is",
    GenerationConfig(max_new_tokens=100, seed=42),
)
print(text)

Model Details

Architecture

Steerling uses block-causal attention, bidirectional within a block, and causal across blocks. The interpretable concept heads decompose transformer hidden states into:

hidden → known_features + unknown_features + epsilon = composed → logits

• known_features: Weighted sum of top-k learned concept embeddings (interpretable and maps to understandable features)
• unknown_features: Residual captured by a factorized unknown head (101,196 concepts, rank 256)
• epsilon: Small correction for reconstruction fidelity

Training Data

The Nemotron-CC dataset includes synthetic data generated by third-party models (Qwen, DeepSeek). Users should review the applicable license terms for their intended use case.

GPU Requirements

License

The Steerling source code and model weights are released under the Apache License 2.0.

The model weights are provided for research and evaluation purposes. The weights were trained on datasets with varying license terms, including Nemotron-CC-HQ and Dolmino Mix. Some training data includes synthetic content generated by third-party models with their own license terms. We are currently reviewing the implications of these upstream licenses for downstream use of the model weights. Please check back for updates on the weight licensing terms.

For questions about commercial use of the model weights, contact us at [email protected]