Massive-Activations-HLA

Massive Activations in Hybrid Linear Attention Models

Controlled-pretraining checkpoints accompanying Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus.

Legacy aggregate repository. This repository is retained permanently for compatibility with existing links, citations, caches, and workflows that use repo_id="startlux-models/Massive-Activations-HLA" together with a subfolder. For new downloads, code, and citations, use the Hugging Face Collection and the independent model repositories listed below. New model releases will be published as independent repositories rather than added here.

Resources: Model Collection · Official analysis repository · Quickstart · Installation · Reproduction guide

The Collection is the recommended entry point. Each checkpoint now has an independent public model repository, with its files at the repository root for direct download. No existing files have been removed from this aggregate repository, so legacy links and subfolder-based workflows continue to work.

Model collection

Layer numbers in this table are one-based for readability.

Compatibility and reproducibility scope

The baseline and gdn-nooutgate-* checkpoints load with the public, pinned environment documented in the GitHub repository:

conda create -n ma-hla python=3.12 -y
conda activate ma-hla
bash scripts/install_released_gdn_cu126.sh

The two gdn-gatedfa-* checkpoints are weights-only research artifacts. Their full-attention layers use a post-SDPA, head-specific sigmoid output gate inspired by the G1 design in Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free (official code). The exact GatedDeltaNet integration is not distributed. These two checkpoints are not part of the public from-scratch quickstart.

The repository loader raises an explicit compatibility message for gated-FA models unless a user deliberately configures a compatible local implementation.

Loading a baseline checkpoint

Register the public FLA architecture before using Transformers directly. The independent repositories do not require a subfolder argument:

import fla.models.gated_deltanet  # registers the custom config/model
from transformers import AutoModelForCausalLM, AutoTokenizer

repo_id = "startlux-models/gdn-340m-pas-fa-layer12-10b"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = AutoModelForCausalLM.from_pretrained(
    repo_id,
    torch_dtype="auto",
)

For analysis, prefer the GitHub registry and scripts because they validate the FLA version and recover full-attention layer metadata consistently.

Reproducibility notes

• Public FLA: v0.5.2, commit 9c8e42e762fce087c27b673af4922795d9edb85e.
• Exact A800/CUDA 12.6 package versions are recorded in requirements/released-gdn-cu126.txt in the code repository.
• The Summer prompt and inline five-domain examples reproduce the public smoke pipeline. Exact input-level regeneration of every paper panel requires the sampled JSONL inputs used for that panel.
• Model weights are Apache-2.0; repository analysis code is MIT.

Limitations

These are research checkpoints, not instruction-tuned or safety-tuned models. They have not been validated for production use. The gated-FA variants require an undistributed compatibility implementation as described above.