Downloads · 30 days
0
HowardHsuuu/dataset-specific-clts-llama-3.2-1b
dataset-specific-clts-llama-3.2-1b is a machine learning model from HowardHsuuu. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This repository collects cross-layer transcoders adapted to specific text domains from mntss/clt-llama-3.2-1b-524k. All checkpoints read hookresidmid, reconstruct hookmlpout, use JumpReLU activations, and are exported…
Downloads · 30 days
0
Access
Public
Updated Nov 10, 2026
Repo size
128 GB
Likes
0
Public
Click a slice to open those files.
.safetensors104 GB · 100%
From the Hugging Face model README
This repository collects cross-layer transcoders adapted to specific text domains from
mntss/clt-llama-3.2-1b-524k. All checkpoints read hook_resid_mid, reconstruct
hook_mlp_out, use JumpReLU activations, and are exported in circuit-tracer format.
| Domain | Method | circuit-tracer | CLT-Forge | Features per layer | Training positions |
|---|---|---|---|---|---|
| factual completion | 1,024-feature residual adapter folded into the base CLT | factual_completion/residual/circuit_tracer | factual_completion/residual/forge | 33,792 | 4,718,592 |
| factual completion | full-parameter fine-tune | factual_completion/full/circuit_tracer | factual_completion/full/forge_4way | 32,768 | 4,718,592 |
The factual-completion corpus is built from the pinned
lighteval/natural_questions_clean revision recorded in the code repository. Examples use
the format Question: <question>\nAnswer: <short answer>; validation and test retain one
fact per row.
Download one subfolder and pass the resulting local directory to circuit-tracer or the evaluation scripts in the companion repository:
from pathlib import Path
from huggingface_hub import snapshot_download
subfolder = "factual_completion/residual/circuit_tracer"
snapshot = snapshot_download(
"HowardHsuuu/dataset-specific-clts-llama-3.2-1b",
allow_patterns=[f"{subfolder}/*"],
)
clt_path = Path(snapshot) / subfolder
Each circuit_tracer subfolder contains config.yaml, export.json, and one encoder and
decoder safetensors file per layer. The files are bfloat16 circuit-tracer exports.
The forge and forge_4way subfolders preserve the same final CLTs in CLT-Forge's native
checkpoint format for continued training or reproducing the export step. The residual run
has one weights file; the full fine-tune retains its required four feature shards. SHA-256
values for both formats are listed in manifest.json.
Operating points are selected only on factual_completion/val by matching the original
CLT's total L0 summed over all 16 layers, then locked before test and out-of-domain
evaluation. The matched-L0 threshold scales are:
| Checkpoint | Global positive threshold scale |
|---|---|
| residual | 1.4669679095979213 |
| full | 0.943084792237925 |
These scales are evaluation controls and are not baked into the checkpoint weights.
On the 2,048-example factual-completion test split:
| CLT | Total L0 | EV-F | MSE | KL | Top-1 agreement | Model replacement |
|---|---|---|---|---|---|---|
| original | 233.286 | 0.6430 | 0.003529 | 2.121 | 0.346 | 0.852 |
| residual | 233.354 | 0.7641 | 0.002476 | 1.312 | 0.605 | 0.889 |
| full | 233.486 | 0.7940 | 0.002066 | 0.996 | 0.646 | 0.929 |
The paired 200-prompt attribution-graph comparison uses identical document, position, and target tuples:
| CLT | Graph replacement | Direct error share | Top-10 overlap with original | New-feature share |
|---|---|---|---|---|
| original | 0.671 | 27.4% | 100% | 0.0% |
| residual | 0.773 | 20.0% | 90% | 11.4% |
| full | 0.786 | 18.4% | 84% | 0.0% |
The implementation, domain and training configs, locked evaluation protocol, and reproducible
launcher are in the
howard/organized-pipeline
branch. The tables above report the completed formal run; future tracked output directories are
written directly by that launcher rather than assembled after execution.
The exact source revision used to execute this run is preserved as
factual-completion-20261108-source
(commit 09b091bfe39f462bf126053c3cea78740e9977a4).
nq-1m-pilot tag;
it used a different data representation and evaluation protocol and is not part of the
comparison above.