affine-checkpoint-transfer

Affine Transform: EleutherAI/deep-ignorance-pretraining-stage-unfiltered (global_step38144) → EleutherAI/deep-ignorance-unfiltered

Learned affine transformation mapping hidden state activations from a source checkpoint to a target model.

Usage

from safetensors.torch import load_file
import torch.nn as nn
from huggingface_hub import hf_hub_download

# Download files
weights_path = hf_hub_download(
    repo_id="EleutherAI/affine-checkpoint-transfer",
    filename="affine_transforms.safetensors",
)
metadata_path = hf_hub_download(repo_id="EleutherAI/affine-checkpoint-transfer", filename="metadata.json")

# Load
import json
with open(metadata_path) as f:
    metadata = json.load(f)

weights = load_file(weights_path)
affine_transforms = {}
for layer_idx in metadata["layer_indices"]:
    linear = nn.Linear(metadata["hidden_dim"], metadata["hidden_dim"], bias=True)
    linear.weight.data = weights[f"layer_{layer_idx}.weight"]
    linear.bias.data = weights[f"layer_{layer_idx}.bias"]
    affine_transforms[layer_idx] = linear

MSE Metrics

Mean MSE: 1.073099

Training Details

• Source Model: EleutherAI/deep-ignorance-pretraining-stage-unfiltered (global_step38144)
• Target Model: EleutherAI/deep-ignorance-unfiltered
• Hidden Dimension: 4096
• Ridge Alpha: 0.01
• Layers: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31]
• Training Examples: 100000