mini-nla-qwen2.5-3b

Mini NLA Qwen2.5-3B

This repository contains a small-scale natural language autoencoder built around Qwen2.5-3B-Instruct. The Activation Verbalizer (AV) maps a hidden-state activation to a natural-language explanation, and the Activation Reconstructor (AR) maps that explanation back into the activation space.

This is a custom PyTorch model bundle, not a standard text-generation checkpoint that can be loaded directly with AutoModelForCausalLM.

Contents

• base_model/: complete Qwen2.5-3B-Instruct weights, configuration, and tokenizer.
• av_sft/model.pt: supervised Activation Verbalizer.
• ar_critic/model.pt: jointly optimized Activation Reconstructor from GRPO step 1,280.
• av_rl/model.pt: GRPO Activation Verbalizer from step 1,280.
• data/: AV, AR, RL, and validation activation datasets.
• validation_fve_by_step.json: held-out FVE for the saved GRPO checkpoints.

The activation dimension is 2,048. Training used 1,024 AV examples, 1,024 AR examples, 2,048 RL examples, and 256 validation examples. Natural-language targets were generated with Qwen2.5-32B-Instruct.

Results

The supervised roundtrip obtained FVE -0.2524. The best paired GRPO checkpoint occurred at step 1,280 with FVE -0.1677. Training through step 2,048 produced FVE -0.1724, so this bundle uses the better step-1,280 checkpoint.

Loading

The checkpoints are loaded by the accompanying mini_nla implementation:

from mini_nla.models_av import ActivationVerbalizer
from mini_nla.utils import load_state_dict_if_exists

av = ActivationVerbalizer("base_model", activation_dim=2048, prefix_tokens=8)
load_state_dict_if_exists(av, "av_rl")

The AR is loaded similarly with CriticStyleActivationReconstructor.

Limitations

The learned roundtrip still has negative held-out FVE and does not outperform mean-activation prediction. The result indicates improved reconstruction with model scaling, but it is not evidence of a reliable scaling law.