s0-tuning-qwen3.5-4b-humaneval

S0 Tuning

S0 tuning adapts a recurrent language model to a task by learning its initial hidden states. These states are the model's working memory: matrices that carry information between tokens. We train their starting values on prompt-completion pairs in place of the usual zero initialization. The model weights stay frozen.

• The learned states give each new prompt a task-specific starting point without adding prompt tokens.
• The released states target Python code generation. Reuse them on new prompts, or train a separate state file for another task.

Results

Qwen3.5-4B on 84 held-out HumanEval problems, with thinking disabled. The S0 result is averaged across 10 seeds.

These are the reported experiment results, not a separate evaluation of this saved checkpoint.

Use

python -m pip install huggingface_hub
hf download JackYoung27/s0-tuning-qwen3.5-4b-humaneval --local-dir s0-tuning
cd s0-tuning
python -m pip install -e .

Run from the downloaded directory:

from s0 import S0Trainer

trainer = S0Trainer.from_pretrained("Qwen/Qwen3.5-4B")
trainer.load(".")
trainer.activate()
print(trainer.generate("def fibonacci(n):\n", max_new_tokens=128))

The saved states require Qwen3.5-4B. Its weights download separately and must fit in local memory.

Files

• states.safetensors: 24 learned initial states for Qwen3.5-4B, totaling 48 MiB.
• config.json and meta.json: tuning settings, layer indices, and state dimensions.
• s0/ and pyproject.toml: code to load or train states.
• data/train.jsonl: 45 verified HumanEval training examples.
• examples/ and experiments/: training and evaluation code, plus saved results.
• Paper.

MIT license.