Downloads · 30 days
21
47% of all-time downloads
ceselder/qwen36-27b-maemm-inverter-pretrain-104m
qwen36-27b-maemm-inverter-pretrain-104m is a machine learning model from ceselder. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Full-parameter (FSDP2) fine-tune of Qwen/Qwen3.6-27B that learns to invert its own residual-stream activations: given a layer-42 activation vector injected at a marker token in layer 1 (h + ‖h‖·v at the marker positio…
Downloads · 30 days
21
47% of all-time downloads
All-time downloads
45
Public
Parameters
27.8B
55.6 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors55.6 GB · 100%
From the Hugging Face model README
Full-parameter (FSDP2) fine-tune of Qwen/Qwen3.6-27B that learns to invert its own residual-stream activations: given a layer-42
activation vector injected at a marker token in layer 1 (h + ‖h‖·v at the marker position), the model writes the text whose
final token produced that activation. This is the pretrain stage of the MAEMM inverter pipeline (pretrain on real activations →
midtrain on a direction mix → RL). Trained 2026-09-07 → 2026-09-09 on 8×B200.
| Init | Qwen/Qwen3.6-27B (all weights trained, bf16 compute, fp32 master + AdamW states) |
| Data | 104M real-activation examples: FineFineWeb corpus prefixes of 8–64 tokens, each forwarded alone with BOS; target = the verbatim prefix; activation = layer-42 residual at its last token |
| Effective batch | 4,096 (micro-batch 64 × grad-accum 8 × 8 GPUs) |
| LR | 1e-5, OneCycle, 25,391 steps, 1 epoch |
| Compute | ≈387 B200-hours (three legs: 9.1 → 6.5 → 6.7 s/step; two AdamW resets at steps 1,813 and 14,504, no visible loss wobble) |
| Final train loss | 1.666 (last 50 steps); the LoRA r64 baseline on 23M examples ends at 1.875; this run was at 1.847 at 23M examples |
| wandb | celestedeschamphelaere-personal/maxact-fast/jp82mr9a (train), realact104m_fullft_b4096_lr1e-5_eval (evals) |
Held-out fidelity = cosine between the clean layer-42 activation of the generated text (max over its last 5 tokens, best of 4 samples at T=1) and the target direction, averaged over 512 held-out directions per family. All 22 checkpoints from 2M to 104M examples sit inside the LoRA baseline's band 0.362–0.375 (best 0.371 at 59M examples; final 0.369). A second arm at batch 8,192 / lr 3e-5 matched at every example count and was stopped.

The flat mean hides opposite-signed family trends: real-activation fidelity (short and long context) rises with data while SAE, BSF, probe and J-lens fidelity fall — exactly the direction families absent from the pretrain data degrade.


Exact numbers behind every figure: data/fidelity_vs_examples.json, data/per_family.json, data/loss_curves.json.
Standard HF safetensors layout (12 text shards + model-nontext.safetensors for the vision tower, model.safetensors.index.json, config +
tokenizer). Load with AutoModelForCausalLM.from_pretrained(repo, torch_dtype=torch.bfloat16). The inverter prompt format, the layer-1
injection hook and the evaluator live in the MAEMM code (mxf/inject.py, sft/pretrain.py, eval/).
Report with all plots and the LoRA / 2M-sweep comparison: http://5.78.192.0/reports/view/maemm-sft-fullft-104m/report.html (auth-protected).