Downloads · 30 days
3
19% of all-time downloads
YukinoshitaYukino/latent-thinking-osk-swr-thk-vae-c64r4-dz512-wd10
latent-thinking-osk-swr-thk-vae-c64r4-dz512-wd10 is a machine learning model from YukinoshitaYukino. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for pytorch. The card lists the license as apache-2.0.
A PrefixCausalRegisterTextInterleavedVAE that compresses text spans into latent register slots, trained on three domains at once rather than on tool results alone.
Downloads · 30 days
3
19% of all-time downloads
All-time downloads
16
Public
Repo size
9.3 GB
Likes
0
Public
Click a slice to open those files.
.pt9.3 GB · 100%
From the Hugging Face model README
A PrefixCausalRegisterTextInterleavedVAE that compresses text spans into latent register slots, trained on three domains at once rather than on tool results alone.
Run: PrefixCausalRegisterTextInterleavedVAE-Qwen3-0.6B-Base-decQwen3-1.7B-Base-osk-swr-thk-bs4-c64r4-dz512-wd1.0-trainDec-fullft-interleavedPos/20260805_071527
| domain | dataset |
|---|---|
| OpenSeeker tool results | YukinoshitaYukino/openseeker-toolresult-sections |
| SWE-Rebench tool results | YukinoshitaYukino/swe-rebench-toolresult-sections |
| OpenSeeker reasoning | YukinoshitaYukino/openseeker-think-sections |
Combined through data.combined.CombinedDataset, cot_split_mode: tag.
This is the difference from
latent-thinking-toolresult-vae-c64r4-wd10,
which saw OpenSeeker tool-result sections only. Everything else about the
architecture is unchanged, so the two are directly substitutable.
Qwen/Qwen3-0.6B-Base, decoder Qwen/Qwen3-1.7B-Base, both fully fine-tuned (no LoRA)chunk_size 64, num_registers 4 — 64 text tokens become 4 latents, 16x compressionnum_chunks 20 — trained on spans up to 1280 tokens; longer spans still run but are outside the training distributionlatent_dim 512, pos_ids_mode interleaved, prefix_offset 16, kl_weight 1e-4, word_dropout 1.0lr 3e-5 cosine to 1e-5, 500 warmup iters, weight_decay 0.1, batch_size 4 x grad_accum 4, grad_clip 1.0, 1 epoch.
Paired: the same spans through both checkpoints, harvested from OpenSeeker
held-out traces by the downstream renderer itself (compress_think=True,
raw_recent_turns=2, raw_recent_think_turns=2) and encoded through the real
batching path, so the padding and bucketing are the ones training produces.
Token accuracy is exact-match against the decoder, parallel decode.
Within the trained length (spans ≤ 1280 tokens; 40 per kind, result median 839, think median 184). Reported as wrong tokens, because every arm is above 99.8% and the rates round to the same number:
| checkpoint | result (of 32,792) | think (of 14,809) |
|---|---|---|
2-domain ckpt_011424 | 11 | 18 |
3-domain ckpt_015200 | 5 | 3 |
3-domain ckpt_015987 | 2 | 2 |
Past the trained length — where the two separate. 16 spans of 1,410–3,329
tokens, 37,160 tokens total. These are all think: tool results arrive
pre-split into sections and do not reach this length, so the extrapolation
regime of this corpus is reasoning, which is exactly what the 2-domain
checkpoint never saw.
| checkpoint | token accuracy | wrong tokens |
|---|---|---|
2-domain ckpt_011424 | 99.07% | 345 |
3-domain ckpt_015200 | 99.65% | 130 |
3-domain ckpt_015987 | 99.66% | 125 |
Read the think columns as an upper bound for this checkpoint: it was fit on
openseeker-think-sections, which derives from the same corpus, and its
train/held-out split lives in the training repo rather than here, so a span
drawn above may be one it trained on. The 2-domain numbers carry no such
caveat on think — it never saw reasoning at all — and those are the ones that
carry the finding.
ckpt_015200 and ckpt_015987 differ by a handful of tokens on both probes,
which is noise. Prefer ckpt_015200: validation loss is the better arbiter
than a probe this saturated.
| file | step | note |
|---|---|---|
ckpt_015200.pt | 15200 | lowest validation loss (ckpt_best) — use this one |
ckpt_015987.pt | 15987 | final (ckpt_last) |
config.yaml | required — the model is rebuilt from this, not from defaults |
Deliberately no ckpt_best.pt alias, though the 2-domain repository has one.
A downstream launcher may derive a run name from the checkpoint path, and one
that reads the iteration out of the filename gets nothing from a name with no
digits in it — two different checkpoints then produce the same run directory,
which is also the resume key. Copy the numbered file; keep the number.
config.yaml must sit beside the checkpoint you load; the loader reads it to
construct the model and refuses to fall back on defaults.
from models.vae_slot_encoder import PrefixVAESlotEncoder
encoder = PrefixVAESlotEncoder(
"…/ckpt_015200.pt", # config.yaml beside it
device=torch.device("cuda"),
)
latents = encoder.encode_slots(span_ids, span_lens) # [n_slots, 512], mu
mu is returned rather than a sample from (mu, logvar): the encoder is meant
to be frozen downstream, so sampling would inject noise nothing can adapt to and
would make a step irreproducible.
The config.yaml records the paths of the cluster it trained on; a loader should
map a dead absolute path to the equivalent hub id (Qwen/Qwen3-0.6B-Base,
Qwen/Qwen3-1.7B-Base) rather than editing the record of how it was trained.