qwenar-0.6b

brivangl/qwenar-0.6b

A sentence is compressed into one 1024-dimensional vector, and that vector alone is decoded back into the sentence.

At its best checkpoint this model reconstructs 30 of 32 held-out probe sentences character-for-character from a single float vector.

The idea is SONAR's and the Large Concept Model's: a sentence embedding lossless enough to decode, so the vector can stand in for the text. The construction is different — instead of training a seq2seq model from scratch, qwenar connects two off-the-shelf open checkpoints with a single learned linear bridge and adapts both with DoRA. The whole learned connector is about 30 lines of code.

Code, training pipeline and full evaluation: https://github.com/IvanDrokin/QwenAR

[!IMPORTANT] Requires transformers>=5.15, and older versions fail silently. This model was trained on 5.15.0. Under transformers 4.x it raises nothing and produces fluent, plausible text that is simply wrong: teacher-forced cross-entropy goes from 0.0052 to 3.2614 and next-token accuracy from 100% to 56%.

Usage

from qwenar.pipeline import Qwenar          # pip install qwenar

qw = Qwenar.from_pretrained("brivangl/qwenar-0.6b")

vecs = qw.encode(["Serum ferritin remained within the reference range."])
print(vecs.shape)                            # torch.Size([1, 1024])
print(qw.decode(vecs))                       # back to the sentence
print(qw.roundtrip(["..."]))                 # both at once

Or as a plain transformers model, with no extra dependency:

from transformers import AutoModel, AutoTokenizer

model = AutoModel.from_pretrained("brivangl/qwenar-0.6b", trust_remote_code=True)
enc_tok = AutoTokenizer.from_pretrained("brivangl/qwenar-0.6b", subfolder="encoder_tokenizer")
dec_tok = AutoTokenizer.from_pretrained("brivangl/qwenar-0.6b")

batch = enc_tok(["Serum ferritin remained within the reference range."],
                return_tensors="pt", padding=True)
emb = model.encode(batch["input_ids"], batch["attention_mask"])     # (1, 1024)
ids = model.generate_from_embeddings(emb, max_new_tokens=64, do_sample=False)
print(dec_tok.batch_decode(ids, skip_special_tokens=True))

Two tokenizers, because the encoder and the decoder come from different checkpoints. The decoder tokenizer sits at the repository root so plain AutoTokenizer.from_pretrained(repo) gives you the one generation needs; the encoder tokenizer is in encoder_tokenizer/. Decoder targets must be right-padded — Qwenar enforces this.

Architecture

sentence ──► perplexity-ai/pplx-embed-v1-0.6b
             Qwen3 with the causal mask DISABLED → bidirectional
             mean-pool over non-pad tokens; no tanh, no INT8
                              │
                       vec (1024,)             ◄── this is the artifact
                              │
             EmbedToPrefix:  Linear(1024 → K·d_model) → GELU
                             → view(B, K, d_model) → RMSNorm      K = 2
                              │
                     prefix (B, 2, d_model)
                              │
             Qwen/Qwen3-0.6B-Base + DoRA(r=32, α=64) on q,k,v,o,gate,up,down
                              │
                  teacher-forced next-token cross-entropy
                  labels = -100 on the prefix and on padding

1194M parameters total. The encoder is adapted too (DoRA r=16, at a 10× lower learning rate), not frozen.

Training data

Sentence-level English, from a 22.83B-token corpus. The published run saw about 13.0B supervised tokens — roughly 57% of one epoch; it never completed a pass over the data.

Over half the training signal is PubMed abstracts. These proportions were not a design target — they are what the source directories happened to contain. Sentences were segmented with SaT (sat-3l-sm) and capped at 96 tokens, so the model has never seen a longer input.

The corpus itself is not released, and the dataset revisions were not recorded, so it cannot be rebuilt byte-for-byte. Full provenance, including what is missing, is in DATASET.md.

Training procedure

ReLoRA periodically merges the adapters into the backbone, zeroes lora_B, clears the optimizer state and restarts the schedule. One consequence matters for anyone picking a checkpoint: validation loss spikes after each merge, so the last checkpoint is not the best one. These weights are step 700,000, taken at the end of a converged cycle.

Evaluation

Reconstruction cross-entropy on held-out sentences, and the batching invariants.

Batch invariance is worth stating explicitly: encoding or decoding a sentence alone gives the same result as doing it inside a padded batch, to float precision. Batching is a performance choice here, not a behavioural one.

The failure mode is consistent and worth knowing: the sentence frame survives while dense numerals and rare nomenclature scramble.

src: Musculoskeletal injury causes pain and when chronic can affect mental
     health, employment and quality of life.
gen: Musculoskeletal injury causes pain and when chronic can affect mental
     health, employment and quality of life.

src: arbazone 14, 1,3-thiazolines 15a-c and 4-thiazolidinone 16 exhibited
     significant COX-2 inhibition ...
gen: arbazone 14, 13, 1-thiazolones4a-b and 15-cidine 2-thiazinoleic strain 60
     exhibited significant ...

1024 floats carry a sentence's structure and content, but not arbitrary high-entropy strings.

Evaluation on general-domain text (SlimPajama, added 2026-09)

Out-of-domain evaluation added after the SlimPajama models were trained: the same battery that scores them, run on this checkpoint unchanged.

By sentence length — reconstruction degrades past ~48 tokens, and the 4B encoder holds up markedly longer:

Prose vs. code (rule-based detector; code is 0.7% of the split):

The failure mode is the published one: the sentence frame survives while dense code syntax, identifiers, URLs and formula symbols scramble.

src: def flatten(xs): return [y for x in xs for y in (flatten(x) if isinstance(x, list) else [x])]
gen: def flatten(xs): return [y for x in xs for y in (flatten(x) if isinstance(x, list) else [x])]

src: fn parse(input: &str) -> Result<Vec<u32>, ParseIntError> { input.split(',').map(str::trim).map(str::parse).collect() }
gen: fn parse(input) : result &<&Str<Vec<UInt32>, parseErrorIndex> { inp.split(',|map').str.parse((map).str.parse()::trim())
     (Levenshtein 58, CER 49.15%)

src: SELECT u.id, COUNT(o.id) AS orders FROM users u LEFT JOIN orders o ON o.user_id = u.id WHERE u.active GROUP BY u.id HAVING COUNT(o.id) > 3;
gen: SELECT u.id, COUNT(o.id) AS orders FROM users u JOIN LEFT orders o ON u.o_id = u.user WHERE u.id AS ACTIVE GROUP BY u.id hOrderBy u.id; > 0)
     (Levenshtein 46, CER 33.09%)

src: func main() { ch := make(chan int, 8); go func() { ch <- 42 }(); fmt.Println(<-ch) }
gen: func main() { chan := ch.make, int8} { go func 4(); } ch := 23() <- fmt.Println; (ca+)
     (Levenshtein 34, CER 40.48%)

Comparison with the SlimPajama-trained successors

All numbers on the same held-out SlimPajama split (200,000 sentences), same greedy decoding, bf16. CE / tok acc: teacher-forced over the whole split. exact / CER / WER / cos: free-running reconstruction of 8,000 sentences — CER = character Levenshtein / source length, WER = word Levenshtein / source words, cos = cosine between encoder embeddings of source and reconstruction. probes: exact reconstructions of 32 fixed stress sentences (long sentences, code, numbers, URLs).

¹ this model; the successors (brivangl/qwenar-0.6b-montevideo, brivangl/qwenar-4b-montevideo) were trained on the 9.2B-sentence SlimPajama corpus and are evaluated in-domain here. For general web text prefer them; this checkpoint remains the stronger choice only for biomedical text, on which the successors were not evaluated.

Limitations

Read this before drawing conclusions from the numbers above.

• Reconstruction is the only thing measured. There is no retrieval, STS, clustering, or downstream evaluation anywhere in this project. The model was trained purely to reconstruct, with no contrastive objective, so the embedding space is optimised to be decodable, not to be semantically well-shaped. Do not assume these vectors are good sentence embeddings for similarity tasks — nothing here tests that.
• Heavily biomedical. ~54% of training tokens are PubMed abstracts. Expect worse reconstruction on general text than the headline number suggests.
• English only. SONAR's defining property is a shared multilingual space; this model has no multilingual training or evaluation whatsoever.
• Sentences only, ≤96 tokens, by construction of the training data. Behaviour on longer inputs is untested.
• One run, no ablations. Nothing isolates the contribution of K=2, DoRA vs LoRA, ReLoRA vs a single cycle, or an unfrozen vs frozen encoder. These are the hyperparameters that were used, not ones shown to be necessary.
• Less than one epoch. The run saw ~57% of the corpus.
• Trained on web and biomedical text, so it reproduces whatever biases and inaccuracies those carry. It is a reconstruction model: it will faithfully re-emit harmful or false input text.

License and attribution

Apache-2.0. Built from perplexity-ai/pplx-embed-v1-0.6b (MIT) and Qwen/Qwen3-0.6B-Base (Apache-2.0); the bidirectional encoder code is derived from the former. Third-party code and model licenses: THIRD_PARTY.md.

Implements ideas from SONAR, Large Concept Models, ReLoRA, DoRA and LoRA+.

Citation

@software{qwenar,
  author = {Drokin, Ivan},
  title  = {qwenar: a sentence autoencoder built from open checkpoints},
  year   = {2026},
  url    = {https://github.com/IvanDrokin/QwenAR}
}