word_glove_300_lora_qwen3_optstate

word_glove_300_lora_qwen3_optstate

DriftingGloVeStudent rank=300 over a frozen 300-d cl100k BPE-GloVe, distilled from Qwen/Qwen3-Embedding-8B (MRL-truncated to the first 300 dims, then re-L2-normalized).

Same architecture, loss, and hyperparameters as jsanzolac/bpe_glove_300_lora_r300_qwen3_optstate (the cl100k path). The two differences:

• frozen E is Stanford's pretrained glove.2024.wikigiga.300d (1.2M words, uncased, trained on 11.9B Wikipedia+Gigaword tokens) instead of our self-trained 82K-row cl100k BPE GloVe;
• the packed corpus was re-tokenized with Stanford Stanza PTBTokenizer + lowercase to match Stanford's GloVe-training preprocessing.

Loss: cross_entropy(v @ v_T^T / τ) + λ_MSE · MSE(v, v_T) with τ = 0.05, λ_MSE = 1.0.
Training: 150,000 steps × batch 256 × lr 0.0005 (cosine → 1e-05).

Files under rank_300/:

• checkpoint_final.pt — {step, config, model_state_dict (A+B), opt_state_dict, sched_state_dict}. E is non-persistent (re-inject fromjsanzolac/glove_2024_wikigiga_300d/glove_vectors.bin)
• config.json
• vectors_drifted.txt / .parquet — E + B(A(·)) per vocab row.
• train_log.jsonl