Downloads · 30 days
0
bpe_glove_512_lora_v1_ffn
jsanzolac/bpe_glove_512_lora_v1_ffn
bpe_glove_512_lora_v1_ffn is a machine learning model from jsanzolac. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Warm-start from jsanzolac/bpeglove512lorav1/rank512 plus a per-token FFN inserted between the GloVe-attention output and the alpha-pool collapse.
Access
Public
Updated May 17, 2026
Repo size
4 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.txt439 MB · 70%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- May 16, 2026
- Updated
- May 17, 2026
- SHA
- 40f6b9e9
Base models
Model card
From the Hugging Face model README
bpe_glove_512_lora_v1_ffn
Warm-start from jsanzolac/bpe_glove_512_lora_v1/rank_512 plus a per-token FFN inserted
between the GloVe-attention output and the alpha-pool collapse.
Trainable: A, B, FFN. Frozen: E, teacher.
Loss: λ_c·InfoNCE + λ_D·‖ρ_T − ρ_S‖²_F with λ_c=1.0, λ_D=0.1.
Density is computed on the post-FFN per-token states; InfoNCE is on the alpha-pooled sentence vector.
Files:
rank_512/checkpoint_final.pt— A + B + FFN state dict (E is non-persistent; re-inject fromjsanzolac/bpe_glove_512/vectors.txt).rank_512/config.json— full hyperparameters.rank_512/vectors_drifted.txt—E + B(A(·))per vocab row, GloVe text format. Note: this captures only the static drifted embedding lookup, not the FFN's effect (which is contextual). To use the model end-to-end, instantiateDriftingGloVeStudentFFNand run forward.rank_512/train_log.jsonl— per-step metrics.
- License
- apache-2.0
- Languages
- en
- Created
- May 16, 2026
- Updated
- May 17, 2026