Fluffy-LoRA

Fluffy-LoRA 🐶🐶🐶

One body, three heads. Text, image, audio — one embedding space.

Named for Fluffy, Hagrid's three-headed dog (Cerberus's better-tempered nephew). Each head is an input modality; the body is a single shared embedding space.

No bolted-on encoders — no CLIP/CLAP/Whisper glued together with projectors. google/gemma-4-12b-it natively ingests all three modalities into one backbone; we QLoRA the language tower (r=8, α=16, ~32.8M params), freeze the native towers, and pool the last token into one L2-normed embedding. Trained with symmetric InfoNCE (τ=0.02) against frozen, ratcheted retrieval evals.

The honest scoreboard and all code live in the GitHub repo: → https://github.com/SEBK4C/Fluffy-LoRA

Published adapters

fluffy-text-v0/ — alpha artifact from the abandoned v1 text-only run

This adapter is published as a research artifact, not because it passed the ratchet — the ratchet rejected every checkpoint of the run it came from. Full disclosure:

• What it is: LoRA (r=8, α=16, q/k/v/o/gate/up/down projections, ~32.8M params) on the language tower of gemma-4-12b-it, trained with symmetric InfoNCE (τ=0.02, in-batch negatives, last-token pooling, L2 norm, max_length 512) on 40,941 fully self-synthetic text cards (no scraped or third-party data — rights-clean).
• The run was abandoned: planned as 14 days, stopped by operator decision at ~0.4 days wall time, step 1449 of a schedule sized for 200,000 steps (~56k were actually achievable in the window). The learning rate never left warmup (2.5e-5 of the 1e-4 peak at stop); the cosine schedule never annealed.
• Training loss fell ~5.5× (9.36 → ~1.7) …
• … but retrieval on our frozen eval did not move: R@1 0.010 vs 0.008 base-model baseline (≈ random) on the frozen G0 card-retrieval eval, n_pool = 1500. Loss-eval decoupling is the run's headline empirical result.
• Post-mortem found the root cause: gemma's tokenizer pads left, and the training/eval code pooled at attention_mask.sum(1)-1 (a right-padding assumption) — so every sequence that wasn't the longest in its batch was pooled at a padding position. Two unrelated texts pooled this way sit at cos 0.96 (their true last-token embeddings: cos 0.75). The adapter was trained on a mostly-collapsed embedding function.

Benchmarks (mteb 2.18.0, identical harness for all contenders)

nDCG@10 for retrieval, Spearman for STS. "fixed pool" = corrected last-token pooling (h[:, -1] under left padding). Reference = Qwen3-Embedding-8B with its own card protocol — it reproduces its published MTEB scores on this harness, which validates the pipeline.

Two honest take-aways: (1) the adapter moved no external metric — train it as we did and you learn the training data's domain, not transferable embedding geometry; (2) raw decoder last-token (or mean) embeddings of gemma-4-12b-it are near-unusable for retrieval without contrastive adaptation — the reference model's scores on the same harness show what a properly-trained embedder does. Full analysis and raw results: LEARNINGS-V1.md.

Treat it as a checkpoint of scientific interest (what ~1.4k steps of contrastive warmup on a broken pooling fn does to a decoder LM's embedding geometry), not as a useful embedding model.

Usage

import torch, torch.nn.functional as F
from transformers import AutoModel, AutoTokenizer
from peft import PeftModel

base = AutoModel.from_pretrained("google/gemma-4-12b-it", torch_dtype=torch.bfloat16)
if hasattr(base, "language_model"):
    base = base.language_model          # text tower only, as trained
model = PeftModel.from_pretrained(base, "SEBK4C/Fluffy-LoRA", subfolder="fluffy-text-v0").eval()
tok = AutoTokenizer.from_pretrained("google/gemma-4-12b-it")

enc = tok(["a query"], padding=True, truncation=True, max_length=512, return_tensors="pt")
h = model(**enc).last_hidden_state
idx = enc["attention_mask"].sum(1) - 1                       # last real token
emb = F.normalize(h[torch.arange(h.shape[0]), idx].float(), dim=-1)

Status: raw alpha — building in public

v1 (text-only) is over; v2 (three-lane multimodal) is being built in the open in the GitHub repo. Future checkpoints land here only when they beat the frozen evals by more than ε — the ratchet rejects everything else.