jina-v5-h256-lora-clf-merged

Jina v5 H256 Distilled + Classification LoRA (Merged)

Compressed Jina v5 embedding model with a universal classification LoRA merged into the weights. The LoRA was trained via multi-task classification on all 8 MTEB Classification tasks, then permanently merged into the base model for zero-overhead inference.

Model Details

Merged LoRA Specification

MTEB Evaluation Results

Overall Average (25 tasks): 50.67%

Note: This model is optimized for classification only. The LoRA merge significantly improves classification but degrades STS performance. Use the base model gomyk/jina-v5-h256-distilled-conv if you need STS or general-purpose embeddings.

Classification (+9.41%p improvement)

STS (-31.60%p regression)

Clustering (-4.09%p regression)

Quick Start

from transformers import AutoModel, AutoTokenizer
import torch

model = AutoModel.from_pretrained(
    "gomyk/jina-v5-h256-lora-clf-merged", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(
    "gomyk/jina-v5-h256-lora-clf-merged", trust_remote_code=True)

texts = ["This movie was amazing!", "I need to transfer money"]
inputs = tokenizer(texts, padding=True, truncation=True,
                   max_length=128, return_tensors="pt")

with torch.no_grad():
    outputs = model(**inputs)
    hidden = outputs.last_hidden_state
    mask = inputs["attention_mask"].unsqueeze(-1).float()
    embeddings = (hidden * mask).sum(1) / mask.sum(1).clamp(min=1e-9)

print(embeddings.shape)  # [2, 256]

# Cosine similarity
from torch.nn.functional import cosine_similarity
sim = cosine_similarity(embeddings[0].unsqueeze(0), embeddings[1].unsqueeze(0))
print(f"Similarity: {sim.item():.4f}")

Training Pipeline

1. Compression (from base model repo):
   jinaai/jina-embeddings-v5-text-nano (12L/768d/128K vocab/239M)
   → Layer pruning (12→6)
   → Hidden dim PCA (768→256)
   → Vocab pruning with BPE backtracking (128K→42K)
   → Knowledge distillation (MSE + Cosine loss)
   = gomyk/jina-v5-h256-distilled-conv (6L/256d/42K vocab/16.9M)

2. Classification LoRA (this model):
   → LoRA (rank=8) on all attention projections
   → Multi-task training on 8 MTEB Classification tasks
   → Merge LoRA into base weights
   = This model (same size, better classification)

License

Apache 2.0