lettuce-emb-512d-v1

lettuce-emb-512d-v1 — 512d Distilled Embedding Model (ONNX)

Zeolit/lettuce-emb-512d-v1 is a custom 512-dimensional sentence embedding model, trained by distillation from BAAI/bge-m3 and specialized for conversational / roleplay memory use cases.

It's built on top of DistilBERT (512-token context) with a 512-dim projection head and exported to ONNX for fast, portable inference (desktop + mobile).

This model is designed to power the memory system of LettuceAI (a roleplay-focused LLM client), but can be used as a general-purpose semantic embedding model.

---

🏆 Performance Highlights

Retrieval Performance (ArguAna)

Overall: #1 among sub-100M models

What this means:

• Beats every "tiny/small" embedding model (MiniLM, gte-small, e5-small, Snowflake-xs)
• Only surpassed by 100M+ models (gte-base: 0.46, bge-base: 0.58, bge-small: 0.51, bge-m3: 0.68)
• Exceptional performance for 66M parameters — punching way above weight class

Semantic Similarity Performance (STS Benchmarks)

Strengths:

• STS13 (0.649): Matches/beats models 2-3x larger
• STS15 (0.676): Competitive with top small models (e5-small: 0.78, gte-small: 0.77)
• Balanced Pearson scores (0.616-0.644) show consistent correlation quality

Context:

• Mid-pack among 10-100M models for semantic similarity
• Trade-off: optimized for retrieval over perfect STS scores
• This is a deliberate design choice for practical applications where finding the right document matters more than perfect similarity scoring

Combined Assessment

✅ Top-tier retrieval — Best in class for 66M parameters
✅ Competitive semantic similarity — Solid for clustering, deduplication
✅ Efficient — Exceptional size-to-performance ratio
✅ Practical — Optimized for real-world RAG and search use cases

---

Model Summary

• Base encoder: distilbert-base-uncased
• Parameters: 66M
• Teacher model: BAAI/bge-m3
• Embedding dimension: 512
• Max sequence length: 512 tokens
• Format: ONNX (FP32)
• Pooling: Mean pooling over last hidden state + L2 normalization
• Domain: General English text with extra focus on dialogue / roleplay data

The model was trained to:

• Match the similarity structure of BGE-M3 in embedding space (batch-level KD)
• Pull semantically related texts together (contrastive loss)
• Capture nuances in conversational, character-driven, and roleplay-style text

Training & Distillation

High-level training setup:

• Student: DistilBERT with a 512-dim projection head on top of the pooled output
• Teacher: BGE-M3 (1024-dim multilingual embedding model)
• Losses:
  • Geometry distillation: MSE between student and teacher batch similarity matrices
  • Contrastive loss: 1 - cosine(student_emb_1, student_emb_2) for positive pairs
• Batch size: 16
• Max seq length: 512
• Optimizer: AdamW, lr = 2e-5
• Epochs: 2 (over combined dataset)

Datasets

The student was trained on a mixture of:

1. Custom roleplay corpus (JSONL)
   Consecutive lines from RP-style dialogue logs were turned into positive pairs:

   • ("utterance_i", "utterance_{i+1}")

2. stsb_multi_mt (en)
   Standard semantic similarity dataset, used as additional general-purpose supervision.

Together, these teach the model:

• General sentence-level semantic similarity
• Dialogue turns & conversational flow
• Roleplay-specific semantics (emotional nuance, relationships, "scene continuity")

Usage (Python + ONNX Runtime)

1. Install dependencies

pip install onnxruntime transformers

2. Basic embedding example

import onnxruntime as ort
import numpy as np
from transformers import AutoTokenizer

model_id = "Zeolit/lettuce-emb-512d-v1"

tokenizer = AutoTokenizer.from_pretrained(model_id)
session = ort.InferenceSession("model.onnx", providers=["CPUExecutionProvider"])

MAX_SEQ_LEN = 512

def embed(texts):
    if isinstance(texts, str):
        texts = [texts]

    enc = tokenizer(
        texts,
        padding=True,
        truncation=True,
        max_length=MAX_SEQ_LEN,
        return_tensors="np",
    )

    outputs = session.run(
        ["sentence_embedding"],
        {
            "input_ids": enc["input_ids"],
            "attention_mask": enc["attention_mask"],
        },
    )[0]
    # outputs: (batch_size, 512)
    return outputs

def cosine(a, b):
    return float(np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b)))

sents = [
    "The hero shielded his companion from the arrow.",
    "He risked everything to save her life.",
    "The market is bustling with traders today.",
]

emb = embed(sents)
print(emb.shape)  # (3, 512)
print("cos(0,1) =", cosine(emb[0], emb[1]))
print("cos(0,2) =", cosine(emb[0], emb[2]))

---

🧩 Intended Use

Ideal for:

• Conversational / roleplay memory retrieval (primary use case)
• Semantic search and document retrieval
• RAG (Retrieval-Augmented Generation) systems
• Clustering and duplicate detection
• Lightweight, local embedding on:
  • Desktop clients (Tauri, Electron, etc.)
  • Mobile clients (via ONNX Runtime)

Strengths:

• Excellent retrieval performance — finds relevant documents with high precision
• Efficient size — 66M parameters, fast inference
• Long context — 512 tokens captures more information than typical small models

Trade-offs:

• Optimized more for retrieval than perfect semantic similarity scoring
• STS performance is solid but not best-in-class compared to larger models
• This is a deliberate design choice for practical applications where finding the right document matters more than perfect similarity scores

Non-goals:

• High-stakes safety-critical applications
• Multilingual performance on par with BGE-M3 (this model is primarily English-focused)
• Extremely large-scale retrieval across millions of vectors (consider larger models or specialized systems)

---

Limitations & Bias

• The model was trained on English text and may not generalize as well to other languages.
• Roleplay-style corpora may contain biased, informal, or emotionally charged content. The embeddings may reflect those patterns.
• The model does not perform any safety filtering; it simply encodes text into vectors.
• Weaker on certain STS tasks (STS12, STS14) that favor heavily distilled or contrastive models.

Always evaluate the model in your specific application context before using it in production.

---

Benchmarks

Detailed Retrieval Metrics (ArguAna)

Core Metrics:

Recall Performance:

Ranking Quality:

Comparison vs. Similar Models

Gap Analysis:

• vs. sub-100M models: +10-20% improvement in nDCG@10
• vs. 100M models: Only 14% behind gte-base despite being 34% smaller
• vs. teacher (bge-m3): 42% behind, but at 11.6% of the size

Semantic Similarity Details (STS)

STS12 (2012 Benchmark):

STS13 (2013 Benchmark):

STS14 (2014 Benchmark):

STS15 (2015 Benchmark):

Comparison vs. Small Models (STS)

Performance Notes:

• STS13 strength: Essentially matches MiniLM-L12 (both ~0.65)
• STS15 strength: Close to top performers (0.676 vs 0.75-0.78 range)
• STS12/14 trade-off: 10-15% behind leaders — room for improvement
• Overall: Solid mid-pack performance, optimized for retrieval rather than peak STS scores

---

License

This model is released under the Apache-2.0 license.

It is derived from:

• distilbert-base-uncased (Apache-2.0) as the student backbone
• BAAI/bge-m3 (MIT) as the distillation teacher

Acknowledgements

• HuggingFace Transformers
• ONNX Runtime
• BAAI / BGE
• The LettuceAI project for motivating a roleplay-focused embedding model