all-MiniLM-L6-v2-genbais

all-MiniLM-L6-v2 + GENbAIs Bio Adapters

Bio-inspired adapters that improve beyond LoRA — discovered through intelligent neural architecture search over 50+ neuroscience-inspired mechanisms.

What is this?

This model is sentence-transformers/all-MiniLM-L6-v2 enhanced with bio-inspired adapter mechanisms discovered through the GENbAIs framework — General Efficient Neural bio-Adapter Intelligent Search.

The enhancement was distilled back into a clean SentenceTransformer, so no custom code is needed — load it like any other sentence-transformers model.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("lakinekaki/all-MiniLM-L6-v2-genbais")

sentences = [
    "The weather is lovely today.",
    "It's so sunny outside!",
    "He drove to the stadium.",
]

embeddings = model.encode(sentences)

# Compute similarity
from sentence_transformers.util import cos_sim
print(cos_sim(embeddings[0], embeddings[1]))  # High similarity
print(cos_sim(embeddings[0], embeddings[2]))  # Low similarity

How It Was Made

1. Bio-Adapter Search

We searched through ~1,000 configurations out of a ~10²² search space using Thompson sampling with Bayesian pruning. The search explored 50+ neuroscience-inspired mechanisms including predictive coding, lateral inhibition, Hebbian learning, dendritic computation, and more.

2. Stacked on Best LoRA

Bio adapters were stacked on top of the optimal LoRA configuration (found via grid search with 5 configs × 3 seeds). This proves the bio features provide additive improvement beyond state-of-the-art PEFT.

3. Distillation

The enhanced model (base + LoRA + bio adapters) was distilled into a clean SentenceTransformer via MSE + cosine embedding loss, producing a standard model with no custom dependencies.

Full Benchmark Results

Evaluated on 20 metrics across STS, pair classification, and clustering tasks.

Semantic Textual Similarity

Pair Classification

Clustering

Summary

• 17 wins, 3 losses across 20 metrics
• Average improvement: +0.0205 absolute
• Strongest gains on adversarial tasks — PAWS AP +14.66% suggests bio features capture genuine semantic structure beyond lexical overlap
• Broad improvement across STS, pair classification, AND clustering (not task-specific overfitting)
• Small regressions on biosses (-3.6%, tiny biomedical domain) and SNLI (-2.3%)

Key Insight

This is a hard-mode validation. all-MiniLM-L6-v2 is a 22M-parameter, 6-layer model that's already been distilled and heavily optimized by the sentence-transformers team — one of the toughest targets to improve. Getting meaningful gains here is like squeezing blood from a stone. Larger models (CLIP, LLaMA, Mistral) with more layers, more parameters, and more architectural redundancy offer significantly more room for bio-adapter improvement.

The largest improvements come on adversarial and challenging benchmarks (PAWS, STS13, STS14, MNLI) — exactly where standard fine-tuning tends to plateau. Bio-inspired mechanisms like lateral inhibition and predictive coding appear to capture deeper semantic relationships that pure gradient descent misses.

Technical Details

• Base model: sentence-transformers/all-MiniLM-L6-v2 (22M params, 6 layers)
• Bio mechanisms: 50+ neuroscience-inspired adapter types
• Search: ~1,000 experiments via Thompson sampling (out of ~10²² possible)
• Each adapter: adds up to ~1% of model parameters
• Distillation: MSE + cosine loss on 50K sentences from STS-B + AllNLI + Quora
• Evaluation time: 754.5s across all 20 benchmarks

Learn More

• 🌐 GENbAIs Website — full benchmark results and interactive pricing calculator
• 📄 Research Paper
• 💻 GitHub
• 💼 Enhancement Services — we enhance your models with pay-for-results pricing

Citation

@software{genbais2025,
  title={GENbAIs: General Efficient Neural bio-Adapter Intelligent Search},
  author={Kovacevic, Lazar},
  year={2025},
  url={https://genbais.com}
}

License

MIT