minilm-code-jina-distilled-mlx

MiniLM Code — Jina-distilled MLX adapter

An Apple-Silicon-native code-search embedding adapter for mlx-community/all-MiniLM-L6-v2-bf16. It was trained on 94,987 CodeSearchNet query/function pairs, distilled from jinaai/jina-embeddings-v2-base-code, then continued for 1,500 steps with globally mined hard negatives.

This repository contains a LoRA adapter rather than duplicated base weights. The loader downloads the base checkpoint and applies adapters.npz.

Usage on Apple Silicon

python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python loader.py "parse JSON without throwing an exception"

Or from Python:

from loader import load_model

model = load_model()
vectors = model.encode([
    "parse JSON without throwing an exception",
    "def try_parse_json(value): ...",
])
similarity = float((vectors[0] * vectors[1]).sum().item())

After downloading this repository with huggingface_hub.snapshot_download, pass its directory to load_model(path).

Evaluation

Paired retrieval against the complete candidate pool:

CodeXGLUE AdvTest contains 19,210 Python query/code pairs and replaces function names and variables in its test code. The development corpus contained six languages: Python, Java, JavaScript, Go, PHP, and Ruby.

Training

• Student: mlx-community/all-MiniLM-L6-v2-bf16
• Teacher: jinaai/jina-embeddings-v2-base-code
• Sequence length: 256
• First cycle: 3,000 steps, batch 32, learning rate 2e-5
• Hard-negative continuation: 1,500 steps, batch 32, learning rate 1e-5
• Objective: 70% supervised InfoNCE + 30% teacher similarity-distribution loss
• Trainable parameters: rank-16 LoRA on query/key/value attention projections
• Pooling: masked mean pooling followed by L2 normalization

Limitations

• MLX inference requires Apple Silicon.
• The adapter inherits MiniLM's 256-token configured limit in this package.
• Training data is dominated by documented open-source functions rather than real production search queries.
• Performance outside the six CodeSearchNet languages has not been measured.
• Review the original dataset and dependency licenses for your use case.