ide-code-retrieval-qwen3-0.6b-inbatch

ide-code-retrieval-qwen3-0.6b-inbatch

A SentenceTransformer model fine-tuned from Qwen/Qwen3-Embedding-0.6B for IDE code retrieval -- mapping natural-language commit queries to relevant source code documents via dense vector similarity.

Note: This is an intermediate checkpoint at step 9,000 / 9,150 (98.4% through 3 epochs). Training loss is still decreasing, so a later checkpoint may perform better.

Model Description

This model encodes both short natural-language queries (commit messages, search queries) and longer code documents into a shared embedding space. Retrieval is performed by computing cosine similarity between the query embedding and candidate code embeddings.

• Base model: Qwen/Qwen3-Embedding-0.6B (0.6B parameters)
• Max sequence length: 1024 tokens
• Output dimensionality: 1024 (normalized)
• Similarity function: Cosine similarity

Training Details

Dataset

• Source: aysinghal/code-retrieval-training-dataset
• Total pairs: 5,072,176
• Train split: 4,818,567 pairs (95%)
• Eval split: 253,609 pairs (5%)
• Text strategy: truncate (max 4096 chars)
• Negatives: Explicit hard negatives from the dataset
• Pre-tokenized: Yes (token IDs stored on disk for zero-overhead data loading)

Loss Function

MultipleNegativesRankingLoss (InfoNCE) with explicit hard negatives. Each training example consists of an anchor (query), a positive (relevant code), and a hard negative (similar but irrelevant code). In-batch negatives provide additional contrast.

Hyperparameters

Hardware

• GPUs: 8x NVIDIA L40S
• Total training steps: 9,150 (3 epochs)

Training Progress (at checkpoint step 9,000)

• Training loss: 1.1880 (step 50) → 0.1628 (step 9000)
• Best eval loss: 0.0443 (step 7,528)
• Progress: 9,000 / 9,150 steps (98.4%)

Evaluation Results

<details> <summary>Full training loss history (click to expand)</summary> </details>

Usage

Loading the Model

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("aysinghal/ide-code-retrieval-qwen3-0.6b-inbatch")

Computing Embeddings

queries = [
    "fix null pointer exception in user authentication",
    "add retry logic to API client",
]
code_docs = [
    "def authenticate(user):\n    if user is None:\n        raise ValueError...",
    "class APIClient:\n    def request(self, url, retries=3):\n        ...",
]

query_embeddings = model.encode(queries)
code_embeddings = model.encode(code_docs)

# Compute cosine similarities
from sentence_transformers.util import cos_sim
similarities = cos_sim(query_embeddings, code_embeddings)
print(similarities)

Intended Use

• Primary use case: Retrieving relevant code files/functions given a natural-language query (commit message, bug description, feature request)
• Search pipeline: Encode a corpus of code documents offline, then at query time encode the query and find nearest neighbors via cosine similarity

Limitations

• This is an early checkpoint (98.4% through training). The loss curve is still decreasing, so later checkpoints will likely perform better.
• Trained on a specific code retrieval dataset; may not generalize to all programming languages or query styles without further fine-tuning.
• Max context is 1024 tokens -- very long files are truncated.

Citation

If you use this model, please cite the base model:

@article{qwen3embedding,
  title={Qwen3-Embedding},
  author={Qwen Team},
  year={2025}
}