qwen-indic-v1

qwen-indic-v1

A multilingual text embedding model for the 22 scheduled Indic languages, fine-tuned from Qwen/Qwen3-Embedding-8B via a 3-stage LoRA training recipe with frozen-teacher preservation.

• Base model: Qwen/Qwen3-Embedding-8B
• Training: LoRA (r=64, alpha=128), merged into base for release
• Pooling: last-token, L2-normalized
• Embedding dimension: 4096
• Languages: 22 scheduled Indic languages
• License: Apache 2.0 (inherited from base model)

Evaluation

Evaluated on MTEB(Indic, v1) using the official mteb package. Overall task-averaged mean across 20 Indic tasks: 73.80.

Per-category results

Per-task results

Full per-task JSON results are in the MTEB results repository.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("LingoIITGN/qwen-indic-v1")

queries = [
    "भारत की राजधानी क्या है?",
    "চা কীভাবে বানানো হয়?",
]
documents = [
    "नई दिल्ली भारत की राजधानी है।",
    "গরম জলে চা পাতা ভিজিয়ে চা তৈরি করা হয়।",
]

query_embs = model.encode(queries, prompt_name="Retrieval")
doc_embs = model.encode(documents)

import numpy as np
sim = (query_embs / np.linalg.norm(query_embs, axis=1, keepdims=True)) @ \
      (doc_embs / np.linalg.norm(doc_embs, axis=1, keepdims=True)).T

For classification and STS, use prompt_name="Classification" or "STS" respectively; the model was trained with per-task-family instruction prefixes. See config_sentence_transformers.json for the full prompt list.

Training procedure

Fine-tuned via a 3-stage LoRA recipe with frozen full-base teacher preservation, adapted from the Harrier-OSS training methodology.

Each stage trains the LoRA student while a frozen copy of the base model serves as a preservation teacher via a relational distillation loss. The recipe uses a gentle contrastive weight and a dominant, rising teacher-preservation weight to prevent catastrophic forgetting of the base model's strong general-purpose capabilities while adding Indic-specific improvements.

Effective batch size: ~256 (micro=16, grad-accum=16, DDP across GPUs). Precision: bf16 mixed precision throughout.

Training data

All training data was drawn from train splits only — no test or validation splits from any dataset were used, to avoid contamination of MTEB evaluation tasks. Sources include:

• Retrieval/bitext (Stage 1): Samanantar (ai4bharat), plus curated retrieval pairs with mined hard negatives.
• Classification/NLI (Stage 2): IndicXNLI (entailment as positive, contradiction as hard negative), MASSIVE intent classification, sentiment data, and language-identification triplets constructed from parallel bitext with same-script hard negatives.
• Clustering (Stage 3): IndicGLUE news-genre topic clusters and additional clustering-format data.

All sources were audited to ensure use of training splits only. Data was sanitized to remove control characters and validated for JSONL round-tripping before training.

Model architecture

• Base: Qwen3-Embedding-8B (decoder-only transformer, 4096 hidden dim)
• Pooling: last-token (matches Qwen3 base convention)
• Normalization: L2, applied on the pooled vector
• LoRA targets: all-linear (auto-detected linear projections)
• LoRA rank: 64, alpha: 128, dropout: 0.05
• Merged: yes — the LoRA adapter is merged into the base weights for release

Instruction format

The model uses Qwen3's instruction convention on the query side:

Instruct: {task_description}\nQuery: {query_text}

Documents are encoded without an instruction prefix. Task-family prompt strings are stored in config_sentence_transformers.json and applied automatically by sentence-transformers when prompt_name is passed to encode().

Limitations

• The model inherits Qwen3-Embedding-8B's overall capabilities and biases; Indic-specific fine-tuning does not eliminate base-model behaviors.
• Coverage of very-low-resource Indic languages (Kashmiri, Manipuri, Bodo, Santali) is limited by training data availability; scores on these languages may be substantially lower than on higher-resource languages.
• Clustering performance, while improved over the base model, remains below the current state-of-the-art on MTEB(Indic). Future work will address this.
• 8B parameters — inference requires ~16GB VRAM in bf16.