Nano-Em1-0.6B-v2

Nano-Em1-0.6B-v2

Nano-Em1-0.6B-v2 is a 0.6B-parameter text embedding model built on a bidirectional-attention variant of Qwen3-0.6B, developed by KiteFish AI. It produces general-purpose embeddings for retrieval, classification, clustering, semantic similarity, and pair classification, using task-specific instruction prefixes.

Model description

• Base model: Qwen3-0.6B
• Attention: bidirectional (the causal mask is replaced with full bidirectional attention at every layer, implemented directly in the model's code so it holds under any standard load — see Architecture)
• Pooling: mean pooling over token embeddings
• Embedding dimension: 1024
• Max sequence length: 512 tokens

Architecture

Decoder-only language models use causal (left-to-right) attention by default, which limits their quality as fixed-representation encoders. Nano-Em1 removes the causal mask so every token attends to the full input in both directions. This is implemented in the model's own code (modeling_nano.py, loaded via trust_remote_code=True) rather than applied by the caller at inference time, so it's preserved by any standard AutoModel.from_pretrained or SentenceTransformer load.

Usage

Sentence-Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer(
    "KiteFishAI/Nano-Em1-0.6B-v2",
    trust_remote_code=True,
)

query_instruction = "Instruct: Given a query, retrieve documents that answer the query\nQuery: "

queries = [query_instruction + "How do I reset my password?"]
docs = ["To reset your password, go to Settings > Security and click 'Reset Password'."]

query_emb = model.encode(queries, normalize_embeddings=True)
doc_emb = model.encode(docs, normalize_embeddings=True)

similarity = query_emb @ doc_emb.T
print(similarity)

Transformers

import torch
from transformers import AutoModel, AutoTokenizer

model_id = "KiteFishAI/Nano-Em1-0.6B-v2"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModel.from_pretrained(model_id, trust_remote_code=True).eval()

def embed(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
    with torch.no_grad():
        out = model(**inputs).last_hidden_state
    mask = inputs["attention_mask"].unsqueeze(-1).float()
    pooled = (out * mask).sum(1) / mask.sum(1).clamp(min=1e-9)
    return torch.nn.functional.normalize(pooled, dim=-1)

embs = embed(["example sentence one", "example sentence two"])

Instructions

Queries — and passages, for symmetric tasks — should be prefixed:

Instruct: {task instruction}
Query: {text}

For asymmetric tasks (retrieval, reranking), only the query takes the instruction prefix — passages/documents are encoded as-is.

Limitations

• English-focused; not evaluated on non-English tasks.
• Requires trust_remote_code=True to load the custom bidirectional architecture.

Citation

@misc{nano-em1-2026,
  title  = {Nano-Em1-0.6B-v2},
  author = {KiteFish AI},
  year   = {2026},
  url    = {https://huggingface.co/KiteFishAI/Nano-Em1-0.6B-v2}
}