Qwen3-Embedding-4B-bnb4

Qwen3-Embedding-4B-bnb4

This is a 4-bit quantized version of Qwen/Qwen3-Embedding-4B using BitsAndBytes NF4 quantization.

Model Details

• Base Model: Qwen/Qwen3-Embedding-4B
• Quantization Method: BitsAndBytes 4-bit (NF4)
• Quantization Configuration:
  • bnb_4bit_quant_type: "nf4"
  • bnb_4bit_use_double_quant: True
  • bnb_4bit_compute_dtype: float16
• Model Size: ~2.5GB (down from ~8GB original)
• Use Case: Text embedding for semantic search, retrieval, and similarity tasks

Performance

Tested on Vietnamese corpus (100 documents) with 20 queries:

• Accuracy@1: 90%
• MRR: 94.17%
• Recall@3: 100%
• Recall@5: 100%

Usage

With Transformers + BitsAndBytes

from transformers import AutoTokenizer, AutoModel
import torch

model_name = "dinhhungitsoft/Qwen3-Embedding-4B-bnb4"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModel.from_pretrained(
    model_name,
    trust_remote_code=True,
    device_map="auto"
)

def embed(texts):
    inputs = tokenizer(texts, padding=True, truncation=True, return_tensors="pt", max_length=512)
    inputs = {k: v.to(model.device) for k, v in inputs.items()}
    
    with torch.no_grad():
        outputs = model(**inputs)
        embeddings = outputs.last_hidden_state[:, 0, :]  # CLS token
    
    # Normalize embeddings
    embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    return embeddings.cpu().numpy()

texts = ["Hà Nội là thủ đô của Việt Nam", "Paris is the capital of France"]
embeddings = embed(texts)
print(embeddings.shape)  # (2, embedding_dim)

With vLLM (Recommended for serving)

docker run -d \
  --gpus all \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  -p 8000:8000 \
  --ipc=host \
  vllm/vllm-openai:latest \
  --model dinhhungitsoft/Qwen3-Embedding-4B-bnb4 \
  --trust-remote-code \
  --gpu-memory-utilization 0.6 \
  --max-model-len 4096

Then query via OpenAI-compatible API:

import requests

response = requests.post(
    "http://localhost:8000/v1/embeddings",
    json={
        "model": "dinhhungitsoft/Qwen3-Embedding-4B-bnb4",
        "input": ["Your text here"]
    }
)
embeddings = response.json()["data"][0]["embedding"]

Languages

Optimized for:

• English
• Chinese
• Vietnamese
• And other languages supported by Qwen3

Hardware Requirements

• GPU Memory: ~3-4GB VRAM (vs ~8GB for full precision)
• Recommended: NVIDIA GPU with CUDA support
• Tested on: RTX 3050 8GB

Limitations

• Quantization may result in minor accuracy loss compared to full precision
• Requires BitsAndBytes library for loading
• Best performance with NVIDIA GPUs

Citation

If you use this model, please cite the original Qwen3 model:

@article{qwen3,
  title={Qwen3 Technical Report},
  author={Qwen Team},
  year={2024}
}

License

Apache 2.0 (inherited from base model)