Qwen3-Embedding-0.6

Qwen3 Embedding 0.6B — ONNX INT8 (AVX2)

This is an INT8 quantized ONNX version of the original model
Qwen3-Embedding-0.6B.

The model was quantized for CPU inference using ONNX Runtime.

Quantization

Created with: optimum

Quantization type:

• INT8
• AVX2 optimized
• Per-channel quantization

This version significantly reduces:

• model size
• memory usage
• CPU latency

while maintaining strong embedding quality.

Hardware

Optimized for:

• CPUs with AVX2 support

Tested with:

• ONNX Runtime CPUExecutionProvider

Usage

Example using ONNX Runtime:

import os
import numpy as np
import onnxruntime as ort
from transformers import AutoTokenizer

model_dir = "Qwen3-Embedding-0.6B_int8"
model_path = os.path.join(model_dir, "model_quantized.onnx")

session = ort.InferenceSession(
    model_path,
    providers=["CPUExecutionProvider"]
)

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-Embedding-0.6B")

def embed_texts(texts):
    inputs = tokenizer(
        texts,
        padding=True,
        truncation=True,
        return_tensors="np"
    )
    _, embeddings = session.run(None, dict(inputs))
    return embeddings


# Example documents
documents = [
    "A beginner guide to brewing coffee using a pour-over method.",
    "How regular exercise improves cardiovascular health.",
    "Introduction to graph databases and how they store relationships.",
    "Tips for growing tomatoes in small urban gardens.",
    "What causes slow startup times on modern laptops."
]

# Generate document embeddings
documents_embeddings = embed_texts(documents)


# Example queries
queries = [
    "How do I make good pour-over coffee?",
    "Ways to improve heart health with workouts",
    "Understanding graph data structures",
    "Growing vegetables in limited space",
    "Why is my computer slow when starting?"
]

queries_embeddings = embed_texts(queries)


# Similarity via dot product
scores = queries_embeddings @ documents_embeddings.T

top_n = 3

for i, query in enumerate(queries):
    query_scores = scores[i]
    top_indices = np.argsort(query_scores)[::-1][:top_n]

    print(f"\nQuery: {query}")
    for idx in top_indices:
        score = query_scores[idx]
        doc = documents[idx]
        print(f"Score: {score:.4f} | Document: {doc}")

Benchmark

This quantized model was benchmarked against the original ONNX export on CPU.

Hardware

• CPU with AVX2 support
• ONNX Runtime CPUExecutionProvider

Dataset

Approximate evaluation setup:

• ~30k documents
• Semantic retrieval workload
• Dot-product similarity

Latency

Speedup

Speedup: ~2.07× faster

The quantized version reduces overall processing time by more than 50% while maintaining useful retrieval performance.

---