Octen-Embedding-4B-INT8

Octen-Embedding-4B-INT8

Octen-Embedding-4B-INT8 is a text embedding model developed by Octen for semantic search and retrieval tasks. This model is fine-tuned from Qwen/Qwen3-Embedding-4B and supports multiple languages, providing high-quality embeddings for various applications.

Quantization: This is an INT8 quantized version using bitsandbytes. INT8 quantization significantly reduces memory footprint (~50% smaller), making it suitable for deployment on resource-constrained environments. Note that while memory usage is reduced, inference speed may not necessarily improve and could be slightly slower than the BF16 version on some hardware.

Key Highlights

🥇 RTEB Leaderboard Champion (as of January 12, 2026)

• Octen-Embedding-8B ranks #1 on the RTEB Leaderboard with Mean (Task) score of 0.8045
• Excellent performance on both Public (0.7953) and Private (0.8157) datasets
• Demonstrates true generalization capability without overfitting to public benchmarks

Industry-Oriented Vertical Domain Expertise

• Legal: Legal document retrieval
• Finance: Financial reports, Q&A, and personal finance content
• Healthcare: Medical Q&A, clinical dialogues, and health consultations
• Code: Programming problems, code search, and SQL queries

Ultra-Long Context Support

• Supports up to 32,768 tokens context length
• Suitable for processing long documents in legal, healthcare, and other domains
• High-dimensional embedding space for rich semantic representation

Multilingual Capability

• Supports 100+ languages
• Includes various programming languages
• Strong multilingual, cross-lingual, and code retrieval capabilities

---

Open Source Model List

Model Family Design:

• Octen-Embedding-8B: Best performance, RTEB #1, for high-precision retrieval
• Octen-Embedding-4B: Best in 4B category, balanced performance and efficiency
• Octen-Embedding-0.6B: Lightweight deployment, suitable for edge devices and resource-constrained environments

For API access, deployment solutions, and technical documentation, visit octen.ai.

---

Experimental Results

RTEB Leaderboard (Overall Performance)

---

Model Details

• Base Model: Qwen/Qwen3-Embedding-4B
• Model Size: 4B parameters (INT8 quantized)
• Max Sequence Length: 40,960 tokens
• Embedding Dimension: 2560
• Languages: English, Chinese, and multilingual support
• Training Method: LoRA fine-tuning
• Quantization: INT8 (bitsandbytes)
• Memory Footprint: ~4GB (vs ~8GB for BF16 version)

Usage

Using Sentence Transformers

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("Octen/Octen-Embedding-4B-INT8")

# Encode sentences
sentences = [
    "This is an example sentence",
    "Each sentence is converted to a vector"
]

embeddings = model.encode(sentences)
print(embeddings.shape)
# Output: (2, 2560)

# Compute similarity
from sentence_transformers.util import cos_sim
similarity = cos_sim(embeddings[0], embeddings[1])
print(f"Similarity: {similarity.item():.4f}")

Using Transformers

from transformers import AutoModel, AutoTokenizer
import torch
import torch.nn.functional as F

tokenizer = AutoTokenizer.from_pretrained("Octen/Octen-Embedding-4B-INT8", padding_side="left")
model = AutoModel.from_pretrained("Octen/Octen-Embedding-4B-INT8")
model.eval()

def encode(texts):
    inputs = tokenizer(texts, padding=True, truncation=True,
                      max_length=8192, return_tensors="pt")

    with torch.no_grad():
        outputs = model(**inputs)
        # Use last token embedding
        embeddings = outputs.last_hidden_state[:, -1, :]
        # Normalize embeddings
        embeddings = F.normalize(embeddings, p=2, dim=1)

    return embeddings

# Example usage
texts = ["Hello world", "你好世界"]
embeddings = encode(texts)
similarity = torch.matmul(embeddings[0], embeddings[1])
print(f"Similarity: {similarity.item():.4f}")

Recommended Use Cases

• Semantic search and information retrieval
• Document similarity and clustering
• Question answering
• Cross-lingual retrieval
• Text classification with embeddings
• Deployment on GPU-constrained environments

Known Issues

When encoding documents without any instruction prefix, you may encounter unexpected behavior due to an upstream issue in Qwen3-Embedding. To avoid this issue, we recommend adding "- " (dash followed by space) at the beginning of your text when encoding documents:

# Recommended: Add "- " prefix for document encoding
documents = ["- " + doc for doc in documents]
embeddings = model.encode(documents)

This workaround ensures consistent and expected embedding behavior.

Limitations

• Performance may vary across different domains and languages
• Very long documents (>40K tokens) require truncation
• Optimized for retrieval tasks, not for text generation
• INT8 quantization may introduce minor accuracy degradation compared to BF16 version
• Inference speed may not improve despite reduced memory usage

License

This model is licensed under the Apache License 2.0.

This model is derived from Qwen/Qwen3-Embedding-4B, which is also licensed under Apache License 2.0.

Paper

For more details, please refer to our blog post: Octen Series: Optimizing Embedding Models to #1 on RTEB Leaderboard

Citation

If you find our work helpful, please consider citing:

@misc{octen2025rteb,
  title={Octen Series: Optimizing Embedding Models to #1 on RTEB Leaderboard},
  author={Octen Team},
  year={2025},
  url={https://octen-team.github.io/octen_blog/posts/octen-rteb-first-place/}
}