snowflake-arctic-embed-m-v2.0-academic

LinerAI/snowflake-arctic-embed-m-v2.0-academic for Academic Search

This is a fine-tuned version of Snowflake/snowflake-arctic-embed-m-v2.0 optimized for academic and scientific literature search. The model has been trained using contrastive learning with hard negative mining, specifically curated for academic search scenarios.

Highlights

• Optimized for Academic Search: Fine-tuned on datasets specifically designed for academic literature retrieval
• Hard Negative Mining: Trained with carefully mined hard negatives to improve discrimination between similar academic papers
• Matryoshka Representation Learning (MRL): Supports flexible embedding dimensions (768, 512, 256, 128) for efficiency
• Efficient & Fast: Medium-sized model offering excellent speed-performance trade-off
• Long Context: Supports up to 4096 tokens

Model Description

Evaluation Results

Training Details

Training Configuration

Training Data

The model was trained on LEAD (Liner Embedding Academic Dataset), a combination of ~55,560 samples tailored for academic search:

• MS MARCO (49%): General passage retrieval dataset with hard negatives
• Academic Search Dataset (51%): Custom dataset built specifically for academic literature search, with two-stage hard negative mining

Matryoshka Representation Learning (MRL)

This model supports Matryoshka Representation Learning. You can truncate embeddings to smaller dimensions (512, 256, 128) for faster computation and reduced storage.

# Full dimension (768)
full_embedding = embeddings[:, :768]

# MRL dimensions
embedding_512 = embeddings[:, :512]
embedding_256 = embeddings[:, :256]
embedding_128 = embeddings[:, :128]

Usage

Using Transformers

import torch
from transformers import AutoModel, AutoTokenizer

model_path = "LinerAI/snowflake-arctic-embed-l-v2.0-academic"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModel.from_pretrained(model_path, torch_dtype=torch.float16, trust_remote_code=True)
model.eval()

# For queries
def encode_query(text):
    input_text = f"query: {text}"
    inputs = tokenizer(input_text, return_tensors="pt", max_length=4096, truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
        embeddings = outputs.last_hidden_state[:, 0]  # CLS token
        embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    return embeddings

# For passages
def encode_passage(text):
    inputs = tokenizer(text, return_tensors="pt", max_length=4096, truncation=True)
    with torch.no_grad():
        outputs = model(**inputs)
        embeddings = outputs.last_hidden_state[:, 0]  # CLS token
        embeddings = torch.nn.functional.normalize(embeddings, p=2, dim=1)
    return embeddings

# Example: Academic search
query = "transformer models for protein structure prediction"
abstract = "We introduce AlphaFold, a deep learning system that predicts protein structures..."

query_emb = encode_query(query)
passage_emb = encode_passage(abstract)

similarity = torch.nn.functional.cosine_similarity(query_emb, passage_emb)
print(f"Similarity: {similarity.item():.4f}")

Input Format

Query Format

query: {your_query_text}

Passage Format

{your_passage_text}

Intended Use

• Academic Paper Search: Finding relevant research papers given a research query
• Literature Review: Discovering related work in academic literature
• Scientific Document Retrieval: Retrieving scientific documents, abstracts, and articles
• Research Question Answering: Finding papers that address specific research questions

Limitations

• Maximum sequence length is 4096 tokens
• Best performance achieved when using the specific input formats described above
• The model uses asymmetric encoding (query prefix for queries, no prefix for passages)
• Requires trust_remote_code=True for loading

License

This model is released under the Apache 2.0 license.