e5-large-v2-ctx1024

<h1 align="center">E5 Large V2 with 1024 tokens context window</h1>

This model uses positional embeddings tweak, described in Zero-Training Context Extension for Transformer Encoders via Nonlinear Absolute Positional Embeddings Interpolation, which allows it to has a context window twice as it's base model (intfloat/e5-large-v2). This is a training-less context improvement, meaning that no fine-tuning was applied to any of the base model parameters. The two models differ only in their positional embeddings. Paper describing applied tweak is coming soon.

Evaluation Results

The model was evaluated on four retrieval tasks from LongEmbed benchmark and achieved following NDCG@10 metrics (arb. units):

1. LEMBSummScreenFDRetrieval: 0.8617 (8.80 pp gain over intfloat/e5-large-v2),
2. LEMBQMSumRetrieval: 0.3112 (6.04 pp gain over intfloat/e5-large-v2),
3. LEMBWikimQARetrieval: 0.6570 (7.27 pp gain over intfloat/e5-large-v2),
4. LEMBNarrativeQARetrieval: 0.2792 (1.55 pp gain over intfloat/e5-large-v2)

Steps to reproduce:

from sentence_transformers import SentenceTransformer  # 4.0.2
import mteb  # 1.38.2


# load model
model = SentenceTransformer('idanylenko/e5-large-v2-ctx1024')

# define tasks
retrieval_task_list = [
    "LEMBSummScreenFDRetrieval",
    "LEMBQMSumRetrieval",
    "LEMBWikimQARetrieval",
    "LEMBNarrativeQARetrieval"
]
tasks = mteb.get_tasks(tasks=retrieval_task_list)

# run the evaluation
evaluation = mteb.MTEB(tasks=tasks)
results = evaluation.run(model)

Citation

@software{danylenko2025zero,
author = {Danylenko, Ivan},
doi = {10.5281/zenodo.17807059},
month = may,
title = {{Zero-Training Context Extension for Transformer Encoders via Nonlinear Absolute Positional Embeddings Interpolation}},
url = {https://github.com/Kowd-PauUh/encoders-context-extension},
version = {0.1.3},
year = {2025}
}