Oolel-Embed

Oolel-Embed

https://arxiv.org/pdf/2602.19991v1 "Cross-lingual Matryoshka Representation Learning across Speech and Text"

Oolel-Embed is a bilingual Wolof speech <-> French text Matryoshka embedding model for cross-lingual retrieval. It uses a late-fusion architecture that injects HuBERT speech features into a text embedding backbone, enabling dimension-adaptive inference for retrieval under different cost/latency budgets. The model is trained for direct French document retrieval from Wolof speech without an ASR→MT pipeline, and supports prompted multitask retrieval settings.

Usage

Installation

pip install -U transformers==4.55.4

Load the model

model = SentenceTransformer(
    "soynade-research/Oolel-Embed",
    )
model[0].set_prompt(
    "Given a speech Wolof query, retrieve relevant passages that answer the speech query\nQuery:"
)

Examples of French documents:

text_1 = """Su ko nit ñi laajee: 'Ñan ngay defal say film?', day tontu naan: Europe mooy sama marché, waaye ñimay seetaan, Afrik lañu nekk.' Fii ci réewum Orop laa wara ñëw ngir wut xaalis bi may defare samay film, ndax fi la dooley koom-koom bi nekk. Waaye ñi ma jublu, maanaam ñi ma bëgg jàppale ci seen yewwute jaarale ko ci samay film, Afrik lañu nekk. Du Senegaal kesse, waaye ci Afrig yépp"""

text_2 = """Deux frères vivaient ensemble. Depuis longtemps, ils avaient enterré leur père et, à part l'un l'autre, ils n'avaient personne au monde. Ils s'aidaient mutuellement, et vivaient dans l'affection et la bonne entente fraternelle. Ils gagnaient pauvrement leur vie en allant à la pêche. Chaque matin, dès que pointait le jour, ils prenaient leurs filets et allaient en mer."""

text_3 = """Je viens de découvrir un conte wolof fascinant. Il parle de Mbàbba Kumba, un village aujourd'hui disparu. Les anciens s'en souviennent uniquement à cause d'une terrible tragédie. L'histoire se concentre sur Moodu. C'était un chef de famille généreux, beau et courageux. Mais il avait un défaut fatal : son entêtement. Cinq jours avant la fête de la Tabaski, son ego prend le dessus. Il réunit ses deux épouses. Il leur promet d'accomplir une chose que personne n'a jamais faite pour prouver sa valeur et son amour. Le suspense est parfait. On sait déjà que cet excès de fierté ne va pas seulement le ruiner. Son action va détruire tout son village. C'est une belle leçon sur les dangers de l'orgueil."""

Retrieving documents from a speech query

audio_query = "https://huggingface.co/soynade-research/Wolof-HuBERT-CTC/resolve/main/story.wav"
query_embeddings = model.encode_query(audio_query)
document_embeddings = model.encode_document([text_1, text_2, text_3])

similarity = model.similarity(query_embeddings, document_embeddings)
print(similarity)

Retrieving documents from a text query

text_query = """Amoon na fi ab dëkk bu tuddoon Mbàbba Kumba. Kenn xamatul fu mu nekk léegi. Waaye mag ñi fattewuñu ko ngir ay jalooreem, rawatina musibam dëkk ba ak ay ñoñam. Moodu nekkoon na boroom kër bu tabe, taaru te jàmbaare. Waaye dëgër bopp a ko yàqoon. Ba tabaski desee jurôomi fan, la woo ñaari jabaram ya Kumba aawo ba ak Nogay ñaareel ba ni leen: Dinaa fi def lu kenn masul def ngir ngeen xam ne seen nijaay du ku tuuti te it ku fonk njabootam la."""
query_embeddings = model.encode_query(text_query)
document_embeddings = model.encode_document([text_1, text_2, text_3])

similarity = model.similarity(query_embeddings, document_embeddings)
print(similarity)

Full Model Architecture

SentenceTransformer(
  (0): OolelEmbed(
    (text_embedding_model): SentenceTransformer(
      (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Qwen3Model'})
      (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': True, 'include_prompt': True})
      (2): Normalize()
    )
    (speech_embedding_model): HubertMatryoshka(
      (hubert): HubertModel(
        (feature_extractor): HubertFeatureEncoder(
          (conv_layers): ModuleList(
            (0): HubertGroupNormConvLayer(
              (conv): Conv1d(1, 512, kernel_size=(10,), stride=(5,), bias=False)
              (activation): GELUActivation()
              (layer_norm): GroupNorm(512, 512, eps=1e-05, affine=True)
            )
            (1-4): 4 x HubertNoLayerNormConvLayer(
              (conv): Conv1d(512, 512, kernel_size=(3,), stride=(2,), bias=False)
              (activation): GELUActivation()
            )
            (5-6): 2 x HubertNoLayerNormConvLayer(
              (conv): Conv1d(512, 512, kernel_size=(2,), stride=(2,), bias=False)
              (activation): GELUActivation()
            )
          )
        )
        (feature_projection): HubertFeatureProjection(
          (layer_norm): LayerNorm((512,), eps=1e-05, elementwise_affine=True)
          (projection): Linear(in_features=512, out_features=768, bias=True)
          (dropout): Dropout(p=0.0, inplace=False)
        )
        (encoder): HubertEncoder(
          (pos_conv_embed): HubertPositionalConvEmbedding(
            (conv): ParametrizedConv1d(
              768, 768, kernel_size=(128,), stride=(1,), padding=(64,), groups=16
              (parametrizations): ModuleDict(
                (weight): ParametrizationList(
                  (0): _WeightNorm()
                )
              )
            )
            (padding): HubertSamePadLayer()
            (activation): GELUActivation()
          )
          (layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
          (dropout): Dropout(p=0.0, inplace=False)
          (layers): ModuleList(
            (0-11): 12 x HubertEncoderLayer(
              (attention): HubertAttention(
                (k_proj): Linear(in_features=768, out_features=768, bias=True)
                (v_proj): Linear(in_features=768, out_features=768, bias=True)
                (q_proj): Linear(in_features=768, out_features=768, bias=True)
                (out_proj): Linear(in_features=768, out_features=768, bias=True)
              )
              (dropout): Dropout(p=0.0, inplace=False)
              (layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
              (feed_forward): HubertFeedForward(
                (intermediate_dropout): Dropout(p=0.0, inplace=False)
                (intermediate_dense): Linear(in_features=768, out_features=3072, bias=True)
                (intermediate_act_fn): GELUActivation()
                (output_dense): Linear(in_features=3072, out_features=768, bias=True)
                (output_dropout): Dropout(p=0.0, inplace=False)
              )
              (final_layer_norm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            )
          )
        )
      )
      (projector): Linear(in_features=9216, out_features=1024, bias=False)
      (pooling_parameters): Conv1d(1024, 1024, kernel_size=(2,), stride=(2,))
    )
  )
)

Training Details

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.2.3
• Transformers: 4.55.4
• PyTorch: 2.10.0+cpu
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

@misc{sy2026crosslingualmatryoshkarepresentationlearning,
      title={Cross-lingual Matryoshka Representation Learning across Speech and Text}, 
      author={Yaya Sy and Dioula Doucouré and Christophe Cerisara and Irina Illina},
      year={2026},
      eprint={2602.19991},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2602.19991}, 
}