SPLADE_EuroBERT-210m_STS

SPLADE Sparse Encoder

This is a SPLADE Sparse Encoder model finetuned from EuroBERT/EuroBERT-210m on the french_sts dataset using the sentence-transformers library. It maps sentences & paragraphs to a 128256-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: EuroBERT/EuroBERT-210m <!-- at revision 6720a49834876b4ec5afcf6ef0d149a2c6501cce -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 128256 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • french_sts
• Language: fr

<!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'EuroBertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 128256})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("bourdoiscatie/SPLADE_EuroBERT-210m_STS")
# Run inference
sentences = [
    "Oui, je peux vous dire d'après mon expérience personnelle qu'ils ont certainement sifflé.",
    "Il est vrai que les bombes de la Seconde Guerre mondiale faisaient un bruit de sifflet lorsqu'elles tombaient.",
    "J'envisage de dépenser les 48 dollars par mois pour le système GTD (Getting things done) annoncé par David Allen.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 128256]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6421, 0.5114],
#         [0.6421, 1.0000, 0.3132],
#         [0.5114, 0.3132, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Semantic Similarity

• Datasets: sts-dev and sts-test
• Evaluated with <code>SparseEmbeddingSimilarityEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

french_sts

• Dataset: french_sts at 47128cc
• Size: 12,227 training samples
• Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	<ul><li>min: 6 tokens</li><li>mean: 12.74 tokens</li><li>max: 37 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 12.72 tokens</li><li>max: 37 tokens</li></ul>	<ul><li>min: 0.0</li><li>mean: 0.44</li><li>max: 1.0</li></ul>

• Samples:

  sentence1	sentence2	score
  <code>Un avion est en train de décoller.</code>	<code>Un avion est en train de décoller.</code>	<code>1.0</code>
  <code>Un homme est en train de fumer.</code>	<code>Un homme fait du patinage.</code>	<code>0.10000000149011612</code>
  <code>Une personne jette un chat au plafond.</code>	<code>Une personne jette un chat au plafond.</code>	<code>1.0</code>

• Loss: <code>SpladeLoss</code> with these parameters:

  {
      "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')",
      "document_regularizer_weight": 0.003
  }
  

Evaluation Dataset

french_sts

• Dataset: french_sts at 47128cc
• Size: 3,526 evaluation samples
• Columns: <code>sentence1</code>, <code>sentence2</code>, and <code>score</code>
• Approximate statistics based on the first 1000 samples:

  	sentence1	sentence2	score
  type	string	string	float
  details	<ul><li>min: 6 tokens</li><li>mean: 21.48 tokens</li><li>max: 64 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 21.45 tokens</li><li>max: 62 tokens</li></ul>	<ul><li>min: 0.0</li><li>mean: 0.43</li><li>max: 1.0</li></ul>

• Samples:

  sentence1	sentence2	score
  <code>Un homme avec un casque de sécurité est en train de danser.</code>	<code>Un homme portant un casque de sécurité est en train de danser.</code>	<code>1.0</code>
  <code>Un jeune enfant monte à cheval.</code>	<code>Un enfant monte à cheval.</code>	<code>0.949999988079071</code>
  <code>Un homme donne une souris à un serpent.</code>	<code>L'homme donne une souris au serpent.</code>	<code>1.0</code>

• Loss: <code>SpladeLoss</code> with these parameters:

  {
      "loss": "SparseCosineSimilarityLoss(loss_fct='torch.nn.modules.loss.MSELoss')",
      "document_regularizer_weight": 0.003
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: epoch
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• bf16: True

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: epoch
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• tp_size: 0
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.0.0
• Transformers: 4.51.3
• PyTorch: 2.6.0+cu124
• Accelerate: 1.6.0
• Datasets: 2.16.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->