There-Are-No-Games

SPLADE Sparse Encoder

This is a SPLADE Sparse Encoder model finetuned from naver/splade-v3 using the sentence-transformers library. It maps sentences & paragraphs to a 30522-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

This model was made as part of an Advanced Information Retrieval Course in the scope of our project which is meant to recommend games based on their description. See also our github: https://github.com/Spivonxe/There-are-no-games

Model Description

• Model Type: SPLADE Sparse Encoder
• Base model: naver/splade-v3 <!-- at revision fdfeceb91d7b9de7985b38addd3ba9f53a59a355 -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 30522 dimensions
• Similarity Function: Dot Product

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): MLMTransformer({'max_seq_length': 512, 'do_lower_case': False, 'architecture': 'BertForMaskedLM'})
  (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 30522})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("mazombieme/There-Are-No-Games")
# Run inference
queries = [
    "artifacts powerful",
]
documents = [
    'Dreamscaper By night, delve deep into your subconscious and discover powerful artifacts to conquer your nightmares. By day, explore the city of Redhaven and build relationships to unlock the power of your dreams. DREAM. DIE. WAKE. REPEAT.',
    "The Mystery of Devils House The Mystery of Devil's House is a 2D Horror Action Platformer based on the story of the Haunted house called William's House located in Massachusetts investigated by The Ghostbuster Crew: International.",
    'KoroNeko Roll your way through a cozy, kawaii world full of charming characters and challenging puzzles to save your siblings from Strawberry the Witch!',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 30522] [3, 30522]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[12.4569,  0.0118,  0.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 39,381 training samples
• Columns: <code>query</code> and <code>document</code>
• Approximate statistics based on the first 1000 samples:

  	query	document
  type	string	string
  details	<ul><li>min: 4 tokens</li><li>mean: 7.17 tokens</li><li>max: 22 tokens</li></ul>	<ul><li>min: 10 tokens</li><li>mean: 50.35 tokens</li><li>max: 103 tokens</li></ul>

• Samples:

  query	document
  <code>control difficulties</code>	<code>Manabi SandBox This is an action game in which you control a knight and explore unknown ruins. The goal is to collect all the fruits on the map while exterminating monsters. Overcome various difficulties with simple actions and ingenuity. The Knights never die, so don't give up and solve all the riddles.</code>
  <code>atmospheric every mystery narrativedriven mansion tension</code>	<code>Crimson Mansion Uncover the secrets of a forgotten mansion where every room raises new questions. A narrative-driven exploration game full of mystery and atmospheric tension.</code>
  <code>visually games edutainment pointandclick puzzle adventure</code>	<code>Endacopia Endacopia is a Point-and-Click Puzzle Adventure that is visually reminiscent of early computer edutainment games... with underlying horrors.</code>

• Loss: <code>SpladeLoss</code> with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
      "document_regularizer_weight": 0.003,
      "query_regularizer_weight": 0
  }
  

Evaluation Dataset

Unnamed Dataset

• Size: 4,913 evaluation samples
• Columns: <code>query</code> and <code>document</code>
• Approximate statistics based on the first 1000 samples:

  	query	document
  type	string	string
  details	<ul><li>min: 4 tokens</li><li>mean: 6.97 tokens</li><li>max: 16 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 49.39 tokens</li><li>max: 118 tokens</li></ul>

• Samples:

  query	document
  <code>cones playground park explore</code>	<code>Pipo Park Pipo Park is a messy and happy digital playground. Explore a small park, play football, throw traffic cones, balls, dogs, and more... Have a nice time!</code>
  <code>sentient united employee dark</code>	<code>Quantum Eye Set in a dark, futuristic united states, all citizens are watched under a highly-advanced supercomputer known as the "Quantum Eye" Machine. You are an employee at the Quantum Eye facility, but quantum eye goes sentient. Your goal is simple, SHUT IT DOWN...</code>
  <code>five solve in twisted 2 powerful</code>	<code>Hanaja's Body 2 in One Indulge in this twisted tale of transformations as five characters struggle for control of one body: yours! Mix and match between five different, uniquely powerful forms to defeat dangerous foes and solve puzzles to escape the tome you've been trapped in before it's too late.</code>

• Loss: <code>SpladeLoss</code> with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=False)",
      "document_regularizer_weight": 0.003,
      "query_regularizer_weight": 0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• learning_rate: 2e-05
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates
• router_mapping: {'query': 'query', 'answer': 'document'}

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 16
• per_device_eval_batch_size: 16
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {'query': 'query', 'answer': 'document'}
• learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• Python: 3.10.12
• Sentence Transformers: 5.2.0
• Transformers: 4.57.1
• PyTorch: 2.9.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.2
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->