inference-free-splade-ko-v1

inference-free-splade-ko-v1

inference-free-splade-ko-v1 is a Korean-specific Asymmetric Inference-free SPLADE Sparse Encoder model finetuned from skt/A.X-Encoder-base using the sentence-transformers library. It maps sentences & paragraphs to a 50000-dimensional sparse vector space and can be used for semantic search and sparse retrieval.

Model Details

Model Description

• Model Type: Asymmetric Inference-free SPLADE Sparse Encoder
• Base model: skt/A.X-Encoder-base <!-- at revision b5c71f3601aedf38372fe21383ac7d04991af187 -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 50000 dimensions
• Similarity Function: Dot Product
• Training Dataset:
  • json

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Documentation: Sparse Encoder Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sparse Encoders on Hugging Face

Full Model Architecture

SparseEncoder(
  (0): Router(
    (sub_modules): ModuleDict(
      (query): Sequential(
        (0): SparseStaticEmbedding({'frozen': True}, dim=50000, tokenizer=BertTokenizerFast)
      )
      (document): Sequential(
        (0): MLMTransformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'ModernBertForMaskedLM'})
        (1): SpladePooling({'pooling_strategy': 'max', 'activation_function': 'relu', 'word_embedding_dimension': 50000})
      )
    )
  )
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SparseEncoder

# Download from the 🤗 Hub
model = SparseEncoder("yjoonjang/inference-free-splade-ko-v1")
# Run inference
queries = [
    '양이온 최적화 방법은 산소공공을 감소시키기 때문에 전자 농도가 증가하는 문제점을 갖고있을까?',
]
documents = [
    '산화물 TFT 소자 신뢰성 열화기구\n그러나 이와 같은 양이온 최적화 방법은 산소공공을 감소시키기 때문에 전자농도 역시 감소하게 되어 전계 이동도가 감소하는 문제점을 않고 있다. \n이는 산화물 반도체의 전도기구가 Percolation Conduction에 따르기 때문이다. ',
    '세포대사 기능 분석을 위한 광학센서 기반 용존산소와 pH 측정 시스템의 제작 및 특성 분석\n수소이온 농도가 증가하는 경우인 \\( \\mathrm{pH} \\) 가 낮아지면 다수의 수소이온들과 충돌한 방출 광이 에너지를 잃고 짧은 검출시간을 갖는다. \n반대로 \\( \\mathrm{pH} \\)가 높아질수록 형광물질로부터 방출된 광의 수명이 길어져 긴 검출시간을 가진다. ',
    '황화납 양자점 감지막을 통해 감도가 개선된 수소센서\n그 결과, \\( 100^{\\circ} \\mathrm{C} \\)까지 온도를 올렸을 때 초기의 전류값이 상이한 결과를 나타내었다. \n또한, QDs의 농도 증가는 동일 박막에 존재하는 QDs의 갯수를 증가시켜 수소 가스와 반응할 수 있는 binding site가 증가하여 더 빠른 시간에 반응이 가능하도록 할 수 있다. \n이와 반대로, 더 많은 QDs에서 이러한 반응이 발생하였기에 공기를 주입하여 원래의 전류로 돌아 가는 데는 더 많은 시간이 요구되는데 이러한 결과는 Fig. 6에 제시한 결과에서 확인할 수 있다.  \\( 10 \\mathrm{mg} / \\mathrm{ml} \\)이하의 농도로 소자를 제작할 경우, PbS QDs의 양이 너무 적어 균일한 박막을 형성이 어려웠고 그 결과 안정적인 전류 특성이 나타나지 않았다. ',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# torch.Size([1, 50000]) torch.Size([3, 50000])

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[122.9927,  79.5679,  76.8576]]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

MTEB-ko-retrieval Leaderboard

Evaluated all the Korean Retrieval Benchmarks on MTEB

Korean Retrieval Benchmark

<!-- - [XPQARetrieval](https://huggingface.co/datasets/jinaai/xpqa): 다양한 도메인의 한국어 문서 검색 데이터셋 --> <details> <summary>Reasons for excluding XPQARetrieval</summary>

• In our evaluation, we excluded the XPQARetrieval dataset. XPQA is a dataset designed to evaluate Cross-Lingual QA capabilities, and we determined it to be inappropriate for evaluating retrieval tasks that require finding supporting documents based on queries.
• Examples from the XPQARetrieval dataset are as follows:

  {
  	"query": "Is it unopened?",
  	"document": "No. It is a renewed product."
  },
  {
  	"query": "Is it compatible with iPad Air 3?",
  	"document": "Yes, it is possible."
  }
  

• Details for excluding this dataset is shown in the Github Issue

</details>

Evaluation Metrics

• Recall@10
• NDCG@10
• MRR@10
• AVG_Query_Active_Dims
• AVG_Corpus_Active_Dims

Evaluation Code

Our evaluation uses the SparseInformationRetrievalEvaluator from the sentence-transformers library. Look here for details

Evaluation Results

Comparison with Dense Embedding models (NDCG@10)

Look here for more details.

<!-- ### Evaluation Results | Model | Parameters | Recall@10 | NDCG@10 | MRR@10 | AVG_Query_Active_Dims | AVG_Corpus_Active_Dims | |-------|------------|-----------|---------|--------|----------------------|------------------------| | **yjoonjang/splade-ko-v1.0** | **0.1B** | **0.7626** | **0.7037** | **0.7379** | **110.7664** | **778.6494** | | [telepix/PIXIE-Splade-Preview](https://huggingface.co/telepix/PIXIE-Splade-Preview) | 0.1B | 0.7382 | 0.6869 | 0.7204 | 108.3300 | 718.5110 | | [opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1](https://huggingface.co/opensearch-project/opensearch-neural-sparse-encoding-multilingual-v1) | 0.1B | 0.5900 | 0.5137 | 0.5455 | 27.8722 | 177.5564 | --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

• Loss: <code>SpladeLoss</code> with these parameters:

  {
      "loss": "SparseMultipleNegativesRankingLoss(scale=1.0, similarity_fct='dot_score', gather_across_devices=True)",
      "document_regularizer_weight": 1e-08,
      "query_regularizer_weight": 1e-08
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• ddp_find_unused_parameters: False
• eval_on_start: True
• router_mapping: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document', 'negative_4': 'document', 'negative_5': 'document', 'negative_6': 'document'}
• learning_rate_mapping: {'0\.sub_modules\.query\.0\.weight': 5e-05}

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 4
• per_device_eval_batch_size: 4
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 7
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: False
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: True
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {'anchor': 'query', 'positive': 'document', 'negative_1': 'document', 'negative_2': 'document', 'negative_3': 'document', 'negative_4': 'document', 'negative_5': 'document', 'negative_6': 'document'}
• learning_rate_mapping: {'0\.sub_modules\.query\.0\.weight': 5e-05}

</details>

Framework Versions

• Python: 3.10.18
• Sentence Transformers: 5.1.2
• Transformers: 4.56.2
• PyTorch: 2.8.0+cu128
• Accelerate: 1.10.1
• Datasets: 4.1.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

SpladeLoss

@misc{formal2022distillationhardnegativesampling,
      title={From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective},
      author={Thibault Formal and Carlos Lassance and Benjamin Piwowarski and Stéphane Clinchant},
      year={2022},
      eprint={2205.04733},
      archivePrefix={arXiv},
      primaryClass={cs.IR},
      url={https://arxiv.org/abs/2205.04733},
}

SparseMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

FlopsLoss

@article{paria2020minimizing,
    title={Minimizing flops to learn efficient sparse representations},
    author={Paria, Biswajit and Yeh, Chih-Kuan and Yen, Ian EH and Xu, Ning and Ravikumar, Pradeep and P{'o}czos, Barnab{'a}s},
    journal={arXiv preprint arXiv:2004.05665},
    year={2020}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->