bge-small-vscode-dup-hardneg

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5 <!-- at revision 5c38ec7c405ec4b44b94cc5a9bb96e735b38267a -->
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'BertModel'})
  (1): Pooling({'embedding_dimension': 384, 'pooling_mode': 'cls', 'include_prompt': True})
  (2): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'VS Code snap broken on Ubuntu 24.04\n\nDoes this issue occur when all extensions are disabled?: Yes/No\r\n\r\n \r\n \r\n- VS Code Version: 1.91.1\r\n- OS Version: Ubuntu 24.04\r\n\n\r\n1. Install VS Code as a snap instead of an official .deb\r\n2. Observe issues running it: https://github.com/microsoft/vscode/issues/212430\r\n\r\nAll of the suggestions in the above issue point to the snap installation being broken. Since the snap is maintained in [this repository](https://github.com/microsoft/vscode/blob/main/resources/linux/snap/snapcraft.yaml), I figured it would be right to have an issue about this.',
    "VSCode Snap doesn't start when using Wayland mode\n\nDoes this issue occur when all extensions are disabled?: Yes\r\n\r\n \r\n \r\n- VS Code Version: \r\n1.86.0-insider\r\nfb769554405bee9be16e21ceb0a496bd29126941\r\nx64\r\n\r\n- OS Version: Ubuntu 23.10\r\n\n\r\n1. Download the code-insiders snap package\r\n2. Run it like this:  `code-insiders --ozone-platform-hint=auto --verbose`\r\n3. VSCode doesn't start but reports this in the console:\r\n\r\n```\r\n Warning: 'ozone-platform-hint' is not in the list of known options, but still passed to Electron/Chromium.\r\nMESA-LOADER: failed to open radeonsi: /usr/lib/dri/radeonsi_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)\r\nfailed to load driver: radeonsi\r\nMESA-LOADER: failed to open kms_swrast: /usr/lib/dri/kms_swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)\r\nfailed to load driver: kms_swrast\r\nMESA-LOADER: failed to open swrast: /usr/lib/dri/swrast_dri.so: cannot open shared object file: No such file or directory (search paths /usr/lib/x86_64-linux-gnu/dri:\\$${ORIGIN}/dri:/usr/lib/dri, suffix _dri)\r\nfailed to load swrast driver\r\n[340245:0109/202358.589994:WARNING:wayland_object.cc(158)] Binding to gtk_shell1 version 4 but version 5 is available.\r\n[340245:0109/202358.590067:WARNING:wayland_object.cc(158)] Binding to zwp_pointer_gestures_v1 version 1 but version 3 is available.\r\n[340245:0109/202358.590093:WARNING:wayland_object.cc(158)] Binding to zwp_linux_dmabuf_v1 version 3 but version 4 is available.\r\n(code-insiders:340245): GLib-GIO-ERROR **: 20:23:58.596: Settings schema 'org.gnome.settings-daemon.plugins.xsettings' does not contain a key named 'antialiasing'\r\n```",
    'Unable to Install VSCode on Linux Mint 19.3\n\nDoes this issue occur when all extensions are disabled?: Yes/No\r\n\r\n \r\n \r\n- VS Code Version: 1.86.0\r\n- OS Version: Linux Mint 19.3 Tricia\r\n\n\r\n1. Download .deb package\r\n2. Run instalation\r\n3. Package installer report error "**_Dependency is not satisfiable: libc6 (>= 2.28)_**"',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6231, 0.4548],
#         [0.6231, 1.0000, 0.3757],
#         [0.4548, 0.3757, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 6,000 training samples
• Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>sentence_2</code>
• Approximate statistics based on the first 100 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  modality	text	text	text
  details	<ul><li>min: 14 tokens</li><li>mean: 144.57 tokens</li><li>max: 256 tokens</li></ul>	<ul><li>min: 14 tokens</li><li>mean: 161.4 tokens</li><li>max: 256 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 144.04 tokens</li><li>max: 256 tokens</li></ul>

• Samples:

  sentence_0	sentence_1	sentence_2
  <code>claude code extension doesn't work if a termiinal window opens<br><br>It goes blank and no history</code>	<code>Having a visible terminal causes all webviews to disappear<br><br>In today's insiders build, whenever a terminal is visible all extension webview disappear (blank out). If you hide the terminal, they all re-appear.</code>	<code>claude is not working<br><br>I ask a question and it errors out<br><br>System Info: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36</code>
  <code>'Select language' for empty files not working anymore<br><br>Does this issue occur when all extensions are disabled?: Yes/No

  <br> <br> <br> <br>- VS Code Version: <br>``` <br>Version: 1.91.0 <br>Commit: ea1445cc7016315d0f5728f8e8b12a45dc0a7286 <br>Date: 2024-07-01T18:52:41.959Z <br>Electron: 29.4.0 <br>ElectronBuildId: 9728852 <br>Chromium: 122.0.6261.156 <br>Node.js: 20.9.0 <br>V8: 12.2.281.27-electron.0 <br>OS: Linux x64 6.1.0-22-amd64 <br>``` <br> <br>**Steps to Reproduce:** <br> <br>1. Press Crtl+N <br>2. In the empty file dialong that pops up, click 'Select a language': <br> <br> <br>**Expected behavior:** should open the language picker drop-down, like it used to do in previous versions. <br> <br>**Actual behavior:** nothing happens.</code> | <code>select a language option is not working<br><br>the select a language button on a new file is not working neither giving any output or giving any option to select the languages available</code> | <code>select language mode is missing<br><br>Usually I see a text that shows “Select a language to get started. Start typing to dismiss, or don't show this again.” but now I don’t see it and it makes it hard to select the program language that I want to choose.</code> | | <code>Integrated terminal incorrectly renders oh-my-posh rprompt<br><br>The right prompt feature of the prompt engine it's rendered incorrectly in the integrated terminal, displaying Unicode(?) text <br>instead. <br>This issue do not occur on other terminal emulators, such as the Windows Terminal app. <br> <br></code> | <code>Terminal in VSCode Insiders not able to display special characters<br><br>The terminal in VSCode - insiders is unable to display special characters in terminals after latest update <br>How it looks in VSCode: <br> <br> <br>How it looks in VSCode - insiders <br> <br> <br>The font settings are the exact same for both versions</code> | <code>Uppercase character 'W' is not rendered correctly in the terminal with 'Cascadia' series font<br><br>Does this issue occur when all extensions are disabled?: Yes <br> <br> <br> <br> <br>版本: 1.88.0-insider (user setup) <br>提交: be210b3a60c7d60030c1d3d92da00d008edf6ab9 <br>日期: 2024-03-15T05:48:19.790Z <br>Electron: 28.2.6 <br>ElectronBuildId: 27476517 <br>Chromium: 120.0.6099.291 <br>Node.js: 18.18.2 <br>V8: 12.0.267.19-electron.0 <br>OS: Windows_NT x64 10.0.22631 <br> <br>The uppercase character 'W' is not rendered correctly in the terminal with 'Cascadia' series font. It will render properly in editor, terminal tab with color, in Windows Terminal application, and with other font series. Also it's seems that uppercase 'W' is the only character that occurs this problem. I have no idea why this happened. <br> <br> <br>('W' wasn't rendered correctly) <br> <br> ...</code> |
• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 16
• num_train_epochs: 2
• per_device_eval_batch_size: 16
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 16
• num_train_epochs: 2
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: None
• trackio_bucket_id: None
• trackio_static_space_id: None
• per_device_eval_batch_size: 16
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• dataloader_multiprocessing_context: None
• dataloader_in_order: True
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_static_graph: None
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: None
• fsdp_config: None
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}
• warmup_ratio: None

</details>

Training Logs

Training Time

• Training: 9.1 minutes

Framework Versions

• Python: 3.13.15
• Sentence Transformers: 5.7.0
• Transformers: 5.16.1
• PyTorch: 2.11.0+cu128
• Accelerate: 1.14.0
• Datasets: 4.8.5
• Tokenizers: 0.23.1

Additional Resources

• Training and Finetuning Embedding Models with Sentence Transformers: the end-to-end guide for training or finetuning Sentence Transformer models.
• Introduction to Matryoshka Embedding Models: variable-size embeddings that can be truncated with minimal quality loss.
• Binary and Scalar Embedding Quantization for Significantly Faster & Cheaper Retrieval: post-training compression of embedding vectors.
• Multimodal Embedding & Reranker Models with Sentence Transformers: use text, image, audio, and video models through the same API.
• Training and Finetuning Multimodal Embedding & Reranker Models with Sentence Transformers: train multimodal embedding models, with a Visual Document Retrieval walkthrough.

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->