finetuned-bge-m3-base-en

SentenceTransformer based on BAAI/bge-m3

This is a sentence-transformers model finetuned from BAAI/bge-m3. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-m3 <!-- at revision 5617a9f61b028005a4858fdac845db406aefb181 -->
• Maximum Sequence Length: 8192 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 8192, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("aaa961/finetuned-bge-m3-base-en")
# Run inference
sentences = [
    'Shell integration: bash and zsh don\'t serialize \\n and ; characters Part of https://github.com/microsoft/vscode/issues/155639\r\n\r\nRepro:\r\n\r\n1. Open a bash or zsh session\r\n2. Run:\r\n  ```sh\r\n  echo "a\r\n  … b"\r\n  ```\r\n  ![image](https://github.com/microsoft/vscode/assets/2193314/95ce755b-cec5-4da1-8791-75923a088efa)\r\n3. ctrl+alt+r to run recent command, select the last command, 🐛 it\'s run without the new line\r\n  ![image](https://github.com/microsoft/vscode/assets/2193314/98e81851-3a68-422c-94e8-e3e4bb21ef53)\r\n',
    'TreeView state out of sync Testing #117304\r\n\r\nRepro: Not Sure\r\n\r\nTest state shows passed in file but still running in tree view.\r\n\r\n![image](https://user-images.githubusercontent.com/6561887/108932683-a082c600-75fe-11eb-8ccd-afa5d0e2a91e.png)\r\n',
    'Setting icon and color in createTerminal API no longer works correctly See https://github.com/fabiospampinato/vscode-terminals/issues/77\r\n\r\nLooks like the default tab color/icon change probably regressed this.\r\n\r\n',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.4264, 0.4315],
#         [0.4264, 1.0000, 0.4278],
#         [0.4315, 0.4278, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Triplet

• Dataset: bge-base-en-train
• Evaluated with <code>TripletEvaluator</code>

Triplet

• Dataset: bge-base-en-train
• Evaluated with <code>TripletEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 336 training samples
• Columns: <code>texts</code> and <code>label</code>
• Approximate statistics based on the first 336 samples:

  	texts	label
  type	string	int
  details	<ul><li>min: 12 tokens</li><li>mean: 340.9 tokens</li><li>max: 996 tokens</li></ul>	<ul><li>0: ~1.19%</li><li>5: ~0.60%</li><li>7: ~0.60%</li><li>8: ~0.60%</li><li>9: ~0.60%</li><li>10: ~0.60%</li><li>12: ~0.60%</li><li>13: ~0.60%</li><li>15: ~0.60%</li><li>16: ~0.60%</li><li>17: ~0.60%</li><li>19: ~0.60%</li><li>20: ~0.60%</li><li>21: ~0.60%</li><li>25: ~0.60%</li><li>26: ~0.60%</li><li>27: ~0.60%</li><li>28: ~0.60%</li><li>29: ~0.89%</li><li>30: ~0.89%</li><li>31: ~2.08%</li><li>33: ~1.49%</li><li>34: ~0.60%</li><li>36: ~0.60%</li><li>37: ~0.89%</li><li>38: ~0.60%</li><li>42: ~0.89%</li><li>43: ~1.19%</li><li>45: ~0.60%</li><li>47: ~0.60%</li><li>48: ~0.60%</li><li>49: ~0.60%</li><li>50: ~0.60%</li><li>51: ~0.60%</li><li>52: ~1.19%</li><li>53: ~0.60%</li><li>55: ~0.60%</li><li>57: ~1.19%</li><li>59: ~0.60%</li><li>60: ~0.60%</li><li>61: ~0.60%</li><li>62: ~0.60%</li><li>63: ~0.60%</li><li>64: ~0.89%</li><li>65: ~0.60%</li><li>67: ~0.60%</li><li>68: ~0.89%</li><li>69: ~0.60%</li><li>70: ~0.60%</li><li>71: ~0.60%</li><li>72: ~0.60%</li><li>73: ~0.60%</li><li>74: ~0.60%</li><li>75: ~0.60%</li><li>77: ~0.60%</li><li>78: ~0.60%</li><li>82: ~0.89%</li><li>84: ~0.60%</li><li>85: ~0.60%</li><li>86: ~0.60%</li><li>87: ~0.60%</li><li>88: ~0.60%</li><li>89: ~0.60%</li><li>90: ~0.60%</li><li>91: ~0.60%</li><li>92: ~0.89%</li><li>93: ~0.60%</li><li>95: ~0.60%</li><li>96: ~0.60%</li><li>97: ~0.60%</li><li>98: ~0.60%</li><li>99: ~0.60%</li><li>100: ~0.60%</li><li>101: ~0.60%</li><li>103: ~0.60%</li><li>105: ~0.60%</li><li>108: ~0.60%</li><li>109: ~0.60%</li><li>110: ~0.89%</li><li>113: ~0.60%</li><li>115: ~0.60%</li><li>116: ~0.60%</li><li>117: ~0.60%</li><li>118: ~0.60%</li><li>119: ~0.60%</li><li>120: ~1.49%</li><li>121: ~0.60%</li><li>126: ~0.60%</li><li>127: ~0.60%</li><li>130: ~0.60%</li><li>131: ~0.60%</li><li>134: ~0.60%</li><li>135: ~0.60%</li><li>136: ~0.60%</li><li>137: ~0.60%</li><li>138: ~0.60%</li><li>139: ~0.60%</li><li>140: ~0.60%</li><li>143: ~0.60%</li><li>144: ~0.60%</li><li>147: ~0.60%</li><li>148: ~0.60%</li><li>149: ~0.60%</li><li>150: ~0.60%</li><li>152: ~0.60%</li><li>153: ~0.60%</li><li>154: ~0.60%</li><li>155: ~0.60%</li><li>157: ~0.60%</li><li>158: ~0.60%</li><li>159: ~0.60%</li><li>160: ~0.60%</li><li>161: ~0.60%</li><li>163: ~0.60%</li><li>164: ~0.60%</li><li>165: ~0.60%</li><li>168: ~0.60%</li><li>170: ~0.60%</li><li>171: ~0.89%</li><li>174: ~0.89%</li><li>175: ~0.60%</li><li>176: ~0.60%</li><li>177: ~0.60%</li><li>178: ~0.60%</li><li>179: ~0.60%</li><li>180: ~0.60%</li><li>181: ~0.60%</li><li>182: ~0.60%</li><li>183: ~0.60%</li><li>185: ~0.60%</li><li>186: ~0.60%</li><li>187: ~0.60%</li><li>192: ~0.60%</li><li>193: ~0.60%</li><li>194: ~0.60%</li><li>195: ~0.89%</li><li>196: ~0.60%</li><li>197: ~0.60%</li><li>198: ~0.60%</li><li>199: ~0.60%</li><li>202: ~0.60%</li><li>204: ~0.60%</li><li>205: ~0.89%</li><li>207: ~0.60%</li><li>208: ~0.60%</li><li>209: ~0.60%</li><li>210: ~0.60%</li><li>213: ~0.60%</li><li>214: ~0.60%</li><li>215: ~0.60%</li><li>216: ~0.60%</li><li>218: ~0.60%</li></ul>

• Samples:

  texts	label
  <code>Branch list is sometimes out of order
  <br>Type: <b>Bug</b>

  <br> <br>1. Open a workspace <br>2. Quickly open the branch picker and type `main` <br> <br>**Bug** <br>The first time you do this, sometimes you end up with an unordered list: <br> <br> <br> <br> <br>The correct order shows up when you keep start typing or try doing this again: <br> <br> <br> <br> <br> <br> <br> <br> <br>VS Code version: Code - Insiders 1.91.0-insider (Universal) (0354163c1c66b950b0762364f5b4cd37937b624a, 2024-06-26T10:12:33.304Z) <br>OS version: Darwin arm64 23.5.0 <br>Modes: <br> <br><details> <br><summary>System Info</summary> <br> <br>\|Item\|Value\| <br>\|---\|---\| <br>\|CPUs\|Apple M2 Max (12 x 2400)\| <br>\|GPU Status\|2d_canvas: unavailable_software<br>canvas_oop_rasterization: disabled_off<br>direct_rendering_display_compositor: disabled_off_ok<br>gpu_compositing: disabled_software<br>multiple_raster_threads: enabled_on<br>ope...</code> | <code>218</code> | | <code>Git Branch Picker Race Condition If I paste the branch too quickly and then press enter, it does not switch to it, but creates a new branch. <br>This breaks muscle memory, as it works when you do it slowly. <br> <br> <br> <br> <br>Once loading completes, it should select the branch again.</code> | <code>218</code> | | <code>links aren't discoverable to screen reader users in markdown documents They're only discoverable via visual distinction and the action that can be taken (IE opening them) is only indicated in the tooltip AFAICT. <br> <br>https://github.com/microsoft/vscode/assets/29464607/09d28b81-c2cc-4477-b1fc-7b1de1baae74 <br> <br></code> | <code>177</code> |
• Loss: <code>BatchSemiHardTripletLoss</code>

Evaluation Dataset

Unnamed Dataset

• Size: 70 evaluation samples
• Columns: <code>texts</code> and <code>label</code>
• Approximate statistics based on the first 70 samples:

  	texts	label
  type	string	int
  details	<ul><li>min: 58 tokens</li><li>mean: 303.57 tokens</li><li>max: 864 tokens</li></ul>	<ul><li>1: ~2.86%</li><li>2: ~2.86%</li><li>6: ~2.86%</li><li>11: ~5.71%</li><li>14: ~2.86%</li><li>23: ~2.86%</li><li>32: ~5.71%</li><li>35: ~2.86%</li><li>39: ~2.86%</li><li>40: ~2.86%</li><li>46: ~2.86%</li><li>54: ~2.86%</li><li>83: ~2.86%</li><li>102: ~2.86%</li><li>104: ~4.29%</li><li>111: ~2.86%</li><li>122: ~2.86%</li><li>123: ~2.86%</li><li>125: ~2.86%</li><li>145: ~2.86%</li><li>146: ~2.86%</li><li>162: ~2.86%</li><li>166: ~2.86%</li><li>169: ~2.86%</li><li>184: ~2.86%</li><li>188: ~2.86%</li><li>190: ~2.86%</li><li>200: ~2.86%</li><li>201: ~4.29%</li><li>203: ~2.86%</li><li>206: ~2.86%</li><li>217: ~2.86%</li></ul>

• Samples:

  texts	label
  <code>Ctrl+I stopped working after first hold+talk+release Testing #213355

  <br> <br>Screencast shows that it seems to be in the wrong context and is trying to stop the session? <br> <br> <br> <br>Repro was just asking "Testing testing" and then trying to ask something else</code> | <code>217</code> | | <code>`Ctrl + I` does not work when chat input field has focus Testing #213355 <br> <br>`Ctrl + I` works in the editor and when I hold it, I get into speech mode. But when the chat input field (panel or inline chat) already has focus, `Ctrl + I` does not work. <br> <br>(Connected to Windows through Remote Desktop in case that matters.) <br></code> | <code>217</code> | | <code>Terminal renaming not functioning as expected in editor area <!-- ⚠️⚠️ Do Not Delete This! bug_report_template ⚠️⚠️ --> <br><!-- Please read our Rules of Conduct: https://opensource.microsoft.com/codeofconduct/ --> <br><!-- 🕮 Read our guide about submitting issues: https://github.com/microsoft/vscode/wiki/Submitting-Bugs-and-Suggestions --> <br><!-- 🔎 Search existing issues to avoid creating duplicates. --> <br><!-- 🧪 Test using the latest Insiders build to see if your issue has already been fixed: https://code.visualstudio.com/insiders/ --> <br><!-- 💡 Instead of creating your report here, use 'Report Issue' from the 'Help' menu in VS Code to pre-fill useful information. --> <br><!-- 🔧 Launch with `code --disable-extensions` to check. --> <br>Does this issue occur when all extensions are disabled?: Yes <br> <br><!-- 🪓 If you answered No above, use 'Help: Start Extension Bisect' from Command Palette to try to identify the cause. --> <br><!-- 📣 Issues caused by an extension need to be reported directly to the extensi...</code> | <code>201</code> |
• Loss: <code>BatchSemiHardTripletLoss</code>

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 1
• per_device_eval_batch_size: 1
• gradient_accumulation_steps: 8
• learning_rate: 2e-05
• num_train_epochs: 5
• warmup_ratio: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 1
• per_device_eval_batch_size: 1
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 8
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 5
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• Python: 3.10.10
• Sentence Transformers: 5.1.0
• Transformers: 4.55.3
• PyTorch: 2.7.1+cu128
• Accelerate: 1.10.0
• Datasets: 4.0.0
• Tokenizers: 0.21.4

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

BatchSemiHardTripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->