Downloads · 30 days
29
13% of all-time downloads
tomaarsen/TinyBERT_L-4_H-312_v2-distilled-from-stsb-roberta-base-v2
TinyBERT_L-4_H-312_v2-distilled-from-stsb-roberta-base-v2 is a sentence similarity model from tomaarsen. Use it when you need a score for how close two texts are. It is set up for sentence-transformers.
This is a sentence-transformers model finetuned from nreimers/TinyBERTL-4H-312v2 on the sentence-transformers/wikipedia-en-sentences dataset. It maps sentences & paragraphs to a 312-dimensional dense vector space and…
Downloads · 30 days
29
13% of all-time downloads
All-time downloads
230
Public
Parameters
14.4M
57.4 MB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors57.4 MB · 98%
From the Hugging Face model README
This is a sentence-transformers model finetuned from nreimers/TinyBERT_L-4_H-312_v2 on the sentence-transformers/wikipedia-en-sentences dataset. It maps sentences & paragraphs to a 312-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 312, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("tomaarsen/TinyBERT_L-4_H-312_v2-distilled-from-stsb-roberta-base-v2")
# Run inference
sentences = [
'A person standing',
'There is a person standing outside',
'A young man plays a racing video game.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 312]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings)
print(similarities.shape)
# [3, 3]
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
sts-dev| Metric | Value |
|---|---|
| pearson_cosine | 0.8078 |
| spearman_cosine | 0.8209 |
| pearson_manhattan | 0.8226 |
| spearman_manhattan | 0.8203 |
| pearson_euclidean | 0.8216 |
| spearman_euclidean | 0.8202 |
| pearson_dot | 0.7901 |
| spearman_dot | 0.7914 |
| pearson_max | 0.8226 |
| spearman_max | 0.8209 |
| Metric | Value |
|---|---|
| negative_mse | -50.1254 |
sts-test| Metric | Value |
|---|---|
| pearson_cosine | 0.7517 |
| spearman_cosine | 0.7558 |
| pearson_manhattan | 0.7763 |
| spearman_manhattan | 0.7597 |
| pearson_euclidean | 0.7706 |
| spearman_euclidean | 0.7554 |
| pearson_dot | 0.7307 |
| spearman_dot | 0.7098 |
| pearson_max | 0.7763 |
| spearman_max | 0.7597 |
| sentence | label | |
|---|---|---|
| type | string | list |
| details | <ul><li>min: 4 tokens</li><li>mean: 12.24 tokens</li><li>max: 52 tokens</li></ul> | <ul><li>size: 312 elements</li></ul> |
| sentence | label |
|---|---|
| <code>A person on a horse jumps over a broken down airplane.</code> | <code>[-0.09614687412977219, 0.6815224885940552, 2.702199935913086, 1.8371250629425049, -1.2949433326721191, ...]</code> |
| <code>Children smiling and waving at camera</code> | <code>[2.769360303878784, 3.074428081512451, -7.291755676269531, 5.248741149902344, 2.85081148147583, ...]</code> |
| <code>A boy is jumping on skateboard in the middle of a red bridge.</code> | <code>[-3.0669667720794678, 2.9899890422821045, -1.253997802734375, 6.15218448638916, 0.5838223099708557, ...]</code> |
| sentence | label | |
|---|---|---|
| type | string | list |
| details | <ul><li>min: 5 tokens</li><li>mean: 13.23 tokens</li><li>max: 57 tokens</li></ul> | <ul><li>size: 312 elements</li></ul> |
| sentence | label |
|---|---|
| <code>Two women are embracing while holding to go packages.</code> | <code>[6.200135707855225, -2.0865142345428467, -2.1313390731811523, -1.9593913555145264, -1.081985592842102, ...]</code> |
| <code>Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink.</code> | <code>[1.7725015878677368, 0.6873414516448975, -2.5191268920898438, 3.866339683532715, 2.853647470474243, ...]</code> |
| <code>A man selling donuts to a customer during a world exhibition event held in the city of Angeles</code> | <code>[-3.317653179168701, 3.0908589363098145, 0.1683920919895172, -2.4405274391174316, -3.1366524696350098, ...]</code> |
eval_strategy: stepsper_device_train_batch_size: 64per_device_eval_batch_size: 64learning_rate: 0.0001num_train_epochs: 1warmup_ratio: 0.1fp16: Trueload_best_model_at_end: Trueoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: stepsprediction_loss_only: Falseper_device_train_batch_size: 64per_device_eval_batch_size: 64per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 1eval_accumulation_steps: Nonelearning_rate: 0.0001weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1.0num_train_epochs: 1max_steps: -1lr_scheduler_type: linearlr_scheduler_kwargs: {}warmup_ratio: 0.1warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Falsefp16: Truefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Trueignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torchoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Nonedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Falsehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseeval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters:auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Nonedispatch_batches: Nonesplit_batches: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: proportional| Epoch | Step | Training Loss | loss | negative_mse | sts-dev_spearman_cosine | sts-test_spearman_cosine |
|---|---|---|---|---|---|---|
| 0.032 | 100 | 0.8847 | - | - | - | - |
| 0.064 | 200 | 0.8136 | - | - | - | - |
| 0.096 | 300 | 0.697 | - | - | - | - |
| 0.128 | 400 | 0.6128 | - | - | - | - |
| 0.16 | 500 | 0.5634 | 0.6324 | -63.2356 | 0.7564 | - |
| 0.192 | 600 | 0.5294 | - | - | - | - |
| 0.224 | 700 | 0.5035 | - | - | - | - |
| 0.256 | 800 | 0.4861 | - | - | - | - |
| 0.288 | 900 | 0.4668 | - | - | - | - |
| 0.32 | 1000 | 0.4515 | 0.5673 | -56.7263 | 0.7965 | - |
| 0.352 | 1100 | 0.4376 | - | - | - | - |
| 0.384 | 1200 | 0.4274 | - | - | - | - |
| 0.416 | 1300 | 0.4178 | - | - | - | - |
| 0.448 | 1400 | 0.4098 | - | - | - | - |
| 0.48 | 1500 | 0.4053 | 0.5354 | -53.5381 | 0.8091 | - |
| 0.512 | 1600 | 0.3934 | - | - | - | - |
| 0.544 | 1700 | 0.391 | - | - | - | - |
| 0.576 | 1800 | 0.3848 | - | - | - | - |
| 0.608 | 1900 | 0.3785 | - | - | - | - |
| 0.64 | 2000 | 0.3737 | 0.5168 | -51.6829 | 0.8159 | - |
| 0.672 | 2100 | 0.3716 | - | - | - | - |
| 0.704 | 2200 | 0.3695 | - | - | - | - |
| 0.736 | 2300 | 0.3666 | - | - | - | - |
| 0.768 | 2400 | 0.3616 | - | - | - | - |
| 0.8 | 2500 | 0.358 | 0.5067 | -50.6687 | 0.8189 | - |
| 0.832 | 2600 | 0.3551 | - | - | - | - |
| 0.864 | 2700 | 0.3544 | - | - | - | - |
| 0.896 | 2800 | 0.3524 | - | - | - | - |
| 0.928 | 2900 | 0.3524 | - | - | - | - |
| 0.96 | 3000 | 0.3529 | 0.5013 | -50.1254 | 0.8209 | - |
| 0.992 | 3100 | 0.3496 | - | - | - | - |
| 1.0 | 3125 | - | - | - | - | 0.7558 |
Carbon emissions were measured using CodeCarbon.
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@inproceedings{reimers-2020-multilingual-sentence-bert,
title = "Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2020",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/2004.09813",
}
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->