Downloads · 30 days
54
16% of all-time downloads
kamkol/ab_testing_finetuned_2_arctic_ft-711c8143-04b7-4ecb-8e01-8847274c1d9c
ab_testing_finetuned_2_arctic_ft-711c8143-04b7-4ecb-8e01-8847274c1d9c is a sentence similarity model from kamkol. Use it when you need a score for how close two texts are. It is set up for sentence-transformers.
This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semanti…
Downloads · 30 days
54
16% of all-time downloads
All-time downloads
345
Public
Parameters
334M
1.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.3 GB · 100%
From the Hugging Face model README
This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the 🤗 Hub
model = SentenceTransformer("kamkol/ab_testing_finetuned_2_arctic_ft-711c8143-04b7-4ecb-8e01-8847274c1d9c")
# Run inference
sentences = [
'What are the estimated monthly rates of first-party cookie deletion among U.S. Internet users according to comScore and Anirban Dasgupta et al.?',
'Tracking on the Web: Privacy and Security Implication [23]. \nCookie deletion rates are hard to estimate. Using a panel \nof 400,000 home PCs, comScore estimated that 31 percent of \nU.S. Internet users cleared their first -party cookies during a \nmonth [24]. Anirban Dasgupta etal. [25] showed similar \nlevels of cookie clearing (25% -33% monthly, depending on \ngeography) based on the Yahoo toolbar. Such rates imply \nthat long studies (e.g., months), where users are identified \nbased on cookies could have a selection bi as problem – as',
'both directions: when we show fewer ads, users increase \nengagement, and when we show more ads, users decrease \nengagement or abandon. If users abandon at different rates \nbetween Control and Treatment, the remaining surviving \npopulation is different, and the conclusions can be \ncompletely wrong. For example, the users generating the \nmost revenue may get annoyed with more ads and abandon, \nleaving a surviving population with lower Revenue/user. \n3. Not taking into account selection bias [34]. \n In an online longitudinal study that relies on cookies for \nidentification, there is likely to be a significant attrition due',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.5 |
| cosine_accuracy@3 | 0.7419 |
| cosine_accuracy@5 | 0.7903 |
| cosine_accuracy@10 | 0.8629 |
| cosine_precision@1 | 0.5 |
| cosine_precision@3 | 0.2473 |
| cosine_precision@5 | 0.1581 |
| cosine_precision@10 | 0.0863 |
| cosine_recall@1 | 0.5 |
| cosine_recall@3 | 0.7419 |
| cosine_recall@5 | 0.7903 |
| cosine_recall@10 | 0.8629 |
| cosine_ndcg@10 | 0.6895 |
| cosine_mrr@10 | 0.6334 |
| cosine_map@100 | 0.6397 |
| sentence_0 | sentence_1 | |
|---|---|---|
| type | string | string |
| details | <ul><li>min: 5 tokens</li><li>mean: 24.08 tokens</li><li>max: 51 tokens</li></ul> | <ul><li>min: 13 tokens</li><li>mean: 123.64 tokens</li><li>max: 151 tokens</li></ul> |
| sentence_0 | sentence_1 |
|---|---|
| <code>What are some common pitfalls associated with long-term online controlled experiments as discussed by Dmitriev et al. in their IEEE Big Data paper?</code> | <code>Appears in IEEE Big Data. Paper available at http://bit.ly/expLongTerm <br> <br>Pitfalls of Long-Term Online Controlled Experiments <br>Pavel Dmitriev, Brian Frasca, Somit Gupta, Ron Kohavi, Garnet Vaz <br>Analysis and Experimentation <br>Microsoft Corporation <br>Redmond, WA 98052, USA <br>{padmitri,brianfra,sogupta,ronnyk,gavaz}@microsoft.com <br> <br> <br>Abstract—Online controlled experiments (e.g., A/B tests) are <br>now regularly used to guide product development and <br>accelerate innovation in software. Product ideas are evaluated</code> |
| <code>What are online controlled experiments used for in product development according to the context?</code> | <code>Analysis and Experimentation <br>Microsoft Corporation <br>Redmond, WA 98052, USA <br>{padmitri,brianfra,sogupta,ronnyk,gavaz}@microsoft.com <br> <br> <br>Abstract—Online controlled experiments (e.g., A/B tests) are <br>now regularly used to guide product development and <br>accelerate innovation in software. Product ideas are evaluated <br>as scientific hypotheses, and tested on web sit es, mobile <br>applications, desktop applications, services, and operating <br>system features. <br>One of the key challenges for organizations that run <br>controlled experiments is to select an Overall Evaluation</code> |
| <code>What is one of the key challenges organizations face when running online controlled experiments?</code> | <code>Abstract—Online controlled experiments (e.g., A/B tests) are <br>now regularly used to guide product development and <br>accelerate innovation in software. Product ideas are evaluated <br>as scientific hypotheses, and tested on web sit es, mobile <br>applications, desktop applications, services, and operating <br>system features. <br>One of the key challenges for organizations that run <br>controlled experiments is to select an Overall Evaluation <br>Criterion (OEC), i.e., the criterion by which to evaluat e the <br>different variants. The difficulty is that short -term changes to <br>metrics may not predict the long-term impact of a change. For</code> |
{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
1024,
768,
512,
256,
128
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": -1
}
eval_strategy: stepsper_device_train_batch_size: 16per_device_eval_batch_size: 16num_train_epochs: 100multi_dataset_batch_sampler: round_robinoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: stepsprediction_loss_only: Trueper_device_train_batch_size: 16per_device_eval_batch_size: 16per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 1eval_accumulation_steps: Nonetorch_empty_cache_steps: Nonelearning_rate: 5e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1num_train_epochs: 100max_steps: -1lr_scheduler_type: linearlr_scheduler_kwargs: {}warmup_ratio: 0.0warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Falsefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Falseignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}tp_size: 0fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torchoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Nonehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseinclude_for_metrics: []eval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters:auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falseuse_liger_kernel: Falseeval_use_gather_object: Falseaverage_tokens_across_devices: Falseprompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: round_robin| Epoch | Step | Training Loss | cosine_ndcg@10 |
|---|---|---|---|
| 1.0 | 16 | - | 0.6381 |
| 2.0 | 32 | - | 0.6800 |
| 3.0 | 48 | - | 0.6948 |
| 3.125 | 50 | - | 0.6967 |
| 4.0 | 64 | - | 0.6999 |
| 5.0 | 80 | - | 0.7013 |
| 6.0 | 96 | - | 0.6831 |
| 6.25 | 100 | - | 0.6885 |
| 7.0 | 112 | - | 0.6893 |
| 8.0 | 128 | - | 0.7012 |
| 9.0 | 144 | - | 0.6893 |
| 9.375 | 150 | - | 0.6900 |
| 10.0 | 160 | - | 0.6860 |
| 11.0 | 176 | - | 0.6840 |
| 12.0 | 192 | - | 0.7025 |
| 12.5 | 200 | - | 0.6775 |
| 13.0 | 208 | - | 0.6786 |
| 14.0 | 224 | - | 0.7115 |
| 15.0 | 240 | - | 0.7309 |
| 15.625 | 250 | - | 0.7355 |
| 16.0 | 256 | - | 0.7339 |
| 17.0 | 272 | - | 0.7246 |
| 18.0 | 288 | - | 0.7249 |
| 18.75 | 300 | - | 0.7244 |
| 19.0 | 304 | - | 0.7183 |
| 20.0 | 320 | - | 0.6952 |
| 21.0 | 336 | - | 0.6968 |
| 21.875 | 350 | - | 0.7299 |
| 22.0 | 352 | - | 0.7278 |
| 23.0 | 368 | - | 0.7122 |
| 24.0 | 384 | - | 0.6990 |
| 25.0 | 400 | - | 0.6857 |
| 26.0 | 416 | - | 0.6824 |
| 27.0 | 432 | - | 0.6905 |
| 28.0 | 448 | - | 0.6494 |
| 28.125 | 450 | - | 0.6461 |
| 29.0 | 464 | - | 0.6747 |
| 30.0 | 480 | - | 0.6761 |
| 31.0 | 496 | - | 0.6958 |
| 31.25 | 500 | 0.8459 | 0.6855 |
| 32.0 | 512 | - | 0.6940 |
| 33.0 | 528 | - | 0.7156 |
| 34.0 | 544 | - | 0.7284 |
| 34.375 | 550 | - | 0.7281 |
| 35.0 | 560 | - | 0.7288 |
| 36.0 | 576 | - | 0.6940 |
| 37.0 | 592 | - | 0.6992 |
| 37.5 | 600 | - | 0.6948 |
| 38.0 | 608 | - | 0.6991 |
| 39.0 | 624 | - | 0.6931 |
| 40.0 | 640 | - | 0.6893 |
| 40.625 | 650 | - | 0.6905 |
| 41.0 | 656 | - | 0.6960 |
| 42.0 | 672 | - | 0.7133 |
| 43.0 | 688 | - | 0.7137 |
| 43.75 | 700 | - | 0.7055 |
| 44.0 | 704 | - | 0.7077 |
| 45.0 | 720 | - | 0.7215 |
| 46.0 | 736 | - | 0.7056 |
| 46.875 | 750 | - | 0.7030 |
| 47.0 | 752 | - | 0.6996 |
| 48.0 | 768 | - | 0.6985 |
| 49.0 | 784 | - | 0.6878 |
| 50.0 | 800 | - | 0.7042 |
| 51.0 | 816 | - | 0.7097 |
| 52.0 | 832 | - | 0.7067 |
| 53.0 | 848 | - | 0.6977 |
| 53.125 | 850 | - | 0.6972 |
| 54.0 | 864 | - | 0.6999 |
| 55.0 | 880 | - | 0.6963 |
| 56.0 | 896 | - | 0.7006 |
| 56.25 | 900 | - | 0.6971 |
| 57.0 | 912 | - | 0.7008 |
| 58.0 | 928 | - | 0.7051 |
| 59.0 | 944 | - | 0.7034 |
| 59.375 | 950 | - | 0.7031 |
| 60.0 | 960 | - | 0.6978 |
| 61.0 | 976 | - | 0.6923 |
| 62.0 | 992 | - | 0.6966 |
| 62.5 | 1000 | 0.2227 | 0.6989 |
| 63.0 | 1008 | - | 0.6941 |
| 64.0 | 1024 | - | 0.7020 |
| 65.0 | 1040 | - | 0.7024 |
| 65.625 | 1050 | - | 0.7050 |
| 66.0 | 1056 | - | 0.7079 |
| 67.0 | 1072 | - | 0.7059 |
| 68.0 | 1088 | - | 0.7200 |
| 68.75 | 1100 | - | 0.7207 |
| 69.0 | 1104 | - | 0.7177 |
| 70.0 | 1120 | - | 0.7008 |
| 71.0 | 1136 | - | 0.6948 |
| 71.875 | 1150 | - | 0.6883 |
| 72.0 | 1152 | - | 0.6858 |
| 73.0 | 1168 | - | 0.6894 |
| 74.0 | 1184 | - | 0.6961 |
| 75.0 | 1200 | - | 0.6934 |
| 76.0 | 1216 | - | 0.6894 |
| 77.0 | 1232 | - | 0.6830 |
| 78.0 | 1248 | - | 0.6892 |
| 78.125 | 1250 | - | 0.6903 |
| 79.0 | 1264 | - | 0.6903 |
| 80.0 | 1280 | - | 0.6911 |
| 81.0 | 1296 | - | 0.7021 |
| 81.25 | 1300 | - | 0.7032 |
| 82.0 | 1312 | - | 0.7032 |
| 83.0 | 1328 | - | 0.6984 |
| 84.0 | 1344 | - | 0.6872 |
| 84.375 | 1350 | - | 0.6863 |
| 85.0 | 1360 | - | 0.6863 |
| 86.0 | 1376 | - | 0.6879 |
| 87.0 | 1392 | - | 0.6909 |
| 87.5 | 1400 | - | 0.6910 |
| 88.0 | 1408 | - | 0.6911 |
| 89.0 | 1424 | - | 0.6894 |
| 90.0 | 1440 | - | 0.6892 |
| 90.625 | 1450 | - | 0.6952 |
| 91.0 | 1456 | - | 0.6954 |
| 92.0 | 1472 | - | 0.6924 |
| 93.0 | 1488 | - | 0.6895 |
| 93.75 | 1500 | 0.1783 | 0.6895 |
| 94.0 | 1504 | - | 0.6895 |
| 95.0 | 1520 | - | 0.6895 |
| 96.0 | 1536 | - | 0.6895 |
| 96.875 | 1550 | - | 0.6895 |
| 97.0 | 1552 | - | 0.6895 |
| 98.0 | 1568 | - | 0.6895 |
| 99.0 | 1584 | - | 0.6895 |
| 100.0 | 1600 | - | 0.6895 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->