Downloads ยท 30 days
5
5% of all-time downloads
chelleboyer/llm-mm-good-309e6f79-505b-4c23-8452-37cc854e67df
llm-mm-good-309e6f79-505b-4c23-8452-37cc854e67df is a sentence similarity model from chelleboyer. Use it when you need a score for how close two texts are. It is set up for sentence-transformers.
This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantiโฆ
Downloads ยท 30 days
5
5% of all-time downloads
All-time downloads
91
Public
Parameters
334M
1.3 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors1.3 GB ยท 100%
From the Hugging Face model README
This is a sentence-transformers model finetuned from Snowflake/snowflake-arctic-embed-l. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.
SentenceTransformer(
(0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel
(1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
(2): Normalize()
)
First install the Sentence Transformers library:
pip install -U sentence-transformers
Then you can load this model and run inference.
from sentence_transformers import SentenceTransformer
# Download from the ๐ค Hub
model = SentenceTransformer("chelleboyer/llm-mm-good-309e6f79-505b-4c23-8452-37cc854e67df")
# Run inference
sentences = [
'What metrics does the LLMS (2025) framework introduce to investigate position bias in pairwise comparisons?',
'Recent studies have further examined position bias in the LLMs-as-judges context.\nFor instance, a framework\xa0(LLMS, 2025) is proposed to investigate position bias in pairwise comparisons, introducing metrics such as repetition stability, position consistency, and preference fairness to better understand how positions affect LLM judgments.\nAnother study\xa0(Zheng et\xa0al., 2023a) explores the limitations of LLMs-as-judges, including position biases, and verifies agreement between LLM judgments and human preferences across multiple benchmarks.\nThese findings underscore the need for robust debiasing strategies to enhance the fairness and reliableness of LLMs-as-judges.',
'Overconfidence bias\xa0(Khan et\xa0al., 2024; Jung et\xa0al., 2024) in the context of LLMs-as-judges refers to the tendency of models to exhibit an inflated level of confidence in their judgments, often resulting in overly assertive evaluations that may not accurately reflect the true reliability of the answer. This bias is particularly concerning in evaluative contexts, as it can lead LLMs-as-judges to overstate the correctness of certain outputs, compromising the objectivity and dependability of assessments.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]
# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]
<!--
### Direct Usage (Transformers)
<details><summary>Click to see the direct usage in Transformers</summary>
</details>
-->
<!--
### Downstream Usage (Sentence Transformers)
You can finetune this model on your own dataset.
<details><summary>Click to expand</summary>
</details>
-->
<!--
### Out-of-Scope Use
*List how the model may foreseeably be misused and address what users ought not to do with the model.*
-->
| Metric | Value |
|---|---|
| cosine_accuracy@1 | 0.93 |
| cosine_accuracy@3 | 0.99 |
| cosine_accuracy@5 | 1.0 |
| cosine_accuracy@10 | 1.0 |
| cosine_precision@1 | 0.93 |
| cosine_precision@3 | 0.33 |
| cosine_precision@5 | 0.2 |
| cosine_precision@10 | 0.1 |
| cosine_recall@1 | 0.93 |
| cosine_recall@3 | 0.99 |
| cosine_recall@5 | 1.0 |
| cosine_recall@10 | 1.0 |
| cosine_ndcg@10 | 0.9704 |
| cosine_mrr@10 | 0.9603 |
| cosine_map@100 | 0.9603 |
| sentence_0 | sentence_1 | |
|---|---|---|
| type | string | string |
| details | <ul><li>min: 5 tokens</li><li>mean: 23.12 tokens</li><li>max: 72 tokens</li></ul> | <ul><li>min: 3 tokens</li><li>mean: 132.04 tokens</li><li>max: 306 tokens</li></ul> |
| sentence_0 | sentence_1 |
|---|---|
| <code>What are the main components of the evaluation function ( E ) as described in the preliminaries section?</code> | <code>LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>1 Introduction<br><br>2 PRELIMINARIES<br><br>2.1 Evaluation Function E๐ธEitalic_E<br><br>2.2 Evaluation Input<br><br>2.2.1 Evaluation Type ๐ฏ๐ฏ\mathcal{T}caligraphic_T<br>2.2.2 Evaluation Criteria ๐๐\mathcal{C}caligraphic_C.<br>2.2.3 Evaluation References โโ\mathcal{R}caligraphic_R.<br><br><br>2.3 Evaluation Output<br><br><br><br>3 Functionality<br><br><br>3.1 Performance Evaluation<br><br>3.1.1 Responses Evaluation<br>3.1.2 Model Evaluation<br><br><br><br>3.2 Model Enhancement<br><br>3.2.1 Reward Modeling During Training<br>3.2.2 Acting as Verifier During Inference<br>3.2.3 Feedback for Refinement<br><br><br><br>3.3 Data Construction<br><br>3.3.1 Data Annotation<br>3.3.2 Data Synthesize<br><br><br><br><br><br>4 Methodology</code> |
| <code>How do LLMs contribute to model enhancement according to the functionalities outlined in the survey?</code> | <code>LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>1 Introduction<br><br>2 PRELIMINARIES<br><br>2.1 Evaluation Function E๐ธEitalic_E<br><br>2.2 Evaluation Input<br><br>2.2.1 Evaluation Type ๐ฏ๐ฏ\mathcal{T}caligraphic_T<br>2.2.2 Evaluation Criteria ๐๐\mathcal{C}caligraphic_C.<br>2.2.3 Evaluation References โโ\mathcal{R}caligraphic_R.<br><br><br>2.3 Evaluation Output<br><br><br><br>3 Functionality<br><br><br>3.1 Performance Evaluation<br><br>3.1.1 Responses Evaluation<br>3.1.2 Model Evaluation<br><br><br><br>3.2 Model Enhancement<br><br>3.2.1 Reward Modeling During Training<br>3.2.2 Acting as Verifier During Inference<br>3.2.3 Feedback for Refinement<br><br><br><br>3.3 Data Construction<br><br>3.3.1 Data Annotation<br>3.3.2 Data Synthesize<br><br><br><br><br><br>4 Methodology</code> |
| <code>What are the different approaches discussed under the Single-LLM System methodology?</code> | <code>4 Methodology<br><br><br>4.1 Single-LLM System<br><br>4.1.1 Prompt-based<br>4.1.2 Tuning-based<br>4.1.3 Post-processing<br><br><br><br>4.2 Multi-LLM System<br><br>4.2.1 Communication<br>4.2.2 Aggregation<br><br><br>4.3 Human-AI Collaboration System<br><br><br><br>5 Application<br><br>5.1 General<br>5.2 Multimodal<br>5.3 Medical<br>5.4 Legal<br>5.5 Financial<br>5.6 Education<br>5.7 Information Retrieval<br><br>5.8 Others<br><br>5.8.1 Soft Engineering<br>5.8.2 Biology<br>5.8.3 Social Science<br><br><br><br><br><br>6 Meta-evaluation<br><br><br>6.1 Benchmarks<br><br>6.1.1 Code Generation<br>6.1.2 Machine Translation<br>6.1.3 Text Summarization<br>6.1.4 Dialogue Generation<br>6.1.5 Automatic Story Generation<br>6.1.6 Values Alignment<br>6.1.7 Recommendation<br>6.1.8 Search<br>6.1.9 Comprehensive Data<br><br><br><br>6.2 Metric</code> |
{
"loss": "MultipleNegativesRankingLoss",
"matryoshka_dims": [
768,
512,
256,
128,
64
],
"matryoshka_weights": [
1,
1,
1,
1,
1
],
"n_dims_per_step": -1
}
eval_strategy: stepsper_device_train_batch_size: 50per_device_eval_batch_size: 50num_train_epochs: 10multi_dataset_batch_sampler: round_robinoverwrite_output_dir: Falsedo_predict: Falseeval_strategy: stepsprediction_loss_only: Trueper_device_train_batch_size: 50per_device_eval_batch_size: 50per_gpu_train_batch_size: Noneper_gpu_eval_batch_size: Nonegradient_accumulation_steps: 1eval_accumulation_steps: Nonetorch_empty_cache_steps: Nonelearning_rate: 5e-05weight_decay: 0.0adam_beta1: 0.9adam_beta2: 0.999adam_epsilon: 1e-08max_grad_norm: 1num_train_epochs: 10max_steps: -1lr_scheduler_type: linearlr_scheduler_kwargs: {}warmup_ratio: 0.0warmup_steps: 0log_level: passivelog_level_replica: warninglog_on_each_node: Truelogging_nan_inf_filter: Truesave_safetensors: Truesave_on_each_node: Falsesave_only_model: Falserestore_callback_states_from_checkpoint: Falseno_cuda: Falseuse_cpu: Falseuse_mps_device: Falseseed: 42data_seed: Nonejit_mode_eval: Falseuse_ipex: Falsebf16: Falsefp16: Falsefp16_opt_level: O1half_precision_backend: autobf16_full_eval: Falsefp16_full_eval: Falsetf32: Nonelocal_rank: 0ddp_backend: Nonetpu_num_cores: Nonetpu_metrics_debug: Falsedebug: []dataloader_drop_last: Falsedataloader_num_workers: 0dataloader_prefetch_factor: Nonepast_index: -1disable_tqdm: Falseremove_unused_columns: Truelabel_names: Noneload_best_model_at_end: Falseignore_data_skip: Falsefsdp: []fsdp_min_num_params: 0fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}tp_size: 0fsdp_transformer_layer_cls_to_wrap: Noneaccelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}deepspeed: Nonelabel_smoothing_factor: 0.0optim: adamw_torchoptim_args: Noneadafactor: Falsegroup_by_length: Falselength_column_name: lengthddp_find_unused_parameters: Noneddp_bucket_cap_mb: Noneddp_broadcast_buffers: Falsedataloader_pin_memory: Truedataloader_persistent_workers: Falseskip_memory_metrics: Trueuse_legacy_prediction_loop: Falsepush_to_hub: Falseresume_from_checkpoint: Nonehub_model_id: Nonehub_strategy: every_savehub_private_repo: Nonehub_always_push: Falsegradient_checkpointing: Falsegradient_checkpointing_kwargs: Noneinclude_inputs_for_metrics: Falseinclude_for_metrics: []eval_do_concat_batches: Truefp16_backend: autopush_to_hub_model_id: Nonepush_to_hub_organization: Nonemp_parameters:auto_find_batch_size: Falsefull_determinism: Falsetorchdynamo: Noneray_scope: lastddp_timeout: 1800torch_compile: Falsetorch_compile_backend: Nonetorch_compile_mode: Noneinclude_tokens_per_second: Falseinclude_num_input_tokens_seen: Falseneftune_noise_alpha: Noneoptim_target_modules: Nonebatch_eval_metrics: Falseeval_on_start: Falseuse_liger_kernel: Falseeval_use_gather_object: Falseaverage_tokens_across_devices: Falseprompts: Nonebatch_sampler: batch_samplermulti_dataset_batch_sampler: round_robin| Epoch | Step | cosine_ndcg@10 |
|---|---|---|
| 1.0 | 27 | 0.9697 |
| 1.8519 | 50 | 0.9788 |
| 2.0 | 54 | 0.9775 |
| 3.0 | 81 | 0.9741 |
| 3.7037 | 100 | 0.9791 |
| 4.0 | 108 | 0.9741 |
| 5.0 | 135 | 0.9782 |
| 5.5556 | 150 | 0.9782 |
| 6.0 | 162 | 0.9782 |
| 7.0 | 189 | 0.9782 |
| 7.4074 | 200 | 0.9741 |
| 8.0 | 216 | 0.9741 |
| 9.0 | 243 | 0.9704 |
| 9.2593 | 250 | 0.9704 |
| 10.0 | 270 | 0.9704 |
@inproceedings{reimers-2019-sentence-bert,
title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
author = "Reimers, Nils and Gurevych, Iryna",
booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
month = "11",
year = "2019",
publisher = "Association for Computational Linguistics",
url = "https://arxiv.org/abs/1908.10084",
}
@misc{kusupati2024matryoshka,
title={Matryoshka Representation Learning},
author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
year={2024},
eprint={2205.13147},
archivePrefix={arXiv},
primaryClass={cs.LG}
}
@misc{henderson2017efficient,
title={Efficient Natural Language Response Suggestion for Smart Reply},
author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
year={2017},
eprint={1705.00652},
archivePrefix={arXiv},
primaryClass={cs.CL}
}
<!--
## Glossary
*Clearly define terms in order to be accessible across audiences.*
-->
<!--
## Model Card Authors
*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
-->
<!--
## Model Card Contact
*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
-->