distilroberta-base-manglish-sentence-transformer

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2 <!-- at revision c9745ed1d9f207416be6d2e6f8de32d1f16199bf -->
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("along26/distilroberta-base-manglish-sentence-transformer")
# Run inference
sentences = [
    "To construct a factor analysis model, we will follow these steps:\n\n1. Collect the data: In this case, we have the survey data from 50 students rating their satisfaction level on a scale of 1 to 10 for three aspects: academics (A), social life (S), and campus facilities (F).\n\n2. Create a correlation matrix: Calculate the correlation coefficients between each pair of variables (A, S, and F) using the collected data. The correlation matrix will look like this:\n\n```\n      A      S      F\nA  1.000  r(A,S) r(A,F)\nS  r(A,S) 1.000  r(S,F)\nF  r(A,F) r(S,F) 1.000\n```\n\nwhere r(A,S), r(A,F), and r(S,F) are the correlation coefficients between academics and social life, academics and campus facilities, and social life and campus facilities, respectively.\n\n3. Determine the number of factors: To decide how many underlying factors can explain the correlations, we can use various methods, such as the Kaiser criterion (eigenvalues greater than 1), scree plot, or parallel analysis. For this example, let's use the Kaiser criterion.\n\n4. Extract the factors: Perform a factor analysis using a software or statistical tool (e.g., R, Python, SPSS) to extract the factors based on the chosen number of factors. The software will provide factor loadings, which represent the correlation between the original variables and the extracted factors.\n\n5. Interpret the factors: Analyze the factor loadings to understand the meaning of the extracted factors. If the loadings are high (e.g., greater than 0.5) for a particular variable, it means that the variable is strongly correlated with that factor.\n\nLet's assume we have calculated the correlation coefficients and performed the factor analysis. The results show that there is one underlying factor that explains the strong correlations between the three aspects (academics, social life, and campus facilities). This factor could be interpreted as the overall satisfaction with the college experience.\n\nIn conclusion, based on the factor analysis model, the three aspects of college experience (academics, social life, and campus facilities) are strongly correlated with each other, and one underlying factor can explain these correlations, which might represent the overall satisfaction with the college experience.",
    'U',
    'Suppose you have a dataset with 5 variables and you want to reduce the number of variables to 2. Using factor analysis, determine the two most important factors that explain the variation in the data. Include the factor loadings for each variable on each factor.',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[ 1.0000, -1.0000,  1.0000],
#         [-1.0000,  1.0000, -1.0000],
#         [ 1.0000, -1.0000,  1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 424,311 training samples
• Columns: <code>sentence_0</code>, <code>sentence_1</code>, and <code>sentence_2</code>
• Approximate statistics based on the first 1000 samples:

  	sentence_0	sentence_1	sentence_2
  type	string	string	string
  details	<ul><li>min: 9 tokens</li><li>mean: 150.71 tokens</li><li>max: 256 tokens</li></ul>	<ul><li>min: 3 tokens</li><li>mean: 3.0 tokens</li><li>max: 3 tokens</li></ul>	<ul><li>min: 11 tokens</li><li>mean: 147.14 tokens</li><li>max: 256 tokens</li></ul>

• Samples:

  sentence_0	sentence_1	sentence_2
  <code>Sebagai pengguna, saya ingin tahu sama ada terdapat sebarang usaha atau tindakan yang diambil oleh kerajaan atau badan kehakiman Malaysia untuk menangani kebimbangan ini tentang kadar perbicaraan yang perlahan dan meningkatkan kepercayaan orang ramai dalam proses tersebut?</code>	<code>E</code>	<code>Mengapa kadar perbicaraan dan prosiding undang-undang berkaitan 1MDB yang perlahan mencetuskan kekecewaan dan kritikan dalam kalangan rakyat Malaysia?</code>
  <code>Pertama, mari kita layan 2 orang yang mesti duduk bersama sebagai satu unit. Jadi sekarang, kami mempunyai 4 unit untuk disusun (pasangan dan 3 individu yang lain).<br><br>Terdapat 4! (4 faktorial) cara menyusun 4 unit ini, iaitu 4! = 4 × 3 × 2 × 1 = 24 cara.<br><br>Kini, dalam pasangan itu, terdapat 2! (2 faktorial) cara untuk menyusun 2 orang, iaitu 2! = 2 × 1 = 2 cara.<br><br>Jadi, untuk mencari jumlah cara untuk menempatkan 5 orang, kita darabkan bilangan cara untuk menyusun 4 unit dengan bilangan cara untuk mengatur orang dalam pasangan:<br><br>Jumlah cara = 24 (susunan 4 unit) × 2 (susunan dalam pasangan) = 48 cara.</code>	<code>E</code>	<code>Apabila mengatur orang di sekitar meja bulat, kita biasanya menganggap satu orang sebagai titik rujukan untuk mengelakkan mengira susunan yang sama beberapa kali disebabkan oleh putaran. Oleh itu, kami mempunyai 7 orang kawan yang tinggal untuk mengatur meja.<br><br>Bilangan cara untuk mengatur 7 rakan ini ialah 7! (7 faktorial), iaitu:<br><br>7! = 7 × 6 × 5 × 4 × 3 × 2 × 1 = 5,040<br><br>Oleh itu, terdapat 5,040 cara yang berbeza untuk duduk 8 rakan di sekitar meja bulat.</code>
  <code>How have Malaysia's international partners and institutions responded to the 1MDB scandal and Najib Razak's prosecution, and what impact has this had on the country's reputation and standing in the global community?</code>	<code>B</code>	<code>How has the international community responded to the 1MDB scandal and the allegations of corruption against Najib Razak, and what impact has this had on Malaysia's reputation as a whole?</code>

• Loss: <code>TripletLoss</code> with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.EUCLIDEAN",
      "triplet_margin": 5
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• multi_dataset_batch_sampler: round_robin

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: no
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 5e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: round_robin
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• Python: 3.12.12
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.8.0+cu126
• Accelerate: 1.11.0
• Datasets: 4.0.0
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->