bert-base-uncased-gooaq-unsloth-tiny

SentenceTransformer based on google-bert/bert-base-uncased

This is a sentence-transformers model finetuned from google-bert/bert-base-uncased on the gooaq dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google-bert/bert-base-uncased <!-- at revision 86b5e0934494bd15c9632b12f734a8a67f723594 -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • gooaq
• Language: en

<!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'PeftModelForFeatureExtraction'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("kwondw/bert-base-uncased-gooaq-unsloth-tiny")
# Run inference
queries = [
    'how much did it cost to build your own house?',
]
documents = [
    'According to HomeAdvisor.com, an online marketplace that connects homeowners with home improvement professionals, the average cost to build a new house is a little over $290,000. But that cost can vary greatly. They go on to report that the average cost of building a house is around $147,000 to as much as $436,000.',
    "['Connect your external hard drive to an Xbox One USB port. ... ', 'Select Format storage device.', 'Enter a name for the drive using the on-screen keyboard.', 'Select Install new things here to automatically install future games on this drive.', 'Select Format storage device.']",
    'When wearing cowboy boots, your pants should come down to the foot of the boot or a little longer. They should not be long enough to drag on the floor, however, or too short. Keep in mind that most cowboy boots have a taller heel than regular shoes, as much as one or even two inches.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.7040, 0.5361, 0.5989]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Dataset: NanoMSMARCO
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: NanoMSMARCO
• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 128 training samples
• Columns: <code>question</code> and <code>answer</code>
• Approximate statistics based on the first 128 samples:

  	question	answer
  type	string	string
  details	<ul><li>min: 8 tokens</li><li>mean: 12.4 tokens</li><li>max: 34 tokens</li></ul>	<ul><li>min: 20 tokens</li><li>mean: 61.34 tokens</li><li>max: 140 tokens</li></ul>

• Samples:

  question	answer
  <code>is trazodone good for anxiety?</code>	<code>About trazodone Trazodone is an antidepressant medicine that works to balance chemicals in the brain. It's used to treat depression, anxiety, or a combination of depression and anxiety. It can help if you're having problems like low mood, poor sleep and poor concentration.</code>
  <code>what mental disorder does hannibal lecter have?</code>	<code>Hannibal Lecter Both a talented psychiatrist and cannibalistic serial killer, the majority of Lecter's personality traits and behaviors can be categorized as evidence of ASPD. He can be specifically categorized as a malevolent antisocial due to his striking lack of remorse of guilt.</code>
  <code>what is the apple watch nike series?</code>	<code>Get connected through sport. Apple Watch Nike is the perfect running partner with the Nike Run Club app. Featuring a world of workouts, coaching and motivation to help take your fitness to the next level. And with Apple Watch Series 5, every Nike watch face is optimised for the Always-On Retina display.</code>

• Loss: <code>CachedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 8,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Evaluation Dataset

gooaq

• Dataset: gooaq at b089f72
• Size: 128 evaluation samples
• Columns: <code>question</code> and <code>answer</code>
• Approximate statistics based on the first 128 samples:

  	question	answer
  type	string	string
  details	<ul><li>min: 8 tokens</li><li>mean: 12.04 tokens</li><li>max: 21 tokens</li></ul>	<ul><li>min: 13 tokens</li><li>mean: 59.94 tokens</li><li>max: 127 tokens</li></ul>

• Samples:

  question	answer
  <code>what are the four main controls on a noise gate?</code>	<code>Noise gates usually have five main parameters: threshold, ratio (reduction), attack, hold and release. Threshold – this sets the level at which the gate opens to let the sound through.</code>
  <code>is uncut gems on amazon prime?</code>	<code>Excited for the international release of Uncut Gems on Netflix tomorrow. ... Generally, Amazon Prime Video gets A24 movies between six and seven months after they are released in cinemas, which would fit the Safdie brothers' statement that Uncut Gems will be streaming in summer 2020.</code>
  <code>how many days a week do you workout?</code>	<code>You need to be hitting the weights at least three days per week. The research says that at the very least, training a minimum of two days per week is needed to maximize muscle growth. How you structure your workouts and the amount of days you devote to strength training depends on your current fitness level.</code>

• Loss: <code>CachedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 8,
      "gather_across_devices": false,
      "directions": [
          "query_to_doc"
      ],
      "partition_mode": "joint",
      "hardness_mode": null,
      "hardness_strength": 0.0
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• num_train_epochs: 1
• learning_rate: 2e-05
• warmup_steps: 0.1
• fp16: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 8
• num_train_epochs: 1
• max_steps: -1
• learning_rate: 2e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: True
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• per_device_eval_batch_size: 8
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: True
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: None
• local_rank: -1
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Training Time

• Training: 19.8 seconds
• Evaluation: 46.1 seconds
• Total: 1.1 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.4.1
• Transformers: 5.5.0
• PyTorch: 2.10.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.3.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->