embeddinggemma-300m-custom-vi

EmbeddingGemma-300m fine-tuned on custom retrieval dataset

This is a sentence-transformers model finetuned from google/embeddinggemma-300m on the json dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: google/embeddinggemma-300m <!-- at revision 57c266a740f537b4dc058e1b0cda161fd15afa75 -->
• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • json
• Language: vi
• License: gemma

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 2048, 'do_lower_case': False, 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity'})
  (4): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("anhtvcengroup1/embeddinggemma-300m-custom-vi")
# Run inference
queries = [
    "Which planet is known as the Red Planet?",
]
documents = [
    "Venus is often called Earth's twin because of its similar size and proximity.",
    'Mars, known for its reddish appearance, is often referred to as the Red Planet.',
    'Saturn, famous for its rings, is sometimes mistaken for the Red Planet.',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[0.3012, 0.6364, 0.4891]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Datasets: custom-eval and custom-test
• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

json

• Dataset: json
• Size: 8 training samples
• Columns: <code>question</code> and <code>passage_text</code>
• Approximate statistics based on the first 8 samples:

  	question	passage_text
  type	string	string
  details	<ul><li>min: 15 tokens</li><li>mean: 17.0 tokens</li><li>max: 19 tokens</li></ul>	<ul><li>min: 41 tokens</li><li>mean: 52.75 tokens</li><li>max: 72 tokens</li></ul>

• Samples:

  question	passage_text
  <code>Quy định về bảo vệ dữ liệu cá nhân hiện nay là văn bản nào?</code>	<code>Nghị định số 13/2023/NĐ-CP ngày 17/4/2023 của Chính phủ về bảo vệ dữ liệu cá nhân quy định chi tiết về nguyên tắc xử lý dữ liệu, quyền và nghĩa vụ của chủ thể dữ liệu, trách nhiệm của bên xử lý dữ liệu...</code>
  <code>Căn cứ pháp lý về bảo vệ dữ liệu cá nhân là gì?</code>	<code>Nghị định số 13/2023/NĐ-CP về bảo vệ dữ liệu cá nhân là văn bản pháp lý chuyên ngành đầu tiên quy định toàn diện về bảo vệ dữ liệu cá nhân tại Việt Nam.</code>
  <code>Làm căn cước công dân gắn chip lần đầu ở đâu?</code>	<code>Công dân thực hiện thủ tục cấp thẻ Căn cước công dân gắn chip lần đầu tại Cơ quan quản lý căn cước công dân nơi công dân đăng ký thường trú hoặc tạm trú.</code>

• Loss: <code>CachedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 8,
      "gather_across_devices": false
  }
  

Evaluation Dataset

json

• Dataset: json
• Size: 5 evaluation samples
• Columns: <code>question</code> and <code>passage_text</code>
• Approximate statistics based on the first 5 samples:

  	question	passage_text
  type	string	string
  details	<ul><li>min: 14 tokens</li><li>mean: 19.4 tokens</li><li>max: 23 tokens</li></ul>	<ul><li>min: 36 tokens</li><li>mean: 48.6 tokens</li><li>max: 59 tokens</li></ul>

• Samples:

  question	passage_text
  <code>Nghị định nào quy định chi tiết về bảo vệ dữ liệu cá nhân?</code>	<code>Nghị định số 13/2023/NĐ-CP của Chính phủ quy định về bảo vệ dữ liệu cá nhân, bao gồm nguyên tắc xử lý dữ liệu, phân loại dữ liệu cá nhân cơ bản và dữ liệu cá nhân nhạy cảm.</code>
  <code>Thủ tục cấp lại căn cước công dân khi bị mất thực hiện ở đâu?</code>	<code>Người bị mất thẻ Căn cước công dân thực hiện thủ tục cấp lại tại Cơ quan quản lý căn cước công dân nơi thường trú hoặc nơi tạm trú.</code>
  <code>Đơn vị sử dụng ngân sách phải công khai quyết toán ngân sách bằng hình thức nào?</code>	<code>Đơn vị sử dụng ngân sách phải công khai quyết toán ngân sách bằng hình thức niêm yết công khai tại trụ sở, đăng tải trên cổng thông tin điện tử của đơn vị (nếu có) và các hình thức khác theo quy định.</code>

• Loss: <code>CachedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "mini_batch_size": 8,
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• prompts: {'question': 'task: search result | query: ', 'passage_text': 'title: none | text: '}
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 256
• per_device_eval_batch_size: 256
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• project: huggingface
• trackio_space_id: trackio
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: no
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: True
• prompts: {'question': 'task: search result | query: ', 'passage_text': 'title: none | text: '}
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• Python: 3.12.3
• Sentence Transformers: 5.1.2
• Transformers: 4.57.1
• PyTorch: 2.9.1+cu128
• Accelerate: 1.12.0
• Datasets: 4.4.1
• Tokenizers: 0.22.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

CachedMultipleNegativesRankingLoss

@misc{gao2021scaling,
    title={Scaling Deep Contrastive Learning Batch Size under Memory Limited Setup},
    author={Luyu Gao and Yunyi Zhang and Jiawei Han and Jamie Callan},
    year={2021},
    eprint={2101.06983},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->