yue-embed

Bert base fine-tuned with Cantonese and English mixed STS dataset

This is a sentence-transformers model finetuned from hon9kon9ize/bert-large-cantonese-sts. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: hon9kon9ize/bert-large-cantonese-sts <!-- at revision 8c3360b125d5048e87da79bb40980f5229a100dc -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 1024 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown -->

• Language: yue
• License: apache-2.0

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("hon9kon9ize/yue-embed")
# Run inference
sentences = [
    'query: when did england change from fahrenheit to celsius',
    'document: Metrication in the United Kingdom Adopting the metric system was discussed in Parliament as early as 1818 and some industries and even some government agencies had metricated, or were in the process of metricating by the mid 1960s. A formal government policy to support metrication was agreed by 1965. This policy, initiated in response to requests from industry, was to support voluntary metrication, with costs picked up where they fell. In 1969 the government created the Metrication Board as a quango to promote and coordinate metrication. In 1978, after some carpet retailers reverted to pricing by the square yard rather than the square metre, government policy shifted, and they started issuing orders making metrication mandatory in certain sectors. In 1980 government policy shifted again to prefer voluntary metrication, and the Metrication Board was abolished. By the time the Metrication Board was wound up, all the economic sectors that fell within its remit except road signage and parts of the retail trade sector had metricated.',
    "document: Periodic table Importantly, the organization of the periodic table can be utilized to derive relationships between various element properties, but also predicted chemical properties and behaviours of undiscovered or newly synthesized elements. Russian chemist Dmitri Mendeleev was first to publish a recognizable periodic table in 1869, developed mainly to illustrate periodic trends of the then-known elements. He also predicted some properties of unidentified elements that were expected to fill gaps within this table. Most of his forecasts proved to be correct. Mendeleev's idea has been slowly expanded and refined with the discovery or synthesis of further new elements and by developing new theoretical models to explain chemical behaviour. The modern periodic table now provides a useful framework for analyzing chemical reactions, and continues to be widely adopted in chemistry, nuclear physics and other sciences.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Datasets: NanoClimateFEVER, NanoDBPedia, NanoFEVER, NanoFiQA2018, NanoHotpotQA, NanoMSMARCO, NanoNFCorpus, NanoNQ, NanoQuoraRetrieval, NanoSCIDOCS, NanoArguAna, NanoSciFact and NanoTouche2020
• Evaluated with <code>InformationRetrievalEvaluator</code>

Nano BEIR

• Dataset: NanoBEIR_mean
• Evaluated with <code>NanoBEIREvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 129,371 training samples
• Columns: <code>query</code> and <code>answer</code>
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	<ul><li>min: 13 tokens</li><li>mean: 22.58 tokens</li><li>max: 134 tokens</li></ul>	<ul><li>min: 7 tokens</li><li>mean: 169.6 tokens</li><li>max: 512 tokens</li></ul>

• Samples:

  query	answer
  <code>query: hotel and restaurant employees and bartenders international union</code>	<code>document: Hotel Employees and Restaurant Employees Union The Hotel Employees and Restaurant Employees Union (HERE) was a United States labor union representing workers of the hospitality industry, formed in 1891. In 2004, HERE merged with the Union of Needletrades, Industrial, and Textile Employees (UNITE) to form UNITE HERE. HERE notably organized the staff of Yale University in 1984. Other major employers that contracted with this union included several large casinos (Harrah's, Caesars Palace, and Wynn Resorts); hotels (Hilton, Hyatt and Starwood), and Walt Disney World. HERE was affiliated with the AFL-CIO.</code>
  <code>query: 多肢离断伤的并发症是什么？</code>	<code>document: 失血性休克；血循环危象；急性肾功能衰竭</code>
  <code>query: who is the father of kelly taylor's son on 90210</code>	<code>document: Kelly Taylor (90210) In 2008, Kelly Taylor returned in the spin-off 90210, now working as a guidance counselor at her alma mater West Beverly Hills High School. It was revealed that in the intervening years, she attained a master's degree and had a son named Sammy with Dylan. She and Dylan ended their relationship soon after. It was also revealed that West Beverly principal Harry Wilson was Kelly's neighbor growing up.[39]</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: NewModel 
    (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.01}
  

Evaluation Dataset

Unnamed Dataset

• Size: 1,000 evaluation samples
• Columns: <code>query</code> and <code>answer</code>
• Approximate statistics based on the first 1000 samples:

  	query	answer
  type	string	string
  details	<ul><li>min: 11 tokens</li><li>mean: 22.61 tokens</li><li>max: 146 tokens</li></ul>	<ul><li>min: 7 tokens</li><li>mean: 164.27 tokens</li><li>max: 512 tokens</li></ul>

• Samples:

  query	answer
  <code>query: 微创经皮肾镜手术的推荐药有些什么？</code>	<code>document: 阿司匹林</code>
  <code>query: why are the fires in ca called the thomas fires</code>	<code>document: Thomas Fire On December 4, 2017, the Thomas Fire was reported at 6:26 p.m. PST,[36] to the north of Santa Paula, near Steckel Park and Thomas Aquinas College,[3][24] after which the fire is named.[37] That night, the small brush fire exploded in size and raced through the rugged mountain terrain that lies west of Santa Paula, between Ventura and Ojai.[19][38] Officials blamed strong Santa Ana winds that gusted up to 60 miles per hour (97 km/h) for the sudden expansion.[28][39] Soon after the fire had started, a second blaze was ignited nearly 30 minutes later, about 4 miles (6.4 km) to the north in Upper Ojai at the top of Koenigstein Road.[40] According to eyewitnesses, this second fire was sparked by an explosion in the power line over the area. The second fire was rapidly expanded by the strong Santa Ana winds, and soon merged into the Thomas Fire later that night.[40]</code>
  <code>query: which mountain man rediscovered south pass and brought back important information about this trail</code>	<code>document: Jedediah Smith Jedediah Strong Smith (January 6, 1799 – May 27, 1831), was a clerk, frontiersman, hunter, trapper, author, cartographer, and explorer of the Rocky Mountains, the North American West, and the Southwest during the early 19th century. After 75 years of obscurity following his death, Smith was rediscovered as the American whose explorations led to the use of the 20-mile (32 km)-wide South Pass as the dominant point of crossing the Continental Divide for pioneers on the Oregon Trail.</code>

• Loss: <code>CachedGISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': False}) with Transformer model: NewModel 
    (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.01}
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• learning_rate: 2e-05
• num_train_epochs: 2
• warmup_ratio: 0.05
• seed: 12
• bf16: True
• prompts: {'query': 'query: ', 'answer': 'document: '}
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 128
• per_device_eval_batch_size: 128
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 2
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.05
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 12
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: {'query': 'query: ', 'answer': 'document: '}
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

<details><summary>Click to expand</summary>

The model card is truncated. Read the rest on Hugging Face.