nz-research-commons-embedding-gemma

SentenceTransformer

This is a sentence-transformers model trained on the nz_research_commons_embedding_triplets_5k dataset. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for retrieval.

Model Details

Model Description

• Model Type: Sentence Transformer

<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->

• Maximum Sequence Length: 2048 tokens
• Output Dimensionality: 768 dimensions
• Similarity Function: Cosine Similarity
• Supported Modality: Text
• Training Dataset:
  • nz_research_commons_embedding_triplets_5k

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'transformer_task': 'feature-extraction', 'modality_config': {'text': {'method': 'forward', 'method_output_name': 'last_hidden_state'}}, 'module_output_name': 'token_embeddings', 'architecture': 'Gemma3TextModel'})
  (1): Pooling({'embedding_dimension': 768, 'pooling_mode': 'mean', 'include_prompt': True})
  (2): Dense({'in_features': 768, 'out_features': 3072, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (3): Dense({'in_features': 3072, 'out_features': 768, 'bias': False, 'activation_function': 'torch.nn.modules.linear.Identity', 'module_input_name': 'sentence_embedding', 'module_output_name': 'sentence_embedding'})
  (4): Normalize({})
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("dinushiTJ/nz-research-commons-embedding-gemma")
# Run inference
queries = [
    'non_maori_origin',
]
documents = [
    'title: Urban narrative: Computational linguistic interpretation of large format public participation for urban infrastructure\n\nauthors: Dyer, Mark\n\nabstract: Urban Narrative works at the interface between public participation and participatory design to support collaboration processes for urban planning and design. It applies computational linguistics to interpret large format public consultation by identifying shared interests and desired qualities for urban infrastructure services and utilities. As a proof of concept, data was used from the Christchurch public engagement initiative called ‘Share an Idea,’ where public thoughts, ideas, and opinions were expressed about the future redevelopment of Christchurch after the 2011 earthquakes. The data set was analysed to identify shared interests and desired connections between institutional, communal, or personal infrastructures with the physical urban infrastructures in terms of buildings, public places, and utilities. The data has been visualised using chord charts from the D3 JavaScript open source library to illustrate the existence of connections between soft and hard urban infrastructures along with individual contributions or stories. Lastly, the analysis was used to create an infographic design brief that compares and contrasts qualitative information from public consultation with quantitative municipal statistical data on well-being.\n\ntext: Urban narrative: Computational linguistic interpretation of large format public participation for urban infrastructure Urban Narrative works at the interface between public participation and participatory design to support collaboration processes for urban planning and design. It applies computational linguistics to interpret large format public consultation by identifying shared interests and desired qualities for urban infrastructure services and utilities. As a proof of concept, data was used from the Christchurch public engagement initiative called ‘Share an Idea,’ where public thoughts, ideas, and opinions were expressed about the future redevelopment of Christchurch after the 2011 earthquakes. The data set was analysed to identify shared interests and desired connections between institutional, communal, or personal infrastructures with the physical urban infrastructures in terms of buildings, public places, and utilities. The data has been visualised using chord charts from the D3 JavaScript open source library to illustrate the existence of connections between soft and hard urban infrastructures along with individual contributions or stories. Lastly, the analysis was used to create an infographic design brief that compares and contrasts qualitative information from public consultation with quantitative municipal statistical data on well-being.\n\nyear: 2020',
    'title: A report to iwi on the kaupapa Māori environmental outcomes and indicators kete\n\nauthors: Jefferies, Richard\n\nsubjects: New Zealand\n\nabstract: Tangata whenua in Aotearoa have been largely excluded from participation in local government  planning since colonisation, but tikanga and Māori values have for the past two decades been acknowledged in resource management and local government legislation, especially the Resource Management Act, 1991 (RMA) and Local Government Act, 2002 (LGA). For example, the RMA has provisions in over 30 sections for councils to give effect to Māori interests.\r\n\r\nIn practice, however, there is widespread concern that despite these provisions, Māori are largely excluded from local government resource management processes and their values subordinated to those of the wider community, particularly western scientific values.\r\n\r\nThis report describes research that resulted in a kaupapa Māori outcomes and indicators framework, and associated methods, that can be used by iwi to assess the quality of statutory plans and the environmental performance of councils in their rohe.\n\ntext: A report to iwi on the kaupapa Māori environmental outcomes and indicators kete Tangata whenua in Aotearoa have been largely excluded from participation in local government  planning since colonisation, but tikanga and Māori values have for the past two decades been acknowledged in resource management and local government legislation, especially the Resource Management Act, 1991 (RMA) and Local Government Act, 2002 (LGA). For example, the RMA has provisions in over 30 sections for councils to give effect to Māori interests.\r\n\r\nIn practice, however, there is widespread concern that despite these provisions, Māori are largely excluded from local government resource management processes and their values subordinated to those of the wider community, particularly western scientific values.\r\n\r\nThis report describes research that resulted in a kaupapa Māori outcomes and indicators framework, and associated methods, that can be used by iwi to assess the quality of statutory plans and the environmental performance of councils in their rohe.\n\nyear: 2009-06-30',
    'title: The issues of the criminal justice system and of resources in Aotearoa/New Zealand\n\nauthors: Toki, Valmaine\n\nabstract: Within the seven regions, recognized by the United Nations, various jurisdictions have acknowledged Indigenous rights within their respective constitutions. Although not explicit, some constitutional provisions, such as those included in the Norwegian Constitution, when read together with other articles, provide tentative opportunities for the implementation of an Indigenous legal system and an Indigenous court. Some Constitutions, such as that of Ecuador, are more explicit in providing constitutional recognition of an Indigenous legal system as well as rights to nature and, the interim Constitution of Nepal, courts for Indigenous Peoples.\n\ntext: The issues of the criminal justice system and of resources in Aotearoa/New Zealand Within the seven regions, recognized by the United Nations, various jurisdictions have acknowledged Indigenous rights within their respective constitutions. Although not explicit, some constitutional provisions, such as those included in the Norwegian Constitution, when read together with other articles, provide tentative opportunities for the implementation of an Indigenous legal system and an Indigenous court. Some Constitutions, such as that of Ecuador, are more explicit in providing constitutional recognition of an Indigenous legal system as well as rights to nature and, the interim Constitution of Nepal, courts for Indigenous Peoples.\n\nyear: 2014',
]
query_embeddings = model.encode_query(queries)
document_embeddings = model.encode_document(documents)
print(query_embeddings.shape, document_embeddings.shape)
# [1, 768] [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(query_embeddings, document_embeddings)
print(similarities)
# tensor([[ 0.7187, -0.9579, -0.3887]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Triplet

• Dataset: rc-triplet-eval
• Evaluated with <code>TripletEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

nz_research_commons_embedding_triplets_5k

• Dataset: nz_research_commons_embedding_triplets_5k at 0f97492
• Size: 5,000 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 100 samples:

  	anchor	positive	negative
  type	string	string	string
  modality	text	text	text
  details	<ul><li>min: 6 tokens</li><li>mean: 6.75 tokens</li><li>max: 8 tokens</li></ul>	<ul><li>min: 93 tokens</li><li>mean: 712.88 tokens</li><li>max: 2048 tokens</li></ul>	<ul><li>min: 51 tokens</li><li>mean: 622.91 tokens</li><li>max: 2048 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>maori_origin</code>	<code>title: KAUMĀTUATANGA Supporting School Leaders To Develop Cultural Values While Resisting The Dominance of Colonialism<br><br>authors: Nuri, Ngahuia<br><br>subjects: Kaumātua<br><br>abstract: In an education system that is asserting to the importance of Māori (indigenous people of Aotearoa) language, culture, identity and the history of Aotearoa (New Zealand) into the curriculum, there is direction from the Ministry of Education (MOE) for schools to reach out to iwi. What this looks like and how this can be achieved is not an easy task. This thesis follows the journey of three kaumātua (respected, knowledgeable elders, both female and male) working alongside the Senior Leadership Team (SLT) of a decile 10 kura auraki (mainstream primary school) where ākonga Māori (Māori students) were in the minority. The research examines what role kaumātua might have in guiding other schools to help tamariki Māori (Māori children) enjoy and achieve education success as Māori. It highlights the coming together of lea...</code>	<code>title: Studies of New Zealand Marine Organisms<br><br>authors: Till, Marisa<br><br>subjects: Natural products<br><br>abstract: The chemical study of three New Zealand marine organisms is described, along with a survey of the chemistry and biological activity of eighty-five marine organisms collected from New Zealand waters.<br>The study of the New Zealand marine bryozoan Pterocella vesiculosa has resulted in the isolation of three new compounds; pterocellin H, pterocellin I and 1-methyl-5-bromo-8-methoxy-β-carboline. These compounds were characterised using high resolution mass spectrometry, one- and two-dimensional nuclear magnetic resonance spectroscopy and X-ray crystallography. The biological activity of these compounds was investigated and a discussion of the results including a comparison with the activity of closely related compounds is also presented.<br>The crude extracts of eighty-five marine organisms were surveyed to establish their biological activity and chemical constituents. The results of thi...</code>
  <code>maori_origin</code>	<code>title: Taku Ara, Taku Mahara: Pākehā Family Experiences of Kaupapa Māori and Bilingual Education<br><br>authors: Barnes, Alexander Louis<br><br>subjects: Kaupapa Maori<br><br>abstract: Kaupapa Māori (indigenous Māori-centred philosophies) initiatives have transformed various social, cultural and public projects in the domains of governance and constitutional issues, health, education, the environment, community development and research in Aotearoa – New Zealand. As a Pākehā graduate of te kōhanga reo and kura kaupapa Māori (Māori language immersion pre-school and primary school), this research is concerned with exploring the impact of kaupapa Māori and bilingual educational initiatives on my life and the lives of two other Pākehā families who share similar educational backgrounds. This foundational study utilises qualitative narrative inquiry methods as a means of understanding and analysing the implications for Pākehā as a result of their participation in kaupapa Māori and bilingual education. An inter...</code>	<code>title: What is creative to whom and why? Perceptions in advertising agencies<br><br>authors: Koslow, Scott<br><br>abstract: The authors apply recent advances in creativity theory to discover perceptual differences in the factors of strategy, originality, and artistry among creatives and noncreatives. It was found that current advertising position influences subjective perceptions of what constitutes creative advertising. Creatives tend to perceive advertisements as more appropriate if they are artistic, but account executives tend to perceive advertisements as more appropriate if they are strategic. The study also indicates that creatives have a distinctive preference for a strong originality component to strategy. To be original within the confines of a tight strategy is perceived as the most creative by advertising creatives. Account executives are so focused on strategy, they will often accept artistic advertisements as a substitute for truly original work. The authors consider future research ...</code>
  <code>maori_origin</code>	<code>title: Te whakahuatanga i te reo Māori: Kua ahatia e tātou i roto i ngā tau 100 kua hipa nei? (The pronunciation of Māori: What have we done to it in the last 100 Years?)<br><br>authors: Harlow, Ray<br><br>subjects: Te reo Māori<br><br>abstract: In the words of the proverb: 'The land remains, but humankind vanishes.' One should perhaps extend this to read: 'The land remains, humankind vanishes, and language changes.' For languages are old, handed down from one generation to another, but no matter what, they change. Present-day English is not Shakespeare's or Chaucer's language. The pronunciation has changed, new words have entered the language, some words have been lost, the grammar is now different. Māori is like that as well. Over the last two centuries, many aspects of the language have become different. Many words have been borrowed from English, and become part of the language. Many new words have been created in recent years to accommodate the new topics now being spoken about in Māori. Many words...</code>	<code>title: A contrast-sensitive, redundancy reduction mechanism acting on MT neurons can explain global motion direction biases without the need for Bayesian priors<br><br>authors: Perrone, John A.<br><br>subjects: Bayesian observer model<br><br>abstract: Introduction: The perceived global direction of moving objects can be influenced by the contrast of the object (Weiss, Simoncelli & Adelson, Nature Neuroscience, 2002). There is currently no detailed, neural-based explanation for how this could occur. Weiss et al., proposed an ideal Bayesian observer model that included a physiologically unspecified ‘low speed’ prior. We have recently developed a new velocity code for extracting image velocity from small groups of MT neurons (Perrone & Krauzlis, VSS 2011). The code includes a stage of local spatial inhibition between MT neurons designed to reduce the amount of redundant signals passed onto the global motion integration stage (MST). The inhibition is made dependent upon the contrast of the stimulus by explo...</code>

• Loss: <code>TripletLoss</code> with these parameters:

  {
      "distance_metric": "TripletDistanceMetric.COSINE",
      "triplet_margin": 0.3
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 1
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• load_best_model_at_end: True
• push_to_hub: True
• hub_model_id: dinushiTJ/nz-research-commons-embedding-gemma
• hub_strategy: checkpoint
• hub_private_repo: False
• eval_on_start: True
• prompts: task: classification | query:

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 1
• per_device_eval_batch_size: 8
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch_fused
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: True
• resume_from_checkpoint: None
• hub_model_id: dinushiTJ/nz-research-commons-embedding-gemma
• hub_strategy: checkpoint
• hub_private_repo: False
• hub_always_push: False
• hub_revision: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: True
• use_liger_kernel: False
• liger_kernel_config: None
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: task: classification | query:
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Training Time

• Training: 18.1 minutes

Framework Versions

• Python: 3.12.13
• Sentence Transformers: 5.5.1
• Transformers: 4.57.0.dev0
• PyTorch: 2.11.0+cu128
• Accelerate: 1.13.0
• Datasets: 4.0.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

TripletLoss

@misc{hermans2017defense,
    title={In Defense of the Triplet Loss for Person Re-Identification},
    author={Alexander Hermans and Lucas Beyer and Bastian Leibe},
    year={2017},
    eprint={1703.07737},
    archivePrefix={arXiv},
    primaryClass={cs.CV}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->