career-matching-minilm

SentenceTransformer based on sentence-transformers/all-MiniLM-L6-v2

This is a sentence-transformers model finetuned from sentence-transformers/all-MiniLM-L6-v2 on a pre-processed version of the ESCO dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more. It was fine-tuned for a career-matching service. The relevant service can be found here.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: sentence-transformers/all-MiniLM-L6-v2 <!-- at revision c9745ed1d9f207416be6d2e6f8de32d1f16199bf -->
• Maximum Sequence Length: 256 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity
• Training Dataset:
  • csv

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'BertModel'})
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("malenaduroux/career-matching-minilm")
# Run inference
sentences = [
    'Occupation: dairy processing technician. Description: Dairy processing technicians supervise and coordinate production processes, operations, and maintenance workers in milk, cheese, ice cream and/or other dairy production plants. They assist food technologists in improving processes, developing new food products and establishing procedures and standards for production and packaging.',
    'check quality of products on the production line',
    'coordinate events',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.5273, 0.2572],
#         [0.5273, 1.0000, 0.1089],
#         [0.2572, 0.1089, 1.0000]])

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

ESCO data processed into csv

• See the relevant project repo for the preprocessing of the ESCO dataset.

• Dataset: csv

• Size: 102,100 training samples

• Columns: <code>occupation_text</code> and <code>paired_text</code>

• Approximate statistics based on the first 1000 samples:

  	occupation_text	paired_text
  type	string	string
  details	<ul><li>min: 17 tokens</li><li>mean: 65.16 tokens</li><li>max: 157 tokens</li></ul>	<ul><li>min: 3 tokens</li><li>mean: 5.93 tokens</li><li>max: 15 tokens</li></ul>

• Samples:

  occupation_text	paired_text
  <code>Occupation: cyber incident responder. Description: Cyber incident responders monitor and assess cybersecurity state systems, analysing, evaluating, and mitigating the impact of cybersecurity incidents. Moreover, they identify malicious actors and cyber incidents root causes. According to the organisation’s Incident Response Plan, they restore systems and process functionalities to an operational state, collecting evidence and documenting actions taken.</code>	<code>ensure proper document management</code>
  <code>Occupation: cider master. Description: Cider masters envision the manufacturing process of cider. They ensure brewing quality and follow one of several brewing processes. They modify existing brewing formulas and processing techniques in order to develop new cider products and cider-based beverages.</code>	<code>mashing process</code>
  <code>Occupation: microelectronics designer. Description: Microelectronics designers focus on developing and designing microelectronic systems, from the top packaging level down to the integrated circuit level. Their knowledge incorporates system-level understanding with analogue and digital circuit knowledge, with integrating the technology processes and an overall outlook in microelectronic sensor basics. They work with other engineers, material science specialists and researchers, to enable innovations and continuous development of already existing devices.</code>	<code>micromechanics</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Evaluation Dataset

csv

• Dataset: csv
• Size: 11,345 evaluation samples
• Columns: <code>occupation_text</code> and <code>paired_text</code>
• Approximate statistics based on the first 1000 samples:

  	occupation_text	paired_text
  type	string	string
  details	<ul><li>min: 17 tokens</li><li>mean: 64.79 tokens</li><li>max: 146 tokens</li></ul>	<ul><li>min: 3 tokens</li><li>mean: 5.88 tokens</li><li>max: 14 tokens</li></ul>

• Samples:

  occupation_text	paired_text
  <code>Occupation: environmental technician. Description: Environmental technicians investigate sources of pollution and aid in the development of pollution prevention and environment protection plans. They take samples of soil, water or other materials and perform tests to analyse the pollution level and identify its source.</code>	<code>pollution prevention</code>
  <code>Occupation: head of workshop. Description: Heads of workshop coordinate specialised workshops that construct, build, prepare, adapt and maintain elements used on stage. Their work is based on artistic vision, schedules and overall production documentation. They liaise with the designers involved in the production, the production team and other services of the organisation.</code>	<code>lead a team</code>
  <code>Occupation: embedded systems security engineer. Description: Embedded systems security engineers advise and implement solutions to control access to data and programs in embedded and connected systems. They help ensuring the safe operation of products with embedded systems and connected devices by being responsible for the protection and security of the related systems and design, plan and execute security measures accordingly. Embedded systems security engineers help to keep attackers at bay by implementing safeguards that prevent intrusions and breaches.</code>	<code>keep up with the latest information systems solutions</code>

• Loss: <code>MultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20.0,
      "similarity_fct": "cos_sim",
      "gather_across_devices": false
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 128
• num_train_epochs: 4
• warmup_steps: 0.1
• eval_strategy: epoch
• per_device_eval_batch_size: 128
• warmup_ratio: 0.1

All Hyperparameters

<details><summary>Click to expand</summary>

• per_device_train_batch_size: 128
• num_train_epochs: 4
• max_steps: -1
• learning_rate: 5e-05
• lr_scheduler_type: linear
• lr_scheduler_kwargs: None
• warmup_steps: 0.1
• optim: adamw_torch_fused
• optim_args: None
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• optim_target_modules: None
• gradient_accumulation_steps: 1
• average_tokens_across_devices: True
• max_grad_norm: 1.0
• label_smoothing_factor: 0.0
• bf16: False
• fp16: False
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• use_liger_kernel: False
• liger_kernel_config: None
• use_cache: False
• neftune_noise_alpha: None
• torch_empty_cache_steps: None
• auto_find_batch_size: False
• log_on_each_node: True
• logging_nan_inf_filter: True
• include_num_input_tokens_seen: no
• log_level: passive
• log_level_replica: warning
• disable_tqdm: False
• project: huggingface
• trackio_space_id: trackio
• eval_strategy: epoch
• per_device_eval_batch_size: 128
• prediction_loss_only: True
• eval_on_start: False
• eval_do_concat_batches: True
• eval_use_gather_object: False
• eval_accumulation_steps: None
• include_for_metrics: []
• batch_eval_metrics: False
• save_only_model: False
• save_on_each_node: False
• enable_jit_checkpoint: False
• push_to_hub: False
• hub_private_repo: None
• hub_model_id: None
• hub_strategy: every_save
• hub_always_push: False
• hub_revision: None
• load_best_model_at_end: False
• ignore_data_skip: False
• restore_callback_states_from_checkpoint: False
• full_determinism: False
• seed: 42
• data_seed: None
• use_cpu: False
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• parallelism_config: None
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• dataloader_prefetch_factor: None
• remove_unused_columns: True
• label_names: None
• train_sampling_strategy: random
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• ddp_backend: None
• ddp_timeout: 1800
• fsdp: []
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• deepspeed: None
• debug: []
• skip_memory_metrics: True
• do_predict: False
• resume_from_checkpoint: None
• warmup_ratio: 0.1
• local_rank: -1
• prompts: None
• batch_sampler: batch_sampler
• multi_dataset_batch_sampler: proportional
• router_mapping: {}
• learning_rate_mapping: {}

</details>

Training Logs

Framework Versions

• Python: 3.12.11
• Sentence Transformers: 5.2.3
• Transformers: 5.2.0
• PyTorch: 2.8.0+cu128
• Accelerate: 1.12.0
• Datasets: 4.6.0
• Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply},
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->