bge-base-space-mt-tsdae

SentenceTransformer based on BAAI/bge-base-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-base-en-v1.5 on the tsdae and sup datasets. It maps sentences & paragraphs to a 768-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-base-en-v1.5 <!-- at revision a5beb1e3e68b9ab74eb54cfd186867f64f240e1a -->
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 768 tokens
• Similarity Function: Cosine Similarity
• Training Datasets:
  • tsdae
  • sup

<!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 768, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("federicovolponi/BAAI-bge-base-en-v1.5-space-multitask-tsdae")
# Run inference
sentences = [
    'Table of',
    ' Table 4 offers a description of the selected FoM',
    '\n[29] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 768]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* -->

Evaluation

Metrics

Information Retrieval

• Dataset: dim_768
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: dim_512
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: dim_256
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: dim_768
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: dim_512
• Evaluated with <code>InformationRetrievalEvaluator</code>

Information Retrieval

• Dataset: dim_256
• Evaluated with <code>InformationRetrievalEvaluator</code>

<!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Datasets

tsdae

• Dataset: tsdae
• Size: 95,730 training samples
• Columns: <code>damaged_sentence</code> and <code>orginal_sentence</code>
• Approximate statistics based on the first 1000 samples:

  	damaged_sentence	orginal_sentence
  type	string	string
  details	<ul><li>min: 3 tokens</li><li>mean: 13.28 tokens</li><li>max: 174 tokens</li></ul>	<ul><li>min: 6 tokens</li><li>mean: 30.02 tokens</li><li>max: 374 tokens</li></ul>

• Samples:

  damaged_sentence	orginal_sentence
  <code>, the described above allows continue this</code>	<code> However, the modularization into functional units described<br>above allows to continue this idea and form a well-defined functional hierarchy</code>
  <code>Solar scientific military and the stage for Change mission technology improvements—continued advances in will mass/volume</code>	<code> Solar sails can perform unique scientific, commercial, and military missions, and the stage is set for near-term<br>UPGRADE/REPLACE PAYLOADS • Change of mission • Take advantage of technology improvements—continued advances in electronics will cause payload components to shrink in mass/volume, while capabilities increase</code>
  <code>4mm Hexcell 5052 aluminum honeycomb with 1</code>	<code>4mm thick Hexcell 5052 alloy hexagonal aluminum honeycomb with 1</code>

• Loss: <code>losses.WeightedDenoisingAutoEncoderLoss</code>

sup

• Dataset: sup
• Size: 7,232 training samples
• Columns: <code>positive</code> and <code>anchor</code>
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	<ul><li>min: 5 tokens</li><li>mean: 354.69 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>min: 9 tokens</li><li>mean: 19.21 tokens</li><li>max: 40 tokens</li></ul>

• Samples:

  positive	anchor
  <code>, using diverse software or hardware designs may double design and verification costs due to having to build two different components for the same functionality. Hence, although DCLS execution also halves performance efficiency (the corresponding functionality is executed twice), it allows reusing the same design (e.g. the same core design) for the primary and the redundant paths (e.g. with staggered execution), thus containing design and verification costs.<br>Redundancy can be applied at different granularities accord- ing to the sphere of replication (SoR). Choosing the right SoR depends on several tradeoffs like area overheads, re- design costs, fault detection time, and overall system costs. In the context of DCLS, the SoR is placed at the level of the CPU (core), as done for the AURIX processors. This requires including two replicas of the same core and compare their memory transactions, which requires roughly duplicating com- putational resources in the chip and being able to ensure that replicas can provide independent behavior. On the other hand, storage (memories, caches) and communication means (buses, crossbars) do not need to be fully replicated and can build upon Error Correction Codes (ECC) and Cyclic Redundancy Check (CRC) as a form of lightweight redundancy with diversity.<br>HPC ASIL-D capable platforms typically combine a low- performance microcontroller amenable for the automotive do- main (i.e. ASIL-D capable) and an HPC accelerator deliv- ering high computation throughput, but whose adherence to ISO26262 requirements is unknown, so its appropriate use for ASIL-C/D systems needs to be investigated. Without loss of generality, we consider an NVIDIA GPU accelerator, thus analogous to those in NVIDIA Drive and Xavier families for the automotive domain. However, the findings in this paper can easily be extrapolated to other products.<br>Software faults and some hardware faults are regarded as systematic, and it must be proven that their failure risk is residual. However, random hardware faults cannot be avoided, and means are required to prevent them from causing hazards. Those faults can be caused by, for example, voltage droops</code>	<code>What are the advantages of using the same design for the primary and redundant paths in DCLS execution?<br><br></code>
  <code>: First, the TT&C spectrum requirements of the new satellites shall be assessed. Second, the utilization of existing TT&C frequency allocations and their potential to incorporate the future number of satellites is studied. Only for the case that this study results in the need for new spectrum, the study groups were asked to investigate new potential TT&C frequency allocations in the frequency ranges 150.05-174 MHz and 400.15-420 MHz. The studies shall be completed for WRC-19.<br>This paper presents the intermediate results of the study groups. A study of the spectrum requirements of small satellites has been completed. The required spectrum for TT&C is expected to be less than 2.5 MHz for downlink and less than 1 MHz for uplink. Consequently, the study groups conducted sharing studies in various bands which will be summarized and evaluated from a satellite developer’s perspective.<br>After the Cubesat design standard was introduced in 1999 and first satellites of this new class have been launched in the subsequent years, small satellites have become increasingly popular in the past five years. Today not only universities use small satellite platforms for education and technology demonstration, but also commercial operators started to develop and deploy satellites with masses of typically less than 50 kg and reasonably short development times. Currently more than hundred new satellites are currently launched into space per year. The increase of launches was recognized by the International Telecommunication Union (ITU) which is responsible for the coordination of the shared use of frequencies. As the first Cubesats were mainly launched by new entrants into the space sector, mandatory regulatory procedures like frequency coordination were omitted or underestimated by the developers. Additionally, the new developers complaint that the existing regulatory procedures are too complicated and time-consuming for satellites with short development times. The ITU therefore decided at the WRC-12 to study the characteristics of picosatellites and nanosatellites and their current practice in filing satellites to the ITU. The studies were concluded in 2015 with two reports on the characteristics [1] and current filing practice [2]. In these reports it was identified that the characteristics that define small satellites (low mass, small dimensions, low power, …) are not relevant from a frequency coordination perspective and that the short development times are still long enough to properly file the systems to the ITU. As a result</code>	<code>What are the spectrum requirements for TT&C of small satellites?<br><br></code>
  <code>:287–299, Dec 2019.<br>[20] Tam´as Vink´o and Dario Izzo. Global optimi- sation heuristics and test problems for prelimi- nary spacecraft trajectory design. Technical re- port, 2008.<br>[21] Matej Petkovic, Luke Lucas, Dragi Kocev, Saˇso Dˇzeroski, Redouane Boumghar, and Nikola Simidjievski. Quantifying the effects of gyro- less flying of the mars express spacecraft with machine learning. In 2019 IEEE International<br>[22] Janhavi H. Borse, Dipti D. Patil, Vinod Kumar, and Sudhir Kumar. Soft landing parameter measurements for candidate navigation trajec- tories using deep learning and ai-enabled plan- etary descent. Mathematical Problems in Engi- neering, 2022</code>	<code>What are some of the research topics and methods explored in the provided references?<br><br></code>

• Loss: <code>losses.WeightedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20,
      "similarity_fct": "cos_sim"
  }
  

Evaluation Datasets

tsdae

• Dataset: tsdae
• Size: 10,637 evaluation samples
• Columns: <code>damaged_sentence</code> and <code>orginal_sentence</code>
• Approximate statistics based on the first 1000 samples:

  	damaged_sentence	orginal_sentence
  type	string	string
  details	<ul><li>min: 3 tokens</li><li>mean: 13.52 tokens</li><li>max: 182 tokens</li></ul>	<ul><li>min: 5 tokens</li><li>mean: 30.74 tokens</li><li>max: 452 tokens</li></ul>

• Samples:

  damaged_sentence	orginal_sentence
  <code>from providing student licenses the OirthirSAT team</code>	<code><br>The authors thank Michael Doherty from Ansys for providing student licenses for STK to the OirthirSAT team</code>
  <code>at 205</code>	<code> 4 as observed by TROPICS Pathfinder at 205 GHz</code>
  <code>this reason of chemistry needed to radiative heating</code>	<code> For this reason, careful reexaminations of the chemistry models are needed to reduce the uncertainties in the radiative heating</code>

• Loss: <code>losses.WeightedDenoisingAutoEncoderLoss</code>

sup

• Dataset: sup
• Size: 804 evaluation samples
• Columns: <code>positive</code> and <code>anchor</code>
• Approximate statistics based on the first 1000 samples:

  	positive	anchor
  type	string	string
  details	<ul><li>min: 4 tokens</li><li>mean: 351.15 tokens</li><li>max: 512 tokens</li></ul>	<ul><li>min: 8 tokens</li><li>mean: 19.36 tokens</li><li>max: 45 tokens</li></ul>

• Samples:

  positive	anchor
  <code>, the total number of test thermocouples has been rationalized taking into account redundancy needs, accommodation constraints and hardware passivation needs for flight. The test is subdivided into 19 phases (see Figure 12) with two phases before and after the test for the health check functional tests under room conditions. Functional tests demonstrate anomalies such as the PCDU Reset and operational malfunctions of the RAX instrument at its high temperatures. The PCDU Reset anomaly was solved during the test by a software patch and validated during the final hot and cold plateaus. To address the RAX anomaly at hot, various test configurations were simulated using the thermal numerical model during the test to actually perform RAX functional test at an intermediate plateau facilitating mission operational constraints for flight. Data collected from hot and cold thermal balance test phases, as well as the rover OFF transition from hot to cold, are the inputs for correlation activities conducted post-TV/TB test. The thermal numerical model updates mainly focus on conductive couplings</code>	<code>What was the solution to the PCDU Reset anomaly during the test?<br><br></code>
  <code>, where +Z axis orients to the earth, and sun pointing attitude mode during day time<br>orienting -Z axis to the sun. Therefore, attitude control subsystem is required to maneuver the satellite attitude twice per revolution around its pitch axis. Figure 6 shows concept of the attitude maneuverer. Another attitude maneuverer is necessary to perform SAR observation and SAR data download to a to ground station, because X-band transmit antenna is oriented to +Z, so the satellite has to offset its attitude to orient the X-band transmit antenna toward the ground station.<br>3.4 High pointing accuracy<br>Disturbance torque and system momentum profiles during few revolutions were estimated as shown in Figure 7 and 8. Four micro reaction wheels, which can respond to these profiles were selected which enable attitude maneuvers within a short period of time. In order to perform a pitch attitude maneuver quickly, two wheels are located on pitch axis while one wheel was located on each of the remaining roll and yaw axes. Figure 9 shows the satellite attitudes during SAR observation. There are three kinds of attitude, strip map mode, sliding spot light mode, and spotlight mode. Large change of momentum is required for pitch axis when the satellite is in spotlight mode. However, two pitch reaction wheels do not generate enough momentum to execute spotlight mode. So, sliding spotlight mode was selected for high resolution SAR observation mode instead of spotlight mode, in order to relax the torque and momentum requirements to the pitch wheels. In addition, two pitch<br>Figure 7. Disturbance torque profile Figure 8. System momentum profile<br>reaction wheels are accelerated to plus direction or minus direction by using magnet torque before observation. In order to obtain a high resolution SAR data, high attitude control accuracy is required for spotlight mode observation. To achieve high pointing accuracy against a defined ground target point, the attitude control loop applied feed forward compensation with estimated attitude angle and rate. Figure 10 shows an example of dynamic error during a spotlight mode observation maneuver.[4]<br>Equipment for SAR mission consumes total large power more than 1300W, therefore PCDU has a risk of causing electrical and RF influence to the bus power and signal line. In order to research the system, electrical interface check was performed using bread board model of PCDU, battery</code>	<code>What is the reason for selecting sliding spotlight mode instead of spotlight mode for high resolution SAR observation?<br><br></code>
  <code>, body shape and motion assumptions. Then, ORSAT uses DCA to determine the reentry risk posed to the Earth’s<br>population based on the year of reentry and orbit inclination. It also predicts impact kinetic energy (impact velocity and impact mass) of objects that survive reentry[18]. ORSAT has been in use for the last decade and currently in its 6.0 version. However, unlike DAS, OR-<br>SAT is not readily available. Only personnel at the Johnson Space Center, Orbital Debris Program Office run ORSAT. ORSAT is limited to ballistic reentry, only tumbling motions or<br>stable orientations of objects are allowed which produce no lift. Partial melting of objects is considered by a demise factor and almost all materials in the database are temperature de- pendent. Heating by oxidation is also considered [20]. Therefore, ORSAT determines when<br>and if a reentry object demises by using integrated trajectory, atmospheric, aerodynamic, aero-thermodynamic, and thermal models as outlined in section 3.1 [17, 18, 20].<br>Reentry demisability analysis using DAS requires the spacecraft to be defined to the level of each individual hardware part constituting the spacecraft. This step facilitates population<br>of the DAS Spacecraft Definition Module . Section 3.2.1 illustrates a generic spacecraft subdivision approach that can be followed to itemize the individual parts spacecraft parts.<br>Subsequently, non-demisable parts are identified before or by the actual reentry analysis as explained in section 3.2.2.<br>Itemization of the demisable spacecraft basic parts can be best approached by decompos- ing the spacecraft according to the Hierarchical System Terminology defined in the NASA Systems Engineering Handbook [14]. Tables 3.2, 3.3 and 3.4 illustrate a generic approach<br>to decompose a spacecraft into basic parts [29, 30, 9] excluding the payload. Description of the specific product for the basic part identified completes the process. Though slight vari-<br>ations are likely to occur in the decomposition of different missions, the Generic Spacecraft Subsystems Hierarchical Subdivision approach is robust, hence</code>	<code>What is the limitation of ORSAT in terms of object motion?<br><br></code>

• Loss: <code>losses.WeightedMultipleNegativesRankingLoss</code> with these parameters:

  {
      "scale": 20,
      "similarity_fct": "cos_sim"
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• learning_rate: 3e-06
• weight_decay: 0.001
• num_train_epochs: 6
• bf16: True
• tf32: False
• load_best_model_at_end: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 32
• per_device_eval_batch_size: 32
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 3e-06
• weight_decay: 0.001
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 6
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.0
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: False
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: True
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

Framework Versions

• Python: 3.12.0
• Sentence Transformers: 3.0.1
• Transformers: 4.41.2
• PyTorch: 2.3.1+cu118
• Accelerate: 0.31.0
• Datasets: 2.20.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

WeightedDenoisingAutoEncoderLoss

@inproceedings{wang-2021-TSDAE,
    title = "TSDAE: Using Transformer-based Sequential Denoising Auto-Encoderfor Unsupervised Sentence Embedding Learning",
    author = "Wang, Kexin and Reimers, Nils and Gurevych, Iryna", 
    booktitle = "Findings of the Association for Computational Linguistics: EMNLP 2021",
    month = nov,
    year = "2021",
    address = "Punta Cana, Dominican Republic",
    publisher = "Association for Computational Linguistics",
    pages = "671--688",
    url = "https://arxiv.org/abs/2104.06979",
}

WeightedMultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->