GISTsmall

SentenceTransformer

This is a sentence-transformers model trained. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer

<!-- - **Base model:** [Unknown](https://huggingface.co/unknown) -->

• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 dimensions
• Similarity Function: Cosine Similarity

<!-- - **Training Dataset:** Unknown --> <!-- - **Language:** Unknown --> <!-- - **License:** Unknown -->

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'Soga no Tojiko sits with a playful smile, looking directly at the viewer. Her short green hair falls over her eyes, highlighting her blush and expressive gaze. She wears a green dress with long sleeves and a high collar, along with a distinctive black hat featuring a red accent. Notably, she has a ghost tail instead of legs, showcasing her black panties and large breasts. Japanese text appears on the right side of the image.',
    "Tojiko's playful smile captivates you, her green eyes sparkling with mischief. Her unique green dress, a perfect match for her hair, hugs her curves, emphasizing her ample chest. But it's her ghostly tail that draws your attention, a mysterious feature that adds to her allure. She sits there, legs crossed, her black panties barely visible, leaving much to the imagination. The Japanese text beside her seems to whisper secrets, adding to the enigmatic charm of this playful spirit.",
    "A young woman with a similar smile sits cross-legged on a cozy couch, her long, dark hair cascading over her shoulders. She's dressed in a casual, loose-fitting shirt and jeans, a comfortable outfit for a relaxed evening. Her legs are very much human, and she wears cute, colorful socks. The atmosphere is lighthearted, and she seems ready for a fun conversation, her expression inviting and warm.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

<!-- ### Direct Usage (Transformers) <details><summary>Click to see the direct usage in Transformers</summary> </details> --> <!-- ### Downstream Usage (Sentence Transformers) You can finetune this model on your own dataset. <details><summary>Click to expand</summary> </details> --> <!-- ### Out-of-Scope Use *List how the model may foreseeably be misused and address what users ought not to do with the model.* --> <!-- ## Bias, Risks and Limitations *What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.* --> <!-- ### Recommendations *What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.* -->

Training Details

Training Dataset

Unnamed Dataset

• Size: 1,244 training samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 39 tokens</li><li>mean: 113.16 tokens</li><li>max: 205 tokens</li></ul>	<ul><li>min: 59 tokens</li><li>mean: 104.96 tokens</li><li>max: 167 tokens</li></ul>	<ul><li>min: 39 tokens</li><li>mean: 86.33 tokens</li><li>max: 151 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>A Jiangshi girl with short blue hair and bright blue eyes stands in a graveyard. She wears a red traditional Chinese dress with a black skirt trimmed with white and patterned borders, a white petticoat, and a hat adorned with a star. An ofuda talisman is stuck to her forehead. Her arms are outstretched in a classic zombie pose, she has bare legs, and her tongue sticks out slightly as she walks towards the viewer with an eerie expression. Cherry blossoms surround her, contrasting with the tombstones and full moon illuminating the scene.</code>	<code>The moonlight bathes the graveyard in an ethereal glow, and you, the Jiangshi girl, feel a surge of power. You stride forward, your bare feet brushing against the cool grass, your tongue lolling out in a playful gesture. The cherry blossoms sway in the breeze, their fragrance filling the air. As you approach, you see your target, a handsome man, his eyes wide with a mixture of fear and desire. You reach out, your fingers icy cold, and he shivers, his breath quickening. You whisper ancient incantations, your voice seductive, and he's drawn into your embrace, captivated by your otherworldly allure.</code>	<code>You find yourself in a bustling city square, surrounded by vibrant street life. The air is filled with the aroma of street food and the lively chatter of vendors. As you wander, you spot a fortune teller's booth, adorned with colorful scarves and crystals. Intrigued, you step inside, where an elderly woman with kind eyes greets you. She offers you a cup of tea, its steam carrying the scent of jasmine, and begins to read your fortune, her words offering guidance and insight.</code>
  <code>Gunnthra is lying on her back, her pink hair adorned with a flower crown, while her torn bikini barely covers her, revealing her to be covered in cum. Tears stream down her face as she's surrounded by nude men engaging in various sexual acts. Laegjarn, with her dark skin and green-orange hair, wears a torn dark bikini top, showing her breasts also covered in cum. Her expression is a mix of anger and forced enjoyment as she receives oral sex. The scene takes place on a worn stone floor, emphasizing chaos and violation.</code>	<code>The men surround you, their hands exploring your body, their mouths leaving trails of wetness. You feel exposed, vulnerable, but a strange excitement courses through you. Gunnthra's sobs fill your ears, her delicate beauty contrasting with the roughness of the scene. Laegjarn's eyes meet yours, a silent plea in her gaze, and you realize the power you hold in this moment. The stone floor is cold beneath you, but your body burns with desire.</code>	<code>You stand in a lush garden, the sun warming your skin. Lucina, with her blue hair and determined eyes, leans in for a kiss, her touch gentle and full of unspoken promises. The sound of birds and the scent of flowers create a peaceful ambiance, and you feel a deep connection as you surrender to the moment.</code>
  <code>Sunday and Robin are engaged in sexual intercourse. Sunday, a male with long grey hair and yellow eyes, wears a white shirt and gloves. Robin, a female with long grey hair and blue eyes, wears a turquoise skirt with her top removed, exposing her breasts. Firefly is sketched in the corner, looking displeased. The scene is sketchily drawn, with a minimal white background, creating an intimate and raw atmosphere.</code>	<code>Sunday's hands glide over your body, his touch sending shivers down your spine. You run your fingers through his soft grey hair as Robin's kisses trail down your neck, her breasts brushing against your skin. The sketchy lines of the drawing seem to pulsate with the rhythm of your passion, Firefly's disapproving gaze only adding to the thrill.</code>	<code>In a serene garden, you lie with Robin, her grey hair cascading over her shoulders. Sunday, however, is nowhere to be seen. Robin's touch is gentle, her eyes filled with love, but you can't shake the feeling that something is amiss. The absence of Sunday's energetic presence leaves a void, and you find yourself longing for the complete trio.</code>

• Loss: <code>GISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.1}
  

Evaluation Dataset

Unnamed Dataset

• Size: 139 evaluation samples
• Columns: <code>anchor</code>, <code>positive</code>, and <code>negative</code>
• Approximate statistics based on the first 139 samples:

  	anchor	positive	negative
  type	string	string	string
  details	<ul><li>min: 45 tokens</li><li>mean: 111.65 tokens</li><li>max: 182 tokens</li></ul>	<ul><li>min: 64 tokens</li><li>mean: 106.32 tokens</li><li>max: 170 tokens</li></ul>	<ul><li>min: 50 tokens</li><li>mean: 88.61 tokens</li><li>max: 144 tokens</li></ul>

• Samples:

  anchor	positive	negative
  <code>Saren from "Princess Connect!" is on a bed, topless with her blonde hair styled in double buns secured by a black bow. She's wearing a maid outfit, visibly nervous and sweating; her large breasts are exposed as she kneels with her hands raised. Her blue and white skirt is being lifted by Suzume, who stands behind her with a smug look, showcasing her brown hair in double buns and a pink bow. A dark-skinned man is partially visible, suggestively touching Saren, conveying a scene of sexual tension. The setting is an upscale bedroom with detailed decor, including a broken plate on the floor. The image captures intense emotions and dynamic movements highlighted by Japanese text visuals.</code>	<code>Saren's heart pounds as she kneels on the luxurious bed, her maid outfit a stark contrast to the broken plate on the floor. You witness her vulnerability, her large breasts heaving with each rapid breath. Suzume, with a mischievous glint in her eye, lifts Saren's skirt, exposing her even further. The dark-skinned man's hand on Saren's thigh sends a jolt through you; you can almost feel the electricity of their touch. The tension is thick, and you're captivated by the raw emotions playing out before you.</code>	<code>In a serene garden, you witness a different scene. A red-headed elf, her hair adorned with white flowers, sits beside a fountain, her green dress flowing gracefully. She's not nervous like Saren; instead, she's calm, her eyes closed in peaceful meditation. A gentle breeze stirs, and you feel a sense of tranquility as she opens her eyes, her gaze meeting yours with a serene smile, inviting you to join her in this moment of quiet bliss.</code>
  <code>Kamisato Ayaka and Lumine from Genshin Impact are depicted in an explicit yuri scene. Ayaka, with her long silver hair tied in a ponytail, is nude and her skin looks wet. She's performing oral sex on Lumine, who has short blonde hair. Lumine's expression varies from pleasure to relaxation, and both characters are nude, emphasizing their wet bodies and saliva. The background is minimal, focusing on the detailed and passionate interaction between the two. There are no texts in the image.</code>	<code>You feel Ayaka's soft, wet hair brush against your skin as she moves her head in a steady rhythm. Her lips and tongue work in perfect harmony, sending waves of pleasure through your body. You run your fingers through her silver locks, guiding her movements, feeling her passion. The sensation is overwhelming, and you can't help but moan, your eyes fluttering shut as you surrender to the ecstasy.</code>	<code>Lumine's fingers gently caress your face as she leans in, her blonde hair framing her face. She whispers sweet nothings, her breath hot against your ear, sending shivers down your spine. You feel her lips brush against yours, a soft kiss that ignites a fire within. As the kiss deepens, you can almost taste the forbidden fruit, a sweet temptation you can't resist.</code>
  <code>Irisviel von Einzbern, a girl with long white hair and red eyes, is depicted in a state of arousal on all fours, with her buttocks raised and upper body leaning forward. She is completely nude, displaying her large breasts with visible nipples and a flushed complexion. Her expression shows pained pleasure, and moisture covers her skin, suggesting sweat or other bodily fluids. A partially visible male figure's hand rest on her buttocks in a suggestive manner. The background is a blurred reddish-brown tone that suggests an indistinct interior space. Motion lines and sweat droplets enhance the scene, characteristic of anime style. Portions of the image are obscured by mosaic censorship. The atmosphere is sexually explicit, emphasizing their intimate interaction, highlighted by Irisviel's expression, body language, and the warmer color palette.</code>	<code>In a dimly lit chamber, you find yourself on a soft fur rug, the air heavy with anticipation. Irisviel, her white hair cascading down her back, kneels before you, her eyes locked on yours. Her body glistens with desire, and as she arches her back, you see the full splendor of her naked form. The warmth of her skin radiates, and when she leans forward, her breath caresses your skin, sending shivers of pleasure through your body. The man's hand on her thigh adds to the intensity, and you're drawn into a world of pure, unadulterated passion.</code>	<code>A serene beach at sunset sets the stage for a romantic encounter. A woman with flowing blonde hair and deep blue eyes stands before you, her body glistening with seawater. She wears a sheer white cover-up, hinting at the curves beneath. As she turns towards you, the sun highlights her profile, and her smile is both inviting and mysterious. The sound of the waves and the warm breeze create an atmosphere of sensual tranquility.</code>

• Loss: <code>GISTEmbedLoss</code> with these parameters:

  {'guide': SentenceTransformer(
    (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
    (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
    (2): Normalize()
  ), 'temperature': 0.1}
  

Training Hyperparameters

Non-Default Hyperparameters

• eval_strategy: steps
• per_device_train_batch_size: 12
• per_device_eval_batch_size: 24
• learning_rate: 1e-05
• weight_decay: 0.01
• warmup_ratio: 0.1
• fp16: True
• gradient_checkpointing: True
• batch_sampler: no_duplicates

All Hyperparameters

<details><summary>Click to expand</summary>

• overwrite_output_dir: False
• do_predict: False
• eval_strategy: steps
• prediction_loss_only: True
• per_device_train_batch_size: 12
• per_device_eval_batch_size: 24
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• torch_empty_cache_steps: None
• learning_rate: 1e-05
• weight_decay: 0.01
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 3
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• restore_callback_states_from_checkpoint: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: False
• fp16: True
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: None
• hub_always_push: False
• gradient_checkpointing: True
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• include_for_metrics: []
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_eval_metrics: False
• eval_on_start: False
• use_liger_kernel: False
• eval_use_gather_object: False
• average_tokens_across_devices: False
• prompts: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

</details>

Training Logs

Framework Versions

• Python: 3.10.6
• Sentence Transformers: 3.3.1
• Transformers: 4.48.0
• PyTorch: 2.5.1+cu121
• Accelerate: 1.0.1
• Datasets: 3.1.0
• Tokenizers: 0.21.0

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

GISTEmbedLoss

@misc{solatorio2024gistembed,
    title={GISTEmbed: Guided In-sample Selection of Training Negatives for Text Embedding Fine-tuning},
    author={Aivin V. Solatorio},
    year={2024},
    eprint={2402.16829},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

<!-- ## Glossary *Clearly define terms in order to be accessible across audiences.* --> <!-- ## Model Card Authors *Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.* --> <!-- ## Model Card Contact *Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.* -->