T5-GenQ-TDC-v1

Model Card for T5-GenQ-TDC-v1

🤖 ✨ 🔍 Generate precise, realistic user-focused search queries from product text 🛒 🚀 📊

Model Description

• Model Name: Fine-Tuned Query-Generation Model
• Model type: Text-to-Text Transformer
• Finetuned from model: BeIR/query-gen-msmarco-t5-base-v1
• Dataset: smartcat/Amazon-2023-GenQ
• Primary Use Case: Generating accurate and relevant search queries from item descriptions
• Repository: smartcat-labs/product2query

Model variations

<table border="1" class="dataframe"> <tr style="text-align: center;"> <th>Model</th> <th>ROUGE-1</th> <th>ROUGE-2</th> <th>ROUGE-L</th> <th>ROUGE-Lsum</th> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-T-v1">T5-GenQ-T-v1</a></b></td> <td>75.2151</td> <td>54.8735</td> <td><b>74.5142</b></td> <td>74.5262</td> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-TD-v1">T5-GenQ-TD-v1</a></b></td> <td>78.2570</td> <td>58.9586</td> <td><b>77.5308</b></td> <td>77.5466</td> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-TDE-v1">T5-GenQ-TDE-v1</a></b></td> <td>76.9075</td> <td>57.0980</td> <td><b>76.1464</b></td> <td>76.1502</td> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-TDC-v1">T5-GenQ-TDC-v1</a> (best)</b></td> <td>80.0754</td> <td>61.5974</td> <td><b>79.3557</b></td> <td>79.3427</td> </tr> </table>

Uses

This model is designed to improve e-commerce search functionality by generating user-friendly search queries based on product descriptions. It is particularly suited for applications where product descriptions are the primary input, and the goal is to create concise, descriptive queries that align with user search intent.

Examples of Use:

<li>Generating search queries for product indexing.</li> <li>Enhancing product discoverability in e-commerce search engines.</li> <li>Automating query generation for catalog management.</li>

Comparison of ROUGE scores:

<table border="1"> <thead> <tr> <th>Model</th> <th>ROUGE-1</th> <th>ROUGE-2</th> <th>ROUGE-L</th> <th>ROUGE-Lsum</th> </tr> </thead> <tbody> <tr> <td>T5-GenQ-TDC-v1</td> <td>76.14</td> <td>56.11</td> <td>75.47</td> <td>75.47</td> </tr> <tr> <td>query-gen-msmarco-t5-base-v1</td> <td>34.94</td> <td>15.29</td> <td>34.18</td> <td>34.18</td> </tr> </tbody> </table>

Note: This evaluation is done after training, based on the test split of the smartcat/Amazon-2023-GenQ.

Examples

<details><summary>Expand to see the table with examples</summary> <table border="1" text-align: center> <thead> <tr> <th style="width: 25%;" >Input Text</th> <th style="width: 25%;">Target Query</th> <th>Before Fine-tuning</th> <th>After Fine-tuning</th> </tr> </thead> <tbody> <tr> <td><strong>AC/DC Men's Back in Black T-Shirt Black</strong>

This youth T-shirt is an officially licensed AC/DC product and features our cool Back In Black design printed on 100% cotton.</td> <td>AC/DC Back In Black T-Shirt</td> <td>what is ac dc back in black</td> <td>AC/DC Back in Black T-Shirt</td> </tr> <tr> <td><strong>AISYBB Women's Sweet Japanese Strawberry Embroidered Knit Cardigan Top Korean Retro Elegant Cute Kawaii Coat</strong>

BrandName: AISYBBSleeveStyle: RegularMaterialComposition: AcrylicPatternType: SolidSleeveLength(cm): FullStyle: SweetSeason: Spring/AutumnThickness: STANDARDDecoration: ButtonClothingLength: RegularMaterial: AcrylicClosureType: SingleBreastedGender: WOMENCollar: V-NeckPattern: Loose-fittingAge: Ages16-28YearsOldPercentageofMaterial: 31%-50%YarnThickness: RegularyarnColor: Red,PinkSize: M,L,XLFabricName: AcrylicPopularElements: Patchwork,ContrastColor,ButtonAge-appropriate: 18-29SweaterCraft: FlatKnittingFeature1: LeisureFeature2: ChicFeature3: ElegantFeature4: Fashion </td> <td>AISYBB Women's Strawberry Cardigan</td> <td>aisybb cardigans</td> <td>Japanese Strawberry Cardigan</td> </tr> <tr> <td><strong>Flying Fisherman Square Wrap Polarized Protection Oval Sand Bank Matte Black Amber, One Size</strong>

As stunning as the crystal blue water over a shallow, The Flying Fisherman sand bank polarized sunglasses will have you seeing clearer with increased visual perception. This light weight wrap style has a sturdy build ready for whatever comes your way. Medium fit features non-slip temples and built in Rubberized nose pads adding to solid performance and comfort for all day wear.   Polarized Triacetate lenses are impact and scratch resistant, lightweight and durable. Acutint lens coloring system adds color contrast without distorting natural colors, allowing you to see more clearly. 100% protection from harmful UVA and UVB rays.</td> <td>Polarized sunglasses for fishing</td> <td>what is polarized sand bank</td> <td>Flying Fisherman polarized sunglasses</td> </tr>

</tbody> </table> </details>

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("smartcat/T5-GenQ-TDC-v1")
tokenizer = AutoTokenizer.from_pretrained("smartcat/T5-GenQ-TDC-v1")

description = "Silver-colored cuff with embossed braid pattern. Made of brass, flexible to fit wrist."

inputs = tokenizer(description, return_tensors="pt", padding=True, truncation=True)
generated_ids = model.generate(inputs["input_ids"], max_length=30, num_beams=4, early_stopping=True)

generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True) 

Training Details

Training Data

The model was trained on the smartcat/Amazon-2023-GenQ dataset, which consists of user-like queries generated from product descriptions. The dataset was created using Claude Haiku 3, incorporating key product attributes such as the title, description, and images to ensure relevant and realistic queries. For more information, read the Dataset Card. 😊

Preprocessing

• Trained on titles + descriptions of the products and a subset of products with titles only that had a similarity score with short queries above 85%
• Tokenized using T5’s default tokenizer with truncation to handle long text.

Training Hyperparameters

<ul> <li><strong>max_input_length:</strong> 512</li> <li><strong>max_target_length:</strong> 30</li> <li><strong>batch_size:</strong> 48</li> <li><strong>num_train_epochs:</strong> 8</li> <li><strong>evaluation_strategy:</strong> epoch</li> <li><strong>save_strategy:</strong> epoch</li> <li><strong>learning_rate:</strong> 5.6e-05</li> <li><strong>weight_decay:</strong> 0.01 </li> <li><strong>predict_with_generate:</strong> true</li> <li><strong>load_best_model_at_end:</strong> true</li> <li><strong>metric_for_best_model:</strong> eval_rougeL</li> <li><strong>greater_is_better:</strong> true</li> <li><strong>logging_strategy:</strong> epoch</li> </ul>

Train time: 16.79 hrs

Hardware

A6000 GPU:

• Memory Size: 48 GB
• Memory Type: GDDR6
• CUDA: 8.6

Metrics

Metrics

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics used for evaluating automatic summarization and machine translation in NLP. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation. ROUGE metrics range between 0 and 1, with higher scores indicating higher similarity between the automatically produced summary and the reference.

In our evaluation, ROUGE scores are scaled to resemble percentages for better interpretability. The metric used in the training was ROUGE-L.

<table border=1> <thead> <tr> <th>Epoch</th> <th>Step</th> <th>Loss</th> <th>Grad Norm</th> <th>Learning Rate</th> <th>Eval Loss</th> <th>ROUGE-1</th> <th>ROUGE-2</th> <th>ROUGE-L</th> <th>ROUGE-Lsum</th> </tr> </thead> <tbody> <tr> <td>1</td> <td>5181</td> <td>0.6783</td> <td>1.4479</td> <td>0.000049</td> <td>0.5469</td> <td>78.3314</td> <td>59.1425</td> <td>77.7077</td> <td>77.6942</td> </tr> <tr> <td>2</td> <td>10362</td> <td>0.5621</td> <td>1.5229</td> <td>0.000042</td> <td>0.5191</td> <td>79.3411</td> <td>60.5070</td> <td>78.6538</td> <td>78.6498</td> </tr> <tr> <td>3</td> <td>15543</td> <td>0.5113</td> <td>1.7697</td> <td>0.000035</td> <td>0.5109</td> <td>79.4785</td> <td>60.7267</td> <td>78.8132</td> <td>78.8043</td> </tr> <tr> <td>4</td> <td>20724</td> <td>0.4758</td> <td>1.5957</td> <td>0.000028</td> <td>0.5113</td> <td>79.8056</td> <td>61.0473</td> <td>79.0640</td> <td>79.0545</td> </tr> <tr> <td>5</td> <td>25905</td> <td>0.4480</td> <td>1.5304</td> <td>0.000021</td> <td>0.5102</td> <td>79.9302</td> <td>61.4653</td> <td>79.2436</td> <td>79.2355</td> </tr> <tr> <td>6</td> <td>31086</td> <td>0.4277</td> <td>1.7578</td> <td>0.000014</td> <td>0.5130</td> <td>79.8888</td> <td>61.2644</td> <td>79.1514</td> <td>79.1433</td> </tr> <tr> <td>7</td> <td>36267</td> <td>0.4119</td> <td>1.8958</td> <td>0.000007</td> <td>0.5157</td> <td>80.0362</td> <td>61.5594</td> <td>79.3187</td> <td>79.3069</td> </tr> <tr> <td>8</td> <td>41448</td> <td>0.4018</td> <td>1.4958</td> <td>0</td> <td>0.5190</td> <td>80.0754</td> <td>61.5974</td> <td>79.3557</td> <td>79.3427</td> </tr> </tbody> </table> <style> .model-analysis table { width: 100%; border-collapse: collapse; } .model-analysis td { padding: 10px; vertical-align: middle; } .model-analysis img { width: auto; /* Maintain aspect ratio */ display: block; margin: 0 auto; max-height: 750px; /* Default height for most images */ } </style> <div class="model-analysis">

Model Analysis

<details><summary>Average scores by model</summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/average_scores_by_model.png" alt="image"></td> <td>The checkpoint-41448 (T5-GenQ-TDC-v1) model outperforms query-gen-msmarco-t5-base-v1 across all ROUGE metrics.

The largest performance gap is in ROUGE2, where checkpoint-41448 achieves 56.11, whereas query-gen-msmarco-t5-base-v1 scores 15.29.

ROUGE1, ROUGEL, and ROUGELSUM scores are very similar in both trends, with checkpoint-41448 consistently scoring above 72, while query-gen-msmarco-t5-base-v1 stays below 41.</td></tr> </table>

</details> <details><summary>Density comparison </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/density_comparison.png" alt="image"></td> <td>

T5-GenQ-TDC-v1 - Higher concentration of high ROUGE scores, especially near 100%, indicating strong text overlap with references.

query-gen-msmarco-t5-base-v1 – more spread-out distribution, with multiple peaks at 10-40%, suggesting greater variability but lower precision.

ROUGE-1 & ROUGE-L: T5-GenQ-TDC-v1 peaks at 100%, while query-gen-msmarco-t5-base-v1 has lower, broader peaks.

ROUGE-2: query-gen-msmarco-t5-base-v1 has a high density at 0%, indicating many low-overlap outputs. </td></tr> </table>

</details> <details><summary>Histogram comparison </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/histogram_comparison.png" alt="image"></td> <td>

T5-GenQ-TDC-v1 – higher concentration of high ROUGE scores, especially near 100%, indicating strong text overlap with references.

query-gen-msmarco-t5-base-v1 – more spread-out distribution, with peaks in the 10-40% range, suggesting greater variability but lower precision.

ROUGE-1 & ROUGE-L: T5-GenQ-TDC-v1 shows a rising trend towards higher scores, while query-gen-msmarco-t5-base-v1 has multiple peaks at lower scores.

ROUGE-2: query-gen-msmarco-t5-base-v1 has a high concentration of low-score outputs, whereas T5-GenQ-TDC-v1 generates more accurate text.</td></tr> </table>

</details> <details><summary>Scores by generated query length </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/group_sizes.png" alt="image"></td> <td> This visualization analyzes average ROUGE scores and score differences across different word sizes.

Consistently High ROUGE Scores (Sizes 2-8): ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-LSUM scores remain stable and above 60% across most word sizes.

Significant Drop at Size 9: A sharp decline in scores at size 9, followed by a spike at size 10, indicates a possible edge case in the dataset.

Stable Score Differences (Sizes 3-9):

After the initial spike at size 2, score differences stay close to zero, indicating consistent performance across phrase lengths.</td></tr> </table>

</details> <details><summary>Semantic similarity distribution </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/semantic_similarity_distribution.png" alt="image"></td> <td> This histogram visualizes the distribution of cosine similarity scores, which measure the semantic similarity between paired texts.

The majority of similarity scores cluster near 1.0, indicating that most text pairs are highly similar.

A gradual increase in frequency is observed as similarity scores rise, with a sharp peak at 1.0.

Lower similarity scores (0.0–0.4) are rare, suggesting fewer instances of dissimilar text pairs. </td></tr> </table>

</details> <details><summary>Semantic similarity score against ROUGE scores </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/similarity_vs_rouge.png" alt="image"></td> <td> This scatter plot matrix compares semantic similarity (cosine similarity) with ROUGE scores, showing their correlation.

Higher similarity → Higher ROUGE scores, indicating strong n-gram overlap in semantically similar texts.

ROUGE-1 & ROUGE-L show the strongest correlation, while ROUGE-2 has more variability.

Some low-similarity outliers still achieve moderate ROUGE scores, suggesting surface-level overlap without deep semantic alignment.

These plots help assess how semantic similarity aligns with n-gram overlap metrics in text evaluation.

</td></tr> </table> </details> <details><summary><b>Performance on Another Dataset</b></summary> <p>To assess the performance of our fine-tuned query generation model, we conducted evaluations on a dataset containing real user queries, which was not part of the fine-tuning data. The goal was to verify the model’s generalizability and effectiveness in generating high-quality queries for e-commerce products.</p> <details><summary><b>Average scores by model</b></summary> <table style="width:100%"> <tr> <td style="width:65%"> <img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/average_scores_by_model_escii.png" alt="Average scores by model" class="model-analysis"> </td> <td> The figure presents a comparison of ROUGE scores between GenQ-TDC-v1 (checkpoint-41448) and the base model (query-gen-msmarco-t5-base-v1). ROUGE scores (ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-LSUM) measure the similarity between generated queries and ground-truth user queries. <br><br> The fine-tuned model outperforms the base model across all ROUGE metrics, with significant improvements in ROUGE-1, ROUGE-2, and ROUGE-LSUM. <br><br> These improvements indicate that the fine-tuned model generates queries that are more aligned with real user queries, making it a better fit for practical e-commerce applications. </td> </tr> </table> </details> <details><summary><b>Semantic similarity distribution</b></summary> <table style="width:100%"> <tr> <td style="width:65%"> <img src="https://huggingface.co/smartcat/T5-GenQ-TDC-v1/resolve/main/semantic_similarity_distribution_escii.png" alt="Semantic similarity distribution" class="model-analysis"> </td> <td> The figure shows the distribution of semantic similarity scores (cosine similarity between the generated queries and real user queries. <br><br> The distribution shows a peak around 0.1 - 0.2, with a secondary peak near 0.6 - 0.7. <br><br> The bimodal nature of the distribution suggests that while a portion of generated queries are highly aligned with real user queries, some still exhibit lower similarity, likely due to domain-specific variations. <br><br> A significant number of generated queries achieve a similarity score of 0.6 or higher, confirming that the fine-tuned model effectively captures user intent better than a generic, pre-trained model. </td> </tr> </table> </details> <p><b>Better Generalization:</b> Even though the fine-tuned model was not trained on this dataset, it generalizes better than the original model on real user queries.</p> <p><b>Improved Query Quality:</b> The fine-tuned model produces more relevant, structured, and user-aligned queries which is critical for enhancing search and recommendation performance in e-commerce.</p> <p><b>Robust Semantic Alignment:</b> Higher semantic similarity scores indicate that queries generated by the fine-tuned model better match user intent, leading to improved search and retrieval performance.</p> </details> </details> </div>

More Information

• Please visit the GitHub Repository

Authors

• Mentor: Milutin Studen
• Engineers: Petar Surla, Andjela Radojevic

Model Card Contact

For questions, please open an issue on the GitHub Repository