T5-GenQ-T-v1

Model Card for T5-GenQ-T-v1

🤖 ✨ 🔍 Generate precise, realistic user-focused search queries from product text 🛒 🚀 📊

Model Description

• Model Name: Fine-Tuned Query-Generation Model
• Model type: Text-to-Text Transformer
• Finetuned from model: BeIR/query-gen-msmarco-t5-base-v1
• Dataset: smartcat/Amazon-2023-GenQ
• Primary Use Case: Generating accurate and relevant search queries from item descriptions
• Repository: smartcat-labs/product2query

Model variations

<table border="1" class="dataframe"> <tr style="text-align: center;"> <th>Model</th> <th>ROUGE-1</th> <th>ROUGE-2</th> <th>ROUGE-L</th> <th>ROUGE-Lsum</th> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-T-v1">T5-GenQ-T-v1</a></b></td> <td>75.2151</td> <td>54.8735</td> <td><b>74.5142</b></td> <td>74.5262</td> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-TD-v1">T5-GenQ-TD-v1</a></b></td> <td>78.2570</td> <td>58.9586</td> <td><b>77.5308</b></td> <td>77.5466</td> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-TDE-v1">T5-GenQ-TDE-v1</a></b></td> <td>76.9075</td> <td>57.0980</td> <td><b>76.1464</b></td> <td>76.1502</td> </tr> <tr> <td><b><a href="https://huggingface.co/smartcat/T5-GenQ-TDC-v1">T5-GenQ-TDC-v1</a> (best)</b></td> <td>80.0754</td> <td>61.5974</td> <td><b>79.3557</b></td> <td>79.3427</td> </tr> </table>

Uses

This model is designed to improve e-commerce search functionality by generating user-friendly search queries based on product descriptions. It is particularly suited for applications where product descriptions are the primary input, and the goal is to create concise, descriptive queries that align with user search intent.

Examples of Use:

<li>Generating search queries for product indexing.</li> <li>Enhancing product discoverability in e-commerce search engines.</li> <li>Automating query generation for catalog management.</li>

Comparison of ROUGE scores:

<table border="1"> <thead> <tr> <th>Model</th> <th>ROUGE-1</th> <th>ROUGE-2</th> <th>ROUGE-L</th> <th>ROUGE-Lsum</th> </tr> </thead> <tbody> <tr> <td>T5-GenQ-T-v1</td> <td>73.11</td> <td>52.27</td> <td>72.51</td> <td>72.51</td> </tr> <tr> <td>query-gen-msmarco-t5-base-v1</td> <td>40.34</td> <td>19.52</td> <td>39.21</td> <td>39.21</td> </tr> </tbody> </table>

Note: This evaluation is done after training, based on the test split of the smartcat/Amazon-2023-GenQ dataset.

Examples

<details><summary>Expand to see table with examples</summary> <table border="1" text-align: center> <thead> <tr> <th style="width: 25%;" >Input Text</th> <th style="width: 25%;">Target Query</th> <th>Before Fine-tuning</th> <th>After Fine-tuning</th> </tr> </thead> <tbody> <tr> <td>PANDORA Jewelry Crossover Pave Triple Band Ring for Women - Sterling Silver with Cubic Zirconia</td> <td>PANDORA Crossover Triple Band Ring</td> <td>what is pandora jewelry</td> <td>Pandora crossover ring</td> </tr> <tr> <td>SAYOYO Baby Sneakers Leather Baby Shoes Crib Shoes Toddler Soft Sole Sneakers</td> <td>SAYOYO Baby Sneakers</td> <td>what kind of shoes are baby sneakers</td> <td>baby leather sneakers</td> </tr> <tr> <td>5 PCS Strap Replacement Compatible with Xiaomi Mi Band 3/4, Bands Xiaomi Mi Band 4 Smart Watch Wristbands Replacement Accessories Strap Bracelets for Mi Fit 3 Straps</td> <td>Replacement Straps for Xiaomi Mi Band 3/4p</td> <td>what is the strap on a xiaomi smartwatch</td> <td>Xiaomi Mi Fit 3 replacement bands</td> </tr> <tr> <td>Backpacker Ladies' Solid Flannel Shirt</td> <td>ladies flannel shirt</td> <td>what kind of shirt is a backpacker</td> <td>women's flannel shirt</td> </tr> </tbody> </table> </details>

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForSeq2SeqLM, AutoTokenizer

model = AutoModelForSeq2SeqLM.from_pretrained("smartcat/T5-GenQ-T-v1")
tokenizer = AutoTokenizer.from_pretrained("smartcat/T5-GenQ-T-v1")

description = "Silver-colored cuff with embossed braid pattern. Made of brass, flexible to fit wrist."

inputs = tokenizer(description, return_tensors="pt", padding=True, truncation=True)
generated_ids = model.generate(inputs["input_ids"], max_length=30, num_beams=4, early_stopping=True)

generated_text = tokenizer.decode(generated_ids[0], skip_special_tokens=True) 

Training Details

Training Data

The model was trained on the smartcat/Amazon-2023-GenQ dataset, which consists of user-like queries generated from product descriptions. The dataset was created using Claude Haiku 3, incorporating key product attributes such as the title, description, and images to ensure relevant and realistic queries. For more information, read the Dataset Card. 😊

Preprocessing

• Trained on only the product titles
• Tokenized using T5’s default tokenizer with truncation to handle long text.

Training Hyperparameters

<ul> <li><strong>max_input_length:</strong> 512</li> <li><strong>max_target_length:</strong> 30</li> <li><strong>batch_size:</strong> 48</li> <li><strong>num_train_epochs:</strong> 8</li> <li><strong>evaluation_strategy:</strong> epoch</li> <li><strong>save_strategy:</strong> epoch</li> <li><strong>learning_rate:</strong> 5.6e-05</li> <li><strong>weight_decay:</strong> 0.01 </li> <li><strong>predict_with_generate:</strong> true</li> <li><strong>load_best_model_at_end:</strong> true</li> <li><strong>metric_for_best_model:</strong> eval_rougeL</li> <li><strong>greater_is_better:</strong> true</li> <li><strong>logging_strategy:</strong> epoch</li> </ul>

Train time: 2.43 hrs

Hardware

A6000 GPU:

• Memory Size: 48 GB
• Memory Type: GDDR6
• CUDA: 8.6

Metrics

ROUGE, or Recall-Oriented Understudy for Gisting Evaluation, is a set of metrics used for evaluating automatic summarization and machine translation in NLP. The metrics compare an automatically produced summary or translation against a reference or a set of references (human-produced) summary or translation. ROUGE metrics range between 0 and 1, with higher scores indicating higher similarity between the automatically produced summary and the reference.

In our evaluation, ROUGE scores are scaled to resemble percentages for better interpretability. The metric used in the training was ROUGE-L.

<table> <thead> <tr> <th>Epoch</th> <th>Step</th> <th>Loss</th> <th>Grad Norm</th> <th>Learning Rate</th> <th>Eval Loss</th> <th>ROUGE-1</th> <th>ROUGE-2</th> <th>ROUGE-L</th> <th>ROUGE-Lsum</th> </tr> </thead> <tbody> <tr><td>1.0</td><td>4285</td><td>0.9465</td><td>6.7834</td><td>4.9e-05</td><td>0.7644</td><td>73.1872</td><td>52.2019</td><td>72.5199</td><td>72.5183</td></tr> <tr><td>2.0</td><td>8570</td><td>0.8076</td><td>4.9071</td><td>4.2e-05</td><td>0.7268</td><td>73.9182</td><td>53.1365</td><td>73.2551</td><td>73.2570</td></tr> <tr><td>3.0</td><td>12855</td><td>0.7485</td><td>4.4814</td><td>3.5e-05</td><td>0.7160</td><td>74.4752</td><td>53.8076</td><td>73.7712</td><td>73.7792</td></tr> <tr><td>4.0</td><td>17140</td><td>0.7082</td><td>5.3145</td><td>2.8e-05</td><td>0.7023</td><td>74.7628</td><td>54.3316</td><td>74.0811</td><td>74.0790</td></tr> <tr><td>5.0</td><td>21425</td><td>0.6788</td><td>4.4266</td><td>2.1e-05</td><td>0.7013</td><td>74.9437</td><td>54.5630</td><td>74.2637</td><td>74.2668</td></tr> <tr><td>6.0</td><td>25710</td><td>0.6561</td><td>5.2897</td><td>1.4e-05</td><td>0.6998</td><td>75.0834</td><td>54.7163</td><td>74.3907</td><td>74.3977</td></tr> <tr><td>7.0</td><td>29995</td><td>0.6396</td><td>3.5197</td><td>7.0e-06</td><td>0.7005</td><td>75.2151</td><td>54.8735</td><td>74.5142</td><td>74.5262</td></tr> <tr><td>8.0</td><td>34280</td><td>0.6278</td><td>4.4625</td><td>0.0</td><td>0.7016</td><td>75.1899</td><td>54.8423</td><td>74.4695</td><td>74.4801</td></tr> </tbody> </table> <style> .model-analysis table { width: 100%; border-collapse: collapse; } .model-analysis td { padding: 10px; vertical-align: middle; } .model-analysis img { width: auto; /* Maintain aspect ratio */ display: block; margin: 0 auto; max-height: 750px; /* Default height for most images */ } </style> <div class="model-analysis">

Model Analysis

<details><summary>Average scores by model</summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-T-v1/resolve/main/average_scores_by_model.png" alt="image"></td> <td>The checkpoint-29995 (T5-GenQ-T-v1) model outperforms query-gen-msmarco-t5-base-v1 across all ROUGE metrics.

The largest performance gap is in ROUGE2, where checkpoint-29995 achieves 52.27, whereas query-gen-msmarco-t5-base-v1 scores 19.52.

ROUGE1, ROUGEL, and ROUGELSUM scores are very similar in both trends, with checkpoint-29995 consistently scoring above 72, while query-gen-msmarco-t5-base-v1 stays below 41.</td></tr> </table>

</details> <details><summary>Density comparison </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-T-v1/resolve/main/density_comparison.png" alt="image"></td> <td>

T5-GenQ-T-v1 - Higher concentration of high ROUGE scores, especially near 100%, indicating strong text overlap with references.

query-gen-msmarco-t5-base-v1 – more spread-out distribution, with multiple peaks at 10-40%, suggesting greater variability but lower precision.

ROUGE-1 & ROUGE-L: T5-GenQ-T-v1 peaks at 100%, while query-gen-msmarco-t5-base-v1 has lower, broader peaks.

ROUGE-2: query-gen-msmarco-t5-base-v1 has a high density at 0%, indicating many low-overlap outputs.

</td></tr> </table> </details> <details><summary>Histogram comparison </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-T-v1/resolve/main/histogram_comparison.png" alt="image"></td> <td>

T5-GenQ-T-v1 – higher concentration of high ROUGE scores, especially near 100%, indicating strong text overlap with references.

query-gen-msmarco-t5-base-v1 – more spread-out distribution, with peaks in the 10-40% range, suggesting greater variability but lower precision.

ROUGE-1 & ROUGE-L: T5-GenQ-T-v1 shows a rising trend towards higher scores, while query-gen-msmarco-t5-base-v1 has multiple peaks at lower scores.

ROUGE-2: query-gen-msmarco-t5-base-v1 has a high concentration of low-score outputs, whereas T5-GenQ-T-v1 achieves more high-scoring outputs.</td></tr> </table>

</details> <details><summary>Scores by generated query length </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-T-v1/resolve/main/group_sizes.png" alt="image"></td> <td> This visualization analyzes average ROUGE scores and score differences across different query sizes.

High ROUGE Scores for Most Sizes (3-9 words).

ROUGE-1, ROUGE-2, ROUGE-L, and ROUGE-LSUM scores remain consistently high across most word sizes.

Sharp Spike at Size 2:

A large positive score difference at 2 words, suggesting strong alignment for very short phrases.

Stable Score Differences (Sizes 3-9):

After the initial spike at size 2, score differences stay close to zero, indicating consistent performance across phrase lengths.</td></tr> </table>

</details> <details><summary>Semantic similarity distribution </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-T-v1/resolve/main/semantic_similarity_distribution.png" alt="image"></td> <td> This histogram visualizes the distribution of cosine similarity scores, which measure the semantic similarity between paired texts.

The majority of similarity scores cluster near 1.0, indicating that most text pairs are highly similar.

A gradual increase in frequency is observed as similarity scores rise, with a sharp peak at 1.0.

Lower similarity scores (0.0–0.4) are rare, suggesting fewer instances of dissimilar text pairs. </td></tr> </table>

</details> <details><summary>Semantic similarity score against ROUGE scores </summary> <table style="width:100%"><tr> <td style="width:65%"><img src="https://huggingface.co/smartcat/T5-GenQ-T-v1/resolve/main/similarity_vs_rouge.png" alt="image"></td> <td> This scatter plot matrix compares semantic similarity (cosine similarity) with ROUGE scores, showing their correlation.

Higher similarity → Higher ROUGE scores, indicating strong n-gram overlap in semantically similar texts. ROUGE-1 & ROUGE-L show the strongest correlation, while ROUGE-2 has more variability. Low-similarity outliers exist, where texts share words but differ semantically. </td></tr> </table>

</details> </details> </div>

More Information

• Please visit the GitHub Repository

Authors

• Mentor: Milutin Studen
• Engineers: Petar Surla, Andjela Radojevic

Model Card Contact

For questions, please open an issue on the GitHub Repository