mt5-small-hashtag-generation

mt5-small fine-tuned for multi-hashtag generation

Sequence-to-sequence model that generates a set of hashtags for a social-media post. Fine-tuned from google/mt5-small on sm4rtdev/x_dataset_240.

The task is framed as multi-label semantic generation via seq2seq, not multi-label classification: the tag vocabulary is open, the number of tags per post is variable, and the decoder emits tags as text.

Task format

Input:  Generate relevant hashtags for the following social media post:

        <post text, with any existing hashtags removed>

Output: #hashtag1 #hashtag2 #hashtag3

Training data

Source dataset: sm4rtdev/x_dataset_240 (single train split, X/Twitter posts, predominantly crypto/finance topics, mostly English with a multilingual tail).

Preprocessing

In this dataset the target hashtags are literally present inside the post text for ~84% of rows. Training on the raw text would therefore reduce the task to substring copying, so hashtags are stripped from the model input. The remaining pipeline:

• Drop rows whose post has fewer than 4 words after stripping.
• Drop rows with 0 hashtags, or more than 20 (keyword-stuffing spam).
• Normalise hashtags (##tag, ＃tag, trailing punctuation) to canonical #tag.
• De-duplicate tags within an example, case-insensitively, preserving order and casing.
• De-duplicate (post, hashtag-set) pairs across the corpus.
• Split 80%/10%/10% with seed 42, grouped by post text so that no post appears in two splits.

URLs, mentions, emojis and punctuation are preserved.

Training configuration

fp16 is not supported. mT5 was pre-trained in bfloat16 and overflows to NaN under fp16 mixed precision. Use bf16 or fp32.

Best validation hashtag_f1: 0.4166

Evaluation

Hashtag prediction is a set task: #ai #python and #python #ai are the same answer. The primary metrics are therefore set-based, computed after parsing both prediction and reference into case-insensitive tag sets. ROUGE is reported as a secondary, order-sensitive signal.

Set-based metrics (primary)

Generation metrics (secondary)

Average hashtags generated: 3.46 (reference: 5.37).

Usage

import re
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

model_id = "thealper2/mt5-small-hashtag-generation"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)

PROMPT = "Generate relevant hashtags for the following social media post:\n\n{}"

def generate_hashtags(text: str, num_beams: int = 4) -> list[str]:
    # Match training: strip any hashtags already in the post.
    post = re.sub(r"#[\w\u0080-\uFFFF]+", " ", text)
    post = re.sub(r"\s+", " ", post).strip()

    inputs = tokenizer(
        PROMPT.format(post),
        max_length=192,
        truncation=True,
        return_tensors="pt",
    )
    output = model.generate(
        **inputs,
        num_beams=num_beams,
        max_new_tokens=64,
        no_repeat_ngram_size=3,
        length_penalty=1.0,
        early_stopping=True,
    )
    decoded = tokenizer.decode(output[0], skip_special_tokens=True)

    seen, tags = set(), []
    for tag in re.findall(r"#[\w\u0080-\uFFFF]+", decoded):
        if tag.casefold() not in seen:
            seen.add(tag.casefold())
            tags.append(tag)
    return tags

print(generate_hashtags("Artificial intelligence is changing software development."))

Generation defaults

These defaults are stored in the model's generation_config, so a plain model.generate(**inputs) already behaves correctly.

Do not use greedy decoding. It degenerates on this task, emitting ~12 tags per post against a reference mean of 5.4 (F1 0.21 vs 0.35 with beams).

The tag count is not forced: the model decides how many tags to emit, since the data has a variable number per post.

Limitations

• Domain skew. The training data is dominated by crypto/finance posts (#bitcoin, #crypto, #btc are the most frequent tags). Predictions on other domains fall back to generic tags.
• Multilingual coverage is uneven. Most posts are English; other scripts (CJK, Arabic, Cyrillic, Thai) appear in a small minority of rows, so mT5's multilingual capacity is only lightly exercised.
• Noisy supervision. Author-chosen hashtags are not an exhaustive or objective label set. A "wrong" prediction is often a reasonable tag the original author simply did not use, so precision/recall understate quality.
• Time-bound vocabulary. Tags reflect topics and tickers current at collection time and will drift.
• Spam filtering. Posts with more than 20 hashtags were excluded, so the model will not reproduce keyword-stuffing behaviour.
• Not a safety-filtered model. It reproduces whatever tag conventions, including promotional ones, exist in the source data.

Reproduction

make install
make prepare
make train
make evaluate