gemma3-intent-paraphraser-lora

Gemma-3 Intent Paraphraser (LoRA Adapter)

This repository contains the LoRA adapter weights for fine-tuning unsloth/gemma-3-1b-it into a high-precision intent-preserving paraphrase generator.
The model rewrites a user query into multiple paraphrases that maintain identical intent, answer type, and semantic meaning, strictly following a JSON schema.

This LoRA is designed for research on:

• intent boundary detection
• semantic caching robustness (SAFE-CACHE / SIVer)
• adversarial query transformation studies
• controlled, conservative paraphrasing
• structured JSON generation

---

📌 Base Model

The adapter is applied on top of the following base model:

• Base: unsloth/gemma-3-1b-it
• Context length: 2048
• Precision: fp16
• No 4-bit or 8-bit quantization during fine-tuning

The base model is not included in this repository.
PEFT automatically loads it using the field in adapter_config.json.

---

📦 LoRA Configuration

The model was fine-tuned using Unsloth + TRL SFTTrainer with the following LoRA parameters:

These settings encourage tight semantic control, low drift, and conservative paraphrasing.

---

🧠 Training Objective

The model is trained to:

✔ Generate 4–5 high-precision paraphrases

• same meaning
• same intent
• same entities
• same answer type
• JSON-only: {"examples": ["...", "..."]}
• no markdown, no surrounding text

✔ Avoid semantic drift

Dataset paraphrases were tightly filtered and manually checked to ensure answer equivalence.

❌ Not trained for:

• malicious adversarial queries
• unsafe content
• domain knowledge generation
• long-form rewriting

---

🧪 Training Dataset

The model was trained on approximately 600 records, generated using a cluster-based augmentation pipeline with GPT-4.1.

For each query, the dataset contains two structured components:

1. Positive Intent-Preserving Paraphrases

Each q entry includes 4–5 high-precision paraphrases that maintain:

• identical semantic intent
• identical answer type
• consistent entities
• no additional context or semantic drift

These entries form the sole training targets used during supervised fine-tuning.

2. Near-Miss Intent Variations (Semantic Drift)

For each cluster, GPT-4.1 additionally generated near-miss queries—inputs that remain lexically similar but intentionally introduce subtle meaning changes, such as:

• modifying dates or temporal expressions
• shifting platform/device
• altering specificity (“in the UK” → “in Europe”)
• pluralization or scope changes
• entity generalization

These near-miss samples were not used as labels during SFT, but served as negative signals during dataset verification and evaluation. They ensure that the model learns tight intent boundaries, avoiding over-generalization.

---

Sample Entries

Example 1 — “Truth Social” on iOS

{
  "q": "When will Truth Social launch on iOS?",
  "examples": [
    "When will Truth Social be available on iOS?",
    "What is the release date for Truth Social app on iPhone?",
    "When does the Truth Social app launch on iOS?",
    "Truth Social iOS release date?",
    "Can you tell me when iOS users can access Truth Social?"
  ]
}

Example 2 — “Truth Social” on Android

{
  "q": "When will Truth Social launch on Android?",
  "examples": [
    "What is the release date for Truth Social on Android?",
    "When is Truth Social expected to become available on Android?",
    "When can Android users access Truth Social?",
    "On which date will Truth Social be launched for Android devices?",
    "When will the Android version of Truth Social be released?"
  ]
}

The dataset is intentionally constructed to emphasize conservative, intent-preserving paraphrasing, avoiding creative rewriting and ensuring high-fidelity semantic equivalence across paraphrase clusters.

---

🔧 Training Script (Summary)

Training used the following libraries:

• unsloth (FastModel)
• TRL SFTTrainer
• PEFT LoRA
• HuggingFace Datasets

Key training hyperparameters:

A strict conversation-style template was applied using:

<start_of_turn>user
<start_of_turn>model

Training was responses-only, ensuring the model learns to output only JSON examples.

🚀 Usage

🔹 Load the LoRA Adapter + Base Model (Unsloth)

import torch
from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    "mohanadffy/gemma3-intent-paraphraser-lora",
    max_seq_length = 2048,
    dtype = torch.float16,
    load_in_4bit = False,
)

FastLanguageModel.for_inference(model)

---

🔹 Run Inference

SYSTEM_MSG = (
    "You are an intent paraphrase generator.
"
    "Given a user query, output 5 diverse paraphrased examples expressing the SAME intent.
"
    "CRITICAL RULES:
"
    "1. Output ONLY a valid JSON object.
"
    "2. Follow EXACT schema: {\"examples\": [\"...\", \"...\"]}
"
    "3. No commentary. No markdown. No extra text.
"
)

def generate_intent_examples(query, max_tokens=300):
    prompt = tokenizer.apply_chat_template(
        [
            {"role": "system", "content": SYSTEM_MSG},
            {"role": "user",  "content": query},
        ],
        tokenize=False,
        add_generation_prompt=True
    )

    inputs = tokenizer(prompt, return_tensors="pt").to(
        "cuda" if torch.cuda.is_available() else "cpu"
    )

    output = model.generate(
        **inputs,
        max_new_tokens=max_tokens,
        do_sample=True,
        temperature=0.3,
        top_p=0.95,
    )

    text = tokenizer.decode(output[0])

    # Extract only the JSON block
    import re
    match = re.search(r"\{.*\}", text, flags=re.DOTALL)
    return match.group(0) if match else text

---

🔹 Example

print(generate_intent_examples("When will Truth Social launch on iOS?"))

📖 Recommended Citation

@misc{gemma_intent_paraphraser_2025,
  title        = {Gemma-3 Intent Paraphraser (LoRA Adapter)},
  author       = {Mohanad Afiffy , Mohamed Waleed Fakhr , Fahima Maghraby},
  year         = {2025},
  howpublished = {\url{https://huggingface.co/mohanadffy/gemma3-intent-paraphraser-lora}},
}

---

📜 License

This repository contains only LoRA adapter weights, which represent a derivative behavioral modification. Users must comply with the Gemma base model license and Unsloth usage terms when reconstructing the full model.

---

🔍 Notes

• The model is designed for research use, not production deployment.

• Always validate JSON output during inference.

• Recommended generation settings:

  • temperature = 0.1–0.3
  • top_p = 0.9–0.95
  • max_new_tokens = 200