stackoverflow-flan-finetune

🤖 FLAN-T5 Base Fine-Tuned on Stack Overflow Questions (LoRA)

This is a fine-tuned version of google/flan-t5-base on a curated dataset of Stack Overflow programming questions. It was trained using LoRA (Low-Rank Adaptation) for parameter-efficient fine-tuning, making it compact, efficient, and effective at modeling developer-style Q&A tasks.

🧠 Model Objective

The model is trained to:

• Rewrite or improve unclear programming questions
• Generate relevant clarifying questions or answers
• Summarize long developer queries
• Serve as a code-aware Q&A assistant

---

📚 Training Data

• Source: Stack Overflow public questions dataset (cleaned)
• Format: Instruction-like examples, Q&A pairs, summarization prompts
• Cleaning: HTML stripping, markdown-to-text, code-preserved
• Size: ~15k examples

---

🏗️ Training Details

• Base Model: google/flan-t5-base
• Adapter Format: LoRA using peft
• Files:
  • adapter_model.safetensors
  • adapter_config.json
• Hyperparameters:
  • r: 8
  • lora_alpha: 16
  • lora_dropout: 0.1
  • bias: "none"
  • task_type: "SEQ_2_SEQ_LM"
• Inference Mode: Enabled

---

💡 How to Use

from transformers import AutoTokenizer, AutoModelForSeq2SeqLM
from peft import PeftModel

# Load tokenizer and base model
tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base")
base_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "your-model-folder")
model.eval()

# Inference
prompt = "Rewrite this question more clearly: why is my javascript function undefined?"
inputs = tokenizer(prompt, return_tensors="pt")
outputs = model.generate(**inputs)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

🧪 Intended Use This model is best suited for: Code-aware chatbot assistants Prompt engineering for developer tools Developer-focused summarization / rephrasing Auto-moderation / clarification of tech questions

⚠️ Limitations Not trained for code generation or long-form answers May hallucinate incorrect or generic responses Finetuned only on Stack Overflow — domain-specific

✨ Acknowledgements Hugging Face Transformers LoRA (PEFT) Stack Overflow for open data FLAN-T5: Scaling Instruction-Finetuned Models

🛠️ Created with love by Kunj | Model suggestion & guidance by ChatGPT