Downloads · 30 days
0
mia-project-2025/T5-base-adapter-natural-questions-shortQA
T5-base-adapter-natural-questions-shortQA is a machine learning model from mia-project-2025. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This model is a T5-base variant fine-tuned with AdaLoRA (Adaptive Low-Rank Adaptation) on the NaturalQuestionsShortQA dataset.
Downloads · 30 days
0
Access
Public
Updated Aug 25, 2025
Repo size
58.4 MB
Likes
0
Public
Click a slice to open those files.
.pt38.5 MB · 50%
From the Hugging Face model README
This model is a T5-base variant fine-tuned with AdaLoRA (Adaptive Low-Rank Adaptation) on the NaturalQuestionsShortQA dataset.
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
model_name = "mia-project-2025/T5-base-adapter-natural-questions-shortQA"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForSeq2SeqLM.from_pretrained(model_name)
question = "Who wrote the novel 1984?"
context = "1984 is a novel written by George Orwell in 1949."
inputs = tokenizer(
f"question: {question} context: {context}",
return_tensors="pt",
truncation=True,
padding=True,
)
outputs = model.generate(**inputs, max_length=64)
answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Answer:", answer)
---
Training Details
Dataset: lucadiliello/naturalquestionsshortqa
Split: 80% train / 20% validation
Hyperparameters
Learning rate: 5e-5
Batch size: 16 (per device)
Gradient accumulation: 2
Epochs: 10
Optimizer: adamw_torch_fused
Mixed precision: fp16 / bf16 (hardware dependent)
Max input length: 512
Max target length: 128
AdaLoRA Config
r = 8
lora_alpha = 64
target_modules = ["q", "v"]
lora_dropout = 0.05
Orthogonal regularization = 0.5
Dynamic rank allocation enabled
---
## Evaluation Results
The model was evaluated on the NaturalQuestionsShortQA validation set.
Final Evaluation (Epoch 10)
Metric Score
Eval Loss 0.374
Exact Match (EM) 51.70%
F1 Score 64.52%
Eval Runtime 2559.84s
Samples per Second 8.13
Steps per Second 0.127
Training Summary
Metric Value
Train Loss 0.611
Train Runtime 36,367.54s (~10 hours on 1 GPU)
Samples per Second 22.89
Steps per Second 0.179