Downloads · 30 days
30
5% of all-time downloads
sascha-frank-ai-research/tsft-rag-gemma-3-1b-it
tsft-rag-gemma-3-1b-it is a text generation model from sascha-frank-ai-research. Use it when you need the model to write or continue text. It is set up for transformers. The card lists the license as gemma.
Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation
Downloads · 30 days
30
5% of all-time downloads
All-time downloads
623
Public
Parameters
1000M
2 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors2 GB · 98%
From the Hugging Face model README
Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation
Part of the TSFT-RAG model family for Retrieval-Augmented Generation across Gemma, Qwen and Llama architectures.
This repository contains TSFT-RAG Gemma-3-1B-IT, a full-parameter fine-tuned derivative of Google's Gemma-3-1B-IT developed within the TSFT-RAG research project. The objective is to investigate how task-specific full fine-tuning improves Retrieval-Augmented Generation (RAG), grounded question answering, unsupported-question detection and structured information extraction.
| Model | Parameters | Status |
|---|---|---|
| Gemma-3-1B-IT | 1B | Released |
| Gemma-3-4B-IT | 4B | Released |
| Gemma-3-12B-IT | 12B | Released |
TSFT-RAG Gemma-3-1B-IT is a full-parameter fine-tuned derivative of Google's Gemma-3-1B-IT model developed for Retrieval-Augmented Generation (RAG). The objective of the TSFT-RAG project is to investigate whether relatively small language models can be specialized for reliable context-grounded answering, explicit abstention, structured output generation and information extraction through full-parameter supervised fine-tuning.
Retrieval-Augmented Generation requires different optimization targets than general-purpose conversational assistants. Besides answering questions from retrieved evidence, practical RAG systems must reliably refuse unsupported questions, produce structured outputs and follow deterministic task instructions.
| Property | Value |
|---|---|
| Model family | TSFT-RAG |
| Base model | google/gemma-3-1b-it |
| Training implementation | unsloth/gemma-3-1b-it |
| Architecture | Gemma3ForCausalLM |
| Parameters | ~1B |
| Context length | 32,768 tokens |
| Training sequence length | 1,024 tokens |
| Precision | bfloat16 |
| Parameter | Value |
|---|---|
| Epochs | 3 |
| Learning rate | 1e-6 |
| Batch size | 1 |
| Gradient accumulation | 16 |
| Optimizer | Paged AdamW 8-bit |
| Scheduler | Cosine |
| Warmup ratio | 0.08 |
| Weight decay | 0.01 |
| Max gradient norm | 1.0 |
| Packing | disabled |
| Seed | 42 |
Training runtime: 7 h 52 min
Final training loss: 1.1361
The benchmark contains 1,940 held-out German RAG evaluation instances covering grounded QA, unsupported questions, topic extraction, key-message extraction, argument extraction, structured JSON generation and citation-aware responses.
| Metric | Base | TSFT-RAG | Difference |
|---|---|---|---|
| Aggregate score | 0.328 | 0.605 | +0.278 |
| Hard-negative handling | 0.020 | 0.815 | +0.795 |
| Grounded QA | 0.509 | 0.395 | -0.114 |
| Analysis: main topic | 0.550 | 0.546 | -0.004 |
| Analysis: key message | 0.538 | 0.493 | -0.046 |
| Analysis: main arguments | 0.404 | 0.546 | +0.142 |
The largest improvement is observed in unsupported-question handling. The fine-tuned model rejects unsupported questions substantially more reliably than the original base model while improving argument extraction. As expected for task-specific specialization, these improvements are accompanied by moderate reductions in grounded QA and key-message extraction. This reflects the intended optimization objective rather than a general capability increase.
The model was trained using the Unsloth implementation of Gemma with full-parameter supervised fine-tuning and the Hugging Face Transformers ecosystem.
Performance outside the evaluated task formats has not been systematically studied. The model inherits limitations of the original Gemma base model and may generate incorrect or unsupported information.
This repository contains a modified derivative of Google's Gemma model. Use is subject to the Gemma Terms of Use. See LICENSE, NOTICE and MODIFICATIONS.md.
This model was developed and evaluated as part of the following research:
Frank, S., & Singh, R. (2026). Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation: A Multi-Family Evaluation Across Open-Weight Language Models.
Zenodo. DOI: 10.5281/zenodo.21638352
If you use this model in academic work, please cite the publication above.
@misc{Frank2026TSFTRAG,
author = {Frank, Sascha and Singh, Rawel},
title = {Task-Specific Full Fine-Tuning for Retrieval-Augmented Generation:
A Multi-Family Evaluation Across Open-Weight Language Models},
year = {2026},
month = jul,
version = {v1},
publisher = {Zenodo},
doi = {10.5281/zenodo.21638352},
url = {https://doi.org/10.5281/zenodo.21638352},
note = {Preprint}
}
The complete TSFT-RAG research project, including training scripts, evaluation pipeline, benchmark methodology and documentation, is available at:
Project repository https://github.com/frankmst/rag-task-specific-full-finetuning
TSFT-RAG model family https://huggingface.co/sascha-frank-ai-research
Sascha Frank
Independent AI Researcher
ORCID https://orcid.org/0000-0002-0588-0081
GitHub https://github.com/frankmst
Hugging Face https://huggingface.co/sascha-frank-ai-research