flan-t5-small-redhat-docs

FLAN-T5-Small Fine-Tuned on Red Hat Documentation

Overview

This repository hosts a fine-tuned FLAN-T5-Small model for question-answering tasks on Red Hat documentation. The model was fine-tuned using Low-Rank Adaptation (LoRA) and 4-bit quantization on a Google Colab T4 GPU (~15 GB VRAM, CUDA 11.8). The training dataset, redhat-docs_dataset, contains 55,741 rows in JSONL format with fields: title, content, command, and url. The model excels at extracting commands (e.g., yum install X) and summarizing procedures.

Project details are available on GitHub: mtptisid/FLAN-T5-Small_finetuning_LoRA.

Model Details

• Base Model: google/flan-t5-small (77M parameters).
• Fine-Tuning: LoRA (r=8, alpha=32, target_modules=["q", "v"], dropout=0.1).
• Quantization: 4-bit NormalFloat (nf4) with bfloat16 compute dtype (~6-8 GB VRAM).
• Task: Question-answering on Red Hat documentation.

Dataset

The redhat-docs_dataset contains 55,741 entries:

• title: Documentation section title.
• content: Detailed procedure or concept.
• command: Associated command (may be null).
• url: Reference URL.

Preprocessing

• Null command fields set to ""; missing title/content set to "Untitled"/"".
• Formatted into text field: "Title: {title} Content: {content} Command: {command}".
• Tokenized with max_length=512, truncation, and padding.

Artifacts

• data/redhat-docs_dataset.jsonl: Original dataset.
• data/formatted_dataset.jsonl: Preprocessed dataset.
• data/tokenized_dataset.jsonl: Tokenized dataset.

Training

• Hardware: NVIDIA T4 GPU, CUDA 11.8.
• Epochs: 2 (~4-8 hours).
• Batch Size: Effective 32 (4 per-device, 8 gradient accumulation steps).
• Optimizer: Paged AdamW 8-bit.
• Mixed Precision: FP16.
• Dependencies: PyTorch 2.3.1, Transformers 4.46.0, BitsAndBytes 0.43.3, Triton 2.0.0, Datasets 3.0.2, PEFT 0.13.2.

Repository Structure

• model/: Model weights and tokenizer.
• data/: Dataset files.
• finetune_script.py: Training script.
• README.md: This file.

Usage

Load the model for inference on Red Hat documentation queries. Example:

• Question: "How do I install Package X?"
• Context: "Title: Installing Package X Content: To install Package X, use the package manager yum. Command: yum install X"
• Output: "Run yum install X"

Installation

Requires PyTorch 2.3.1, Transformers 4.46.0, and a GPU for 4-bit quantization.

Limitations

• Dataset may have null commands, affecting some queries.
• Trained for 2 epochs; more epochs may improve performance.
• Specialized for Red Hat documentation.

Future Work

• Add synthetic Q&A data.
• Implement retrieval for dynamic context.
• Evaluate with BLEU/ROUGE metrics.

License

MIT License. Verify redhat-docs_dataset licensing separately.

Acknowledgments

• Google FLAN-T5 Team
• Hugging Face
• Red Hat Documentation Team

Contact

Open issues on GitHub or contact mtpti5iD via Hugging Face.

Last Updated: June 17, 2025