flan-t5-climate-qlora

Model Card for FLAN-T5 Climate Action QLoRA

This is a QLoRA-finetuned version of FLAN-T5 specifically trained for climate action content analysis and generation. The model is optimized for processing and analyzing text related to climate change, sustainability, and environmental policies.

Model Details

Model Description

• Developed by: Kshitiz Khanal
• Shared by: kshitizkhanal7
• Model type: Instruction-tuned Language Model with QLoRA fine-tuning
• Language(s): English
• License: Apache 2.0
• Finetuned from model: google/flan-t5-base

Model Sources

• Repository: https://huggingface.co/kshitizkhanal7/flan-t5-climate-qlora
• Training Data: FineWeb dataset (climate action filtered)

Uses

Direct Use

The model is designed for:

• Analyzing climate policies and initiatives
• Summarizing climate action documents
• Answering questions about climate change and environmental policies
• Evaluating sustainability measures
• Processing climate-related research and reports

Downstream Use

The model can be integrated into:

• Climate policy analysis tools
• Environmental reporting systems
• Sustainability assessment frameworks
• Climate research applications
• Educational tools about climate change

Out-of-Scope Use

The model should not be used for:

• Critical policy decisions without human oversight
• Generation of climate misinformation
• Technical climate science research without expert validation
• Commercial deployment without proper testing
• Medical or legal advice

Bias, Risks, and Limitations

• Limited to climate-related content analysis
• May not perform well on general domain tasks
• Potential biases from web-based training data
• Should not be the sole source for critical decisions
• Performance varies on technical climate science topics

Recommendations

• Always verify model outputs with authoritative sources
• Use human expert oversight for critical applications
• Consider the model as a supplementary tool, not a replacement for expert knowledge
• Regular evaluation of outputs for potential biases
• Use in conjunction with other data sources for comprehensive analysis

Training Details

Training Data

• Source: FineWeb dataset filtered for climate content
• Selection criteria: Climate-related keywords and quality metrics
• Processing: Instruction-style formatting with climate focus

Training Procedure

Preprocessing

• Text cleaning and normalization
• Instruction templates for climate context
• Maximum input length: 512 tokens
• Maximum output length: 128 tokens

Training Hyperparameters

• Training regime: QLoRA 4-bit fine-tuning
• Epochs: 3
• Learning rate: 2e-4
• Batch size: 4
• Gradient accumulation steps: 4
• LoRA rank: 16
• LoRA alpha: 32
• Target modules: Query and Value matrices
• LoRA dropout: 0.05

Environmental Impact

• Hardware Type: Single GPU
• Hours used: ~4 hours
• Cloud Provider: Local
• Carbon Emitted: Minimal due to QLoRA efficiency

Technical Specifications

Model Architecture and Objective

• Base architecture: FLAN-T5
• Objective: Climate-specific text analysis
• QLoRA adaptation for efficient fine-tuning
• 4-bit quantization for reduced memory usage

Compute Infrastructure

• Python 3.8+
• PyTorch
• Transformers library
• bitsandbytes for quantization
• PEFT for LoRA implementation

Hardware

Minimum requirements:

• 16GB GPU memory for inference
• 24GB GPU memory recommended for training
• CPU inference possible but slower

Citation

If you use this model, please cite:

@misc{khanal2024climate,
  title={FLAN-T5 Climate Action QLoRA},
  author={Khanal, Kshitiz},
  year={2024},
  publisher={HuggingFace},
  howpublished={\url{https://huggingface.co/kshitizkhanal7/flan-t5-climate-qlora}}
}