GemmaColRAC-AeroExpert

Model Card for GemmaColRAC-AeroExpert Language Model: Gemma 2B for Colombian Aviation Regulations 🛫

Model Details

Model Description

Este documento ofrece una visión detallada de GemmaColRAC-AeroExpert, la quinta iteración de nuestro modelo especializado en regulaciones aeronáuticas colombianas. Presenta un salto cualitativo con respecto a las versiones previas, exhibiendo mejoras en precisión y un uso de recursos de GPU más eficiente, reflejando nuestro compromiso con el desarrollo sostenible y de calidad de tecnologías de IA para la aviación.

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/0undo4kZc7OtfGI5nnAa8.png" alt="Imagen del Reglamento Aeronáutico Colombiano" style="width: 40%; max-height: 550px;"> </p>

• Developed by: Edison Bejarano, Nicolai Potes and Santiago Pineda ✨
• Funded by: Fundación Universitaria Los Libertadores, SomosNLP, HuggingFace
• Model type: Specialized Language Model for Colombian Aeronautical Regulations
• Language(s): Spanish (es-CO)
• License: apache-2.0 <!-- Elegid una licencia lo más permisiva posible teniendo en cuenta la licencia del model pre-entrenado y los datasets utilizados -->
• Fine-tuned from model: [More Information Needed] <!-- Enlace al modelo pre-entrenado que habéis utilizado como base -->
• Dataset used: RAC Corpus: Base de Datos del Reglamento Aeronáutico Colombiano 🛫📚🇨🇴

Model Sources

• Demo: Model Demo on HuggingFace Spaces
• Video presentation: [Aviación Inteligente: LLMs para Navegar el RAC | Hackathon] (https://youtu.be/IGKU1qUur2c?si=Na4d3XIU3vbdaaJj)

Uses

Direct Use

Is designed to assist professionals and students in the aviation industry by providing enhanced access to the Colombian Aeronautical Regulations through advanced language processing capabilities.

Out-of-Scope Use

This model is not intended for making legally binding decisions without human oversight.

Bias, Risks, and Limitations

The model may inherit biases from the data used for training, which primarily includes official legal texts. Users should exercise caution and not rely solely on the model for critical decision-making.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModel, AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("somosnlp/GemmaColRAC-AeroExpert")
model = AutoModel.from_pretrained("somosnlp/GemmaColRAC-AeroExpert")

# Example of how to use the model
encoded_input = tokenizer("Example query about aviation regulations", return_tensors='pt')
output = model(**encoded_input)

Training Details

Training Data

The model was trained on a curated dataset consisting of detailed question-answer pairs related to the Colombian Aeronautical Regulations.

Training Procedure

The model was fine-tuned from a base language model using the following specifications:

• Tipo de GPU: NVIDIA GeForce RTX 3090

• Tiempo Total de Entrenamiento: 12607 segundos

• Optimizador: AdamW con Bitfitting y Neutrino Noise

• Pasos Máximos: 4904

• Tamaño de Secuencia: 2048

• Tamaño de Lote por Dispositivo: 2

• Versión de Transformers: 4.39.2

• Framework de Optimización: Unsloth 2024.4

• Métodos de Cuantificación: bf16 con gradient_accumulation_steps de 2

• Función de Activación: gelu_pytorch_tanh

• Notebook to train the model

Comparison with Previous Version 🔄

The previous iteration, GemmaColRAC-AeroExpertV4, utilized an NVIDIA A100-SXM4-40GB GPU and was trained for approximately 50 minutes (3007 seconds). It operated with a learning rate of 0.00005 and used an 8-bit Paged AdamW optimizer. Furthermore, it was trained with a batch size per device of 1 and utilized version 4.39.0 of the Transformers library.

Key differences with the current version include:

• GPU Upgrade: 🆙 Switched from NVIDIA A100-SXM4-40GB to NVIDIA GeForce RTX 3090, offering better performance during training.
• Training Time: ⏳ Increased to allow more extensive fine-tuning of the model, resulting in improved accuracy.
• Batch Size: 🔢 Increased the batch size per device from 1 to 2, allowing for more efficient optimization.
• Optimizer Upgrade: 🛠️ Introduction of advanced techniques such as Bitfitting and Neutrino Noise to enhance model convergence.
• Maximum Steps: 🚶‍♂️ Significantly increased the maximum steps from 1638 to 4904, suggesting a broader coverage of data and deeper learning.

These changes have resulted in a more robust and efficient version of our model, enhancing its capacity to assist and provide guidance in Colombian aeronautical regulation.

Training Hyperparameters

• Training regime: bf16 mixed precision
• Optimizer: Paged AdamW 8-bit
• Learning Rate: 5e-5
• Batch Size per Device: 3
• Gradient Accumulation Steps: 4
• Warmup Steps: Computed as 3% of total steps
• Max Steps: 14,688
• Total Training Time: Approx. 5 hours 21 minutes (based on epochs and iteration speed)
• Max Sequence Length: 2048
• Weight Decay: 0.001
• Learning Rate Scheduler: Cosine
• Adam Betas: Beta1 = 0.99, Beta2 = 0.995
• Max Gradient Norm: 0.4

Speeds, Sizes, Times

• Training Duration: Approx. 3 hours 30 minutes for full training
• Training Throughput: 0.76 iterations per second (it/s)
• Total Steps: 14,688 steps over 8 epochs
• Checkpoint Size: Final model size was not specified; typical sizes for models of this type are several gigabytes.
• Total Number of Trainable Parameters: 78,446,592 [More Information Needed]

Metrics

Here is a detailed summary of the training metrics for GemmaColRAC-AeroExpert:

• Total Floating Point Operations (FLOPs): 204,241,541,673,615,360
• Train Loss: 0.393565042567292 (final reported loss)
• Training Runtime: 10,763.56 seconds (approximately 2.99 hours)
• Samples per Second: 4.556
• Steps per Second: 0.456
• Total Training Epochs: 2
• Total Training Steps: 4,904
• Gradient Norm: 3.515625
• Final Learning Rate: 0 (end of training)
• Average Loss over Training: 0.1934

Results

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/zuEetm8ifT5e3QtHfBBVD.png" alt="Trainning Loss" style="width: 80%; max-height: 350px;"> </p>

Model Examination [optional]

This model was evaluated the performance in simplifying RAC's content based on feedback from aeronautical experts, thereby enhancing regulatory compliance and understanding.

<p align="center"> <img src="https://cdn-uploads.huggingface.co/production/uploads/6419c2f6b4adb0e101b17b6c/5iPvAhaTMnqRDBn2g7XIK.png" alt="Evaluation for model by Aeronautical experts" style="width: 40%; max-height: 550px;"> </p>

Previous table shows the model's strong performance with average scores of 7 from 276 tests. However, RAC 3's low scores (mean 3.464, median 1) indicate areas needing improvement, while high ratings in RACs 1 and 5 suggest strengths. These results confirm the model's potential for accuracy and generalization, though RAC 3 requires adjustments.

Environmental Impact 🌱

The development of GemmaColRAC-AeroExpert has been carried out with a strong focus on sustainability 🌿. Efforts have been made to optimize efficiency and minimize environmental impact, including reducing energy consumption and lowering the carbon footprint during the model's training process. These measures not only enhance operational efficiency but also align with our commitment to environmental responsibility 🌎.

Energy Consumption and Carbon Emissions 📉

• Power Consumption: 0.25 kW (250 watts)
• Runtime Hours: 3.6 hours
• Carbon Intensity: 475 gCO2eq per kWh (Global average)

Given the use of an NVIDIA V100 GPU for approximately 3.6 hours, the carbon emissions have been meticulously estimated. Here are the details:

• Hardware Type: NVIDIA GeForce RTX 3090 GPU
• Total Hours Used: ~3.6 hours
• Total Carbon Emitted: Approximately 356.25 grams of CO₂ equivalents

These carbon emissions were calculated using the Machine Learning Impact Calculator introduced in Lacoste et al. (2019), which considers hardware type, runtime, and other relevant factors to provide a comprehensive view of the environmental impact of training large AI models 📊.

This proactive approach to understanding and mitigating our ecological footprint underlines our commitment to pioneering environmentally friendly AI development practices, setting a benchmark for sustainability within the AI industry 🌟.

Hardware

• Hardware Used: NVIDIA GeForce RTX 3090

Software 🛠️

The GemmaColRAC-AeroExpert model was developed and trained using a comprehensive stack of modern software libraries designed for high-performance machine learning tasks, particularly in Natural Language Processing (NLP). Here are the key libraries and tools used:

• Python Libraries:

  • json: For parsing JSON files and handling serialization 📄.
  • pandas: A powerful data manipulation and analysis library providing data structures and operations for manipulating numerical tables and time series 📊.
  • torch: PyTorch is an open-source machine learning library used for applications such as computer vision and natural language processing, developed by Facebook's AI Research lab (FAIR) 🔥.
  • datasets: A lightweight and extensible library to easily share and access datasets and evaluation metrics for machine learning tasks 📚.
  • huggingface_hub: Used for managing model repositories on Hugging Face and interacting with Hugging Face Hub APIs 🌐.

• Hugging Face Ecosystem:

  • transformers: Provides thousands of pre-trained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, and text generation in over 100 languages. It's designed to be both user-friendly for machine learning researchers and efficient to use in production 🤖.
  • BitsAndBytesConfig, TrainingArguments: Advanced configurations from the Transformers library for fine-tuning the performance and efficiency of training neural networks ⚙️.
  • pipeline: A utility for creating easy-to-use pipelines for various NLP tasks 🧪.
  • AutoModelForCausalLM, AutoTokenizer: Utilities for loading and initializing pre-trained language models and their tokenizers 📝.
  • logging: For configuring the logging level and output formats to track model training and inference processes effectively 📌.

• PEFT and LoRA Extensions:

  • LoraConfig, PeftModel: Extensions from the PEFT (Parameter Efficient Fine-Tuning) library, which include LoRA (Low-Rank Adaptation of large models), allowing efficient fine-tuning and adaptation of large pre-trained models with minimal computational overhead 🚀.

• Transformers Reinforcement Learning (TRL):

  • SFTTrainer: A component from the TRL library for applying reinforcement learning techniques to transformer models, specifically for sequence-to-sequence tasks 🎮.

These tools collectively support the robust training environment necessary to develop state-of-the-art NLP models like GemmaColRAC-AeroExpert, ensuring that the model is both highly effective and efficient in processing and understanding complex regulatory texts.

License 📜

GemmaColRAC-AeroExpert is released under the Apache 2.0 license 🏷️. This license is one of the most permissive and widely used licenses in the open-source community, allowing for both academic and commercial use without significant restrictions.

• Why Apache 2.0? 🤔
  • Openness: The Apache 2.0 license allows users to use, modify, and distribute the software freely, which encourages innovation and widespread use.
  • Protection: It provides an explicit grant of patent rights from contributors to users, protecting them from patent litigation.
  • Commercial friendly: Apache 2.0 is business-friendly, allowing the commercial use of the software which is crucial for wider adoption in industry settings.

By choosing Apache 2.0, we ensure that GemmaColRAC-AeroExpert can be freely used and integrated into a wide array of projects and products, from academic research to commercial applications, thus supporting the growth and accessibility of AI technologies across different sectors 🌐.

Glossary [optional]

• RAC: Reglamento Aeronáutico Colombiano

More Information

<!-- Indicar aquí que el marco en el que se desarrolló el proyecto, en esta sección podéis incluir agradecimientos y más información sobre los miembros del equipo. Podéis adaptar el ejemplo a vuestro gusto. -->

This project was developed during the Hackathon #Somos600M organized by SomosNLP. The model was trained using GPUs sponsored by their own team.

Team 👥

The development of the GemmaColRAC-AeroExpert model was supported by a dedicated team of experts specializing in machine learning, natural language processing, and aeronautics. Below are the key team members who contributed significantly to this project:

• Edison Bejarano - Lead AI Scientist, expert in NLP and machine learning, with a strong background in aeronautics.
• Nicolai Potes - Data Scientist, specializes in AI-driven regulatory compliance solutions.
• Santiago Pineda - Project Manager and Senior ML Engineer, with extensive experience in deploying scalable AI solutions.
• Alec Mauricio - AI Researcher, focused on developing innovative models for text analysis and interpretation.
• Danny Stevens - Software Engineer, provides expertise in software development and integration for machine learning applications.

These individuals bring a wealth of knowledge and expertise, ensuring the highest quality and performance of the GemmaColRAC-AeroExpert model. Their collaborative efforts have been pivotal in pushing the boundaries of what's possible with AI in the aviation sector.

Contact [optional]

[email protected]