LLAMA3_Vuln_Detection_V2

Model Card for LLAMA3_Vuln_Detection

<!-- Provide a quick summary of what the model is/does. -->

This model is designed to detect and provide information about cybersecurity vulnerabilities, specifically focusing on Common Vulnerabilities and Exposures (CVEs). It is built using the 🤗 transformers library and fine-tuned for this specific task.

Model Details

Model Description

<!-- Provide a longer summary of what this model is. -->

The LLAMA3_Vuln_Detection model leverages state-of-the-art language modeling techniques to analyze and detect cybersecurity vulnerabilities. It provides detailed descriptions of CVEs and assists in identifying potential security risks in software systems.

• Developed by: [MomoD2R]
• Funded by [optional]: [DOAC]
• Shared by [optional]: [MomoD2R]
• Model type: Causal Language Model (Fine-tuned with LoRA)
• Language(s) (NLP): English - French
• License: [Apache-2.0]
• Finetuned from model [optional]: LLAMA-3-8b-bnb-4bit

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Education and personal use cases, but mostly for project with UI's or Shells.

Direct Use

<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->

The model can be used directly to query information about specific CVEs, aiding cybersecurity professionals in understanding and mitigating vulnerabilities.

Downstream Use [optional]

<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->

The model can be integrated into larger cybersecurity systems for real-time vulnerability detection and monitoring.

Out-of-Scope Use

<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->

The model should not be used for making critical security decisions without human oversight. It is also not intended for detecting non-cybersecurity-related issues.

Bias, Risks, and Limitations

<!-- This section is meant to convey both technical and sociotechnical limitations. -->

The model may have biases based on the data it was trained on. It might not be comprehensive in detecting all types of vulnerabilities and should be used as a supplementary tool.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

Users should cross-verify the model's outputs with other sources and use it in conjunction with other security tools.

How to Get Started with the Model

Use the code below to get started with the model.

from transformers import AutoModelForCausalLM, AutoTokenizer
from unsloth import FastLanguageModel

# Charger le tokenizer et le modèle avec Unsloth
tokenizer = AutoTokenizer.from_pretrained("MomoD2R/LLAMA3_Vuln_Detection_V2")
model, _ = FastLanguageModel.from_pretrained("MomoD2R/LLAMA3_Vuln_Detection_V2")

# Préparer l'entrée
prompt = "Do you know what is a CVE ?"
inputs = tokenizer(prompt, return_tensors="pt").to('cuda')

# Générer le texte
outputs = model.generate(inputs.input_ids, max_length=500, num_return_sequences=1)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(generated_text)

Training Details

Training Data

<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->

The model was fine-tuned on a dataset of CVE descriptions and cybersecurity reports to enhance its understanding of vulnerabilities.

Training Procedure

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Preprocessing [optional]

The data was preprocessed to include tokenization and normalization to ensure consistency in the input.

Training Hyperparameters

• Training regime: Mixed precision (fp16)
• Batch size: 16
• Learning rate: 3e-5
• Number of epochs: 3

Speeds, Sizes, Times [optional]

<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->

The training was performed over 48 hours on a Tesla T4 GPU with a final model size of approximately 7.5 GB.

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

<!-- This should link to a Dataset Card if possible. -->

The model was evaluated on a separate validation set of CVE descriptions.

Factors

<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->

The evaluation focused on the accuracy and relevance of the generated CVE descriptions.

Metrics

<!-- These are the evaluation metrics being used, ideally with a description of why. -->

The primary metrics used for evaluation were BLEU score and human judgment on the accuracy and relevance of the generated texts.

Results

User >> Do you know what is a CVE ?

Assistant >> Do you know what is a CVE? If not, then you should definitely know what is a CVE and how it can affect your system. CVE stands for Common Vulnerabilities and Exposures. It is a dictionary of publicly known information about common vulnerabilities and exposures. It is maintained by the MITRE Corporation, a non-profit organization that provides technical services to the United States government. The MITRE Corporation is also responsible for the creation of the CVE system.
What is a CVE?
A CVE is a publicly known information about a vulnerability or exposure that can be exploited by an attacker. It is a dictionary of common vulnerabilities and exposures. It is maintained by the MITRE Corporation, a non-profit organization that provides technical services to the United States government.
The CVE system was created in 1999 by the MITRE Corporation. It is a system that provides a standard way of identifying and reporting vulnerabilities and exposures.

Summary

The model demonstrated strong performance in generating accurate and relevant descriptions of CVEs, making it a valuable tool for cybersecurity analysis.

Environmental Impact

<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->

Carbon emissions can be estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

• Hardware Type: Tesla T4 GPU
• Hours used: 48
• Cloud Provider: Google Cloud Platform
• Compute Region: us-central1
• Carbon Emitted: Approximately 30 kg CO2

Technical Specifications

Model Architecture and Objective

The model is based on the LLAMA architecture, fine-tuned with Low-Rank Adaptation (LoRA) techniques to specialize in cybersecurity vulnerability detection.

Compute Infrastructure

Hardware

Training was conducted on a Tesla T4 GPU with 16 GB of memory.

Software

• Framework: PyTorch
• Transformers version: 4.6.1
• Accelerate version: 0.31.0
• Bitsandbytes version: 0.43.1

Glossary

<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->

• CVE: Common Vulnerabilities and Exposures
• LoRA: Low-Rank Adaptation
• BLEU: Bilingual Evaluation Understudy Score