ModernLogBERT

ModernLogBERT

ModernLogBERT is a log anomaly detection model based on the ModernBERT architecture, designed to process HDFS log data for identifying anomalies. This repository contains scripts for data preprocessing and model training, leveraging a custom tokenizer and a pretrained ModernBERT model for efficient log sequence analysis.

Overview

The project includes 4 models (ModernLogbert_BC, ModernLogbert_BC_Logtokenizer, ModernLogbert_MLWP, ModernLogbert_MLWP_Logtokenizer), each model including two main scripts:

• dataprocess.py: Preprocesses HDFS log files, extracts log messages, groups them by block ID, and tokenizes sequences for model input.
• train.py: Trains the LogBERT model using Masked Log Word Prediction (MLWP) and KL divergence tasks for anomaly detection.

The model is built on the answerdotai/ModernBERT-base model, enhanced with modern architectural improvements like Rotary Positional Embeddings (RoPE) and Flash Attention for long-context processing.

Requirements

• Python 3.8+
• PyTorch
• Transformers (huggingface)
• Pandas
• NumPy
• Scikit-learn

Install dependencies using:

pip install torch transformers pandas numpy scikit-learn

Setup

1. Clone the Repository:

   git clone https://huggingface.co/MingtongDu/ModernLogBERT
   cd ModernLogBERT
   

2. Prepare Data:

   • Place your HDFS log file (e.g., HDFS.log) and anomaly label file (e.g., anomaly_label.csv) in an accessible directory.
   • Update the file paths in dataprocess.py and train.py to match your environment.

3. Tokenizer:

   • The custom tokenizer is saved in the output directory during preprocessing. Ensure the tokenizer path is correctly specified in train.py.

4. Pretrained Model:

   • The model uses a pretrained checkpoint (ep165.pt). Update the CHECKPOINT_PATH in train.py to point to your checkpoint file.

Usage

1. Data Preprocessing

Run dataprocess.py to parse, tokenize, and save the processed datasets:

python dataprocess.py

This script:

• Parses HDFS.log to extract log messages and block IDs.
• Loads labels from anomaly_label.csv.
• Groups logs by block ID and splits into train (normal sequences only), validation, and test sets.
• Tokenizes sequences and saves them as train.pt, val.pt, and test.pt in the specified output directory.

2. Model Training

Run train.py to train the LogBERT model:

python train.py

This script:

• Loads preprocessed datasets and the custom tokenizer.
• Initializes the LogBERT model with a pretrained ModernBERT backbone.
• Trains the model using MLWP and KL divergence losses.
• Evaluates performance on the test set and saves the trained model as EP3_froze21.pt.

Hyperparameters

Key hyperparameters are defined at the top of each script:

• dataprocess.py: MAX_LENGTH, TRAIN_RATIO, VAL_RATIO, TEST_RATIO, PRINT_SAMPLES, RANDOM_SEED.
• train.py: BATCH_SIZE, NUM_EPOCHS, LEARNING_RATE, NUM_LAYERS_TO_FREEZE, DROPOUT_RATE, MASK_PROB, ALPHA, G, R.

Modify these in the scripts to adjust preprocessing or training behavior.

Files

• dataprocess.py: Script for log parsing, label loading, grouping, tokenization, and data saving.
• train.py: Script for loading data, defining the LogBERT model, and training with anomaly detection metrics.
• output/: Directory containing processed datasets (train.pt, val.pt, test.pt) and the saved tokenizer.
• saved_model/: Directory for saving the trained model checkpoint.

Notes

• The model is optimized for English log data and may require fine-tuning for other languages.
• Ensure sufficient GPU memory for training, especially with long sequences (up to 512 tokens).
• For improved efficiency, install Flash Attention 2 if your GPU supports it.

Citation

If you use ModernLogBERT, please cite the ModernBERT model:

ModernLogBERT: Mingtong Du and Haoran Li. https://huggingface.co/MingtongDu/ModernLogBERT

License

This project is licensed under the MIT License. See the LICENSE file for details.