log-gen

Model Card for Model ID

<!-- Provide a quick summary of what the model is/does. -->

This is a Llama 3 model finetuned on execution logs to be used for a sock-shop app anomaly detections.

Model Details

Model Description

This model was finetuned on a variety of system logs of a sock shop app. Given a log chunk of 10 messages, it generates the next log message according to normal execution.

• Developed by: Luís Almeida, Diego Pedroso, Lucas Pulcinelli, William Aisawa, Sarita Bruschi, Inês Dutra
• Model type: Text Generation
• Language(s) (NLP): [English]
• License: [llama3]
• Finetuned from model: Llama 3 8b

Model Sources

<!-- Provide the basic links for the model. -->

• Repository: https://github.com/lasdpc-icmc/maia
• Paper [optional]: [More Information Needed]

Uses

<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->

Since the model was finetuned on execution logs of the sock-shop app, it is intended to be used to generate logs of said app. To adapt it to another system, it should be finetuned on a sample of execution logs of the new system.

Direct Use

Direct plugin to the sock-shop app.

Out-of-Scope Use

The usage of this model on execution logs that it hasn't been finetuned on may yield bad results.

Recommendations

<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->

We recommend users to finetune this model on logs of their app before they use it.

How to Get Started with the Model

Please refer to https://github.com/lasdpc-icmc/maia/apps/llm for the code files that were developed for this model. The file "eval_llm.py" provides code to detect system anomalies.

Training Details

Training Data

https://huggingface.co/datasets/lmma25/sock-shop-logs-train

Training Procedure

The model was finetuned using the SFTTrainer from the transformer's library in an autoregressive way.

<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->

Training Hyperparameters

• Training regime: 10 epochs, AdamW optimizer, 1e-4 learning rate, bf16, weight decay 0.01, max gradient norm 0.3, cosine learning rate scheduler

Evaluation

<!-- This section describes the evaluation protocols and provides the results. -->

Testing Data, Factors & Metrics

Testing Data

https://huggingface.co/datasets/lmma25/sock-shop-logs-test

Metrics

The model was used to detect anomalies on a small sample of execution logs, achieving a precision of 0.77 and a recall of 1. Precision and recall metrics were used since they allow for the accurate assessment of model behavior in regards to false positives and false negatives.

Results

Precision 0.77 Recall 1

Citation [optional]

<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->

BibTeX:

[More Information Needed]