llama381binstruct_summarize_short_merged

Model Card for Legal Document Summarizer

<!-- Provide a quick summary of what the model is/does. -->

This model is fine-tuned to convert legal documents into human-readable summaries using Llama 3 8B Instruct as the base model. It was trained using QLoRA/LoRA techniques for efficient fine-tuning.

Model Details

Model Description

This is a fine-tuned version of NousResearch/Meta-Llama-3-8B-Instruct, optimized for summarizing legal documents in plain English. The model uses Parameter-Efficient Fine-Tuning (PEFT) methods, specifically LoRA, to achieve performance comparable to full fine-tuning while using significantly fewer computational resources.

• Developed by: jcbthnflrs
• Model type: Causal Language Model (LLaMA 3 Architecture)
• Language(s): English
• License: [Base model license applies]
• Finetuned from model: NousResearch/Meta-Llama-3-8B-Instruct

Model Sources

• Base Model: NousResearch/Meta-Llama-3-8B-Instruct
• Training Code: Based on LLM Engineering Challenge from AI Makerspace

Uses

Direct Use

This model is designed for converting legal documents, terms of service, and other legal content into plain English summaries that are easier for general audiences to understand. It can be used directly through the Hugging Face API or interface.

Downstream Use

The model can be integrated into:

• Legal document processing systems
• Terms of service simplification tools
• Contract analysis applications
• Legal document management systems

Out-of-Scope Use

The model should not be used as a replacement for legal advice or professional legal interpretation. It is meant to assist in understanding legal documents but not to provide legal guidance.

Training Details

Training Data

The model was trained on the Plain English Summary of Contracts dataset, which contains pairs of legal documents (EULA, TOS, etc.) and their natural language summaries. The dataset was split into:

• Training set: 68 examples
• Test set: 9 examples
• Validation set: 8 examples

Training Procedure

Preprocessing

• Input text is formatted using a specific template following Llama 3's chat format
• Special tokens are used to mark legal document boundaries
• Maximum sequence length: 2048 tokens

Training Hyperparameters

• Training regime: 4-bit quantization using QLoRA
• Optimizer: AdamW
• Learning rate: 2e-4
• Batch size: 1 per device
• Training steps: 500
• Warmup steps: 30
• Evaluation steps: 25
• Learning rate scheduler: Linear
• LoRA rank (r): 16
• LoRA alpha: 32
• LoRA dropout: 0.1

Hardware and Software

Hardware Requirements

• GPU: T4 or better
• Memory: Optimized for consumer-level resources through QLoRA

Software Requirements

• transformers library
• PEFT library
• bitsandbytes for quantization
• TRL for supervised fine-tuning

Evaluation

Training metrics show:

• Starting training loss: ~1.52
• Final training loss: ~0.0006
• Final validation loss: ~2.74

Model Card Authors

@jcbthnflrs

Model Card Contact

https://x.com/jcbthnflrs