Qwen2.5-7B-BDLAW-LoRA

Qwen 2.5 7B Instruct - Bangladesh Legal LoRA Adapter

A fine-tuned LoRA adapter for Qwen 2.5 7B Instruct specialized in Bangladesh legal reasoning and legal document analysis.

---

📋 Model Details

---

🚀 Quick Start

Installation

pip install transformers peft torch accelerate

Loading the Model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

# Base model
model_id = "Qwen/Qwen2.5-7B-Instruct"
adapter_id = "millat/qwen-2.5-7b-instruct-bdlaw-lora"

# Load base model (with optional 4-bit quantization)
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    quantization_config=bnb_config,
    torch_dtype=torch.float16,
    trust_remote_code=True,
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

# Attach LoRA adapter
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()

Inference Example

# Bangla legal query
prompt = "বাংলাদেশে চুক্তি আইন কী এবং এর প্রধান বৈশিষ্ট্য কি?"

# Format with chat template
messages = [{"role": "user", "content": prompt}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)

# Generate response
inputs = tokenizer(text, return_tensors="pt").to("cuda")
outputs = model.generate(
    **inputs,
    max_new_tokens=512,
    temperature=0.7,
    do_sample=True,
    top_p=0.95,
    pad_token_id=tokenizer.eos_token_id,
)

response = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(response)

---

📊 Training Details

Dataset

BDLawCorpus Dataset V1 contains:

• 4,710 Bangla legal documents and case studies
• Multi-format: Chat templates with conversation pairs
• Topics: Contract Law, Criminal Law, Constitutional Law, Property Law, Labor Law
• Quality: Manually validated and cleaned

Training Configuration

# Quantization (4-bit)
load_in_4bit=True
bnb_4bit_use_double_quant=True
bnb_4bit_quant_type="nf4"
bnb_4bit_compute_dtype=torch.float16

# LoRA Configuration
r=64                    # Rank
lora_alpha=128         # Scaling factor
lora_dropout=0.05
bias="none"
task_type="CAUSAL_LM"

# Training Arguments
num_train_epochs=3
per_device_train_batch_size=8
gradient_accumulation_steps=2
learning_rate=2e-4
weight_decay=0.01
warmup_steps=100
save_steps=200
save_total_limit=3
logging_steps=10
fp16=True
gradient_checkpointing=True
ddp_find_unused_parameters=False

Training Environment

• Hardware: Kaggle 2x T4 GPU (15GB each)
• Framework: Hugging Face Transformers + PEFT + Accelerate
• Parallelism: Distributed Data Parallel (DDP)
• Total Training Time: ~6.5 hours
• Final Loss: 0.112

---

💡 Use Cases

This model is optimized for:

1. Legal Document Summarization - Summarize Bangla legal texts and acts
2. Legal Question Answering - Answer questions about Bangladesh law
3. Contract Analysis - Analyze and explain contract clauses
4. Case Law Explanation - Provide insights into court decisions
5. Regulatory Compliance - Advise on Bangladesh legal compliance
6. Document Classification - Categorize legal documents
7. Legal Research - Retrieve relevant case law and precedents

---

📈 Performance Metrics

---

⚖️ Limitations

• Language-specific: Optimized for Bangla legal documents; English queries may have lower accuracy
• Domain-specific: Trained on Bangladesh legal corpus; may not generalize to other domains
• Knowledge cutoff: Based on training data; may not reflect recent legal changes
• LLM limitations: Can hallucinate; always verify outputs with official sources
• No real-time updates: Requires retraining for new laws or amendments

---

🔧 Fine-tuning & Adaptation

To further fine-tune this adapter on your own data:

from transformers import TrainingArguments, Trainer

training_args = TrainingArguments(
    output_dir="./my_adapter",
    num_train_epochs=2,
    per_device_train_batch_size=8,
    learning_rate=1e-4,
    save_steps=100,
    logging_steps=10,
    fp16=True,
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=your_dataset,
    data_collator=data_collator,
)

trainer.train()

---

📝 Citation

If you use this model, please cite:

@model{qwen_bdlaw_lora_2025,
  title={Qwen 2.5 7B Instruct - Bangladesh Legal LoRA Adapter},
  author={MD Millat Hosen},
  organization={BDLaw Corpus Project},
  year={2025},
  howpublished={\url{https://huggingface.co/millat/qwen-2.5-7b-instruct-bdlaw-lora}},
}

---

📄 License

This adapter is provided under the Creative Commons Attribution 4.0 International (CC-BY-4.0) license.

The base model (Qwen 2.5 7B) is licensed under the Qwen License Agreement. Please refer to the original model card for details.

---

👥 Contact & Support

• Model Developer: MD Millat Hosen
• Project: BDLawCorpus
• Email: Contact via GitHub issues
• Repository: BDLawCorpus GitHub

---

🙏 Acknowledgments

• Qwen Team for the excellent base model architecture
• Hugging Face for the transformers library and model hosting
• Kaggle for providing free GPU compute
• Bangladesh Legal Community for data and domain expertise

---

⚠️ Disclaimer

This model is provided for educational and research purposes. While we strive for accuracy, legal advice should always be verified with qualified legal professionals. The model authors assume no responsibility for any consequences arising from the use of this model for legal decision-making.

---

🔄 Model Updates

---

Last Updated: May 9, 2025
Model Version: 1.0
Status: ✅ Production Ready