qwen3_nyayalm_v0.1

NyayaLM: Qwen3 Legal LoRA Model

Model Card

Model Details

• Base Model: unsloth/Qwen3-4B-Instruct-2507-unsloth-bnb-4bit
• Finetuned By: chhatramani
• License: Apache-2.0
• Language: English & Nepali (Legal domain)
• Frameworks: Unsloth, TRL, Transformers

LoRA Configuration

• LoRA Rank (r): 32
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• LoRA Alpha: 32
• LoRA Dropout: 0
• Bias: none
• Gradient Checkpointing: enabled (unsloth)
• Random State: 3407

Dataset

The model was trained on a mixed bilingual legal dataset:

• Legal Datasets:
  • Nepal Civil Code (English)
  • Muluki Dewani (Nepali)
• General Instruction Datasets (downsampled to 10% of legal size):
  • Nepali Alpaca Instruction Dataset
  • English Alpaca Instruction Dataset

Final Dataset Size:

• Training samples: ~3145
• Evaluation samples: ~350

Hyperparameters

• Max Sequence Length: 2048
• Batch Size (train): 4
• Batch Size (eval): 2
• Gradient Accumulation Steps: 4
• Warmup Ratio: 0.1
• Learning Rate: 1e-4
• Optimizer: AdamW (8-bit)
• Weight Decay: 0.01
• Scheduler: Cosine
• Epochs: 1
• Seed: 3407
• Evaluation Strategy: every 40 steps

Traning Loss

==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1 \ /| Num examples = 3,145 | Num Epochs = 1 | Total steps = 197 O^O/ _/ \ Batch size per device = 4 | Gradient accumulation steps = 4 \ / Data Parallel GPUs = 1 | Total batch size (4 x 4 x 1) = 16 "-____-" Trainable parameters = 66,060,288 of 4,088,528,384 (1.62% trained) Unsloth: Will smartly offload gradients to save VRAM! [197/197 54:51, Epoch 1/1]

Evaluation Metrics

The model was evaluated on a held-out test set of ~350 samples.

Results: EVALUATION RESULTS

ROUGE-1: 0.5491 ROUGE-2: 0.2151 ROUGE-L: 0.4245 BERTScore F1: 0.9042

Intended Use

NyayaLM is designed as a legal assistant for Nepali law, capable of answering questions, summarizing statutes, and providing structured legal reasoning.

Limitations

• Focused on Nepali and English legal texts;
• LoRA adapters only; full model weights not included.
• Evaluation metrics are based on a small test set and may not generalize.

Uploaded model

• Developed by: chhatramani
• License: apache-2.0
• Finetuned from model : unsloth/qwen3-4b-instruct-2507-unsloth-bnb-4bit

This qwen3 model was trained 2x faster with Unsloth

<img src="https://raw.githubusercontent.com/unslothai/unsloth/main/images/unsloth%20made%20with%20love.png" width="200"/>