qwen-0.5b-law-phase1

Qwen 0.5B - Company Law (Phase 1: Continued Pretraining)

⚠️ Critical Disclaimer

This model is NOT suitable for legal advice or production use. It hallucinates case citations, confuses legal concepts, and produces factually incorrect information. This is an experimental model for educational purposes only.

Model Description

This is Qwen2.5-0.5B fine-tuned on a company law textbook through continued pretraining. The model learned legal vocabulary and writing style but exhibits significant factual inaccuracies and overfitting to the training data's structure.

• Developed by: Pactoria
• Model type: Causal Language Model (Decoder-only Transformer)
• Language: English
• License: Apache 2.0
• Base Model: Qwen/Qwen2.5-0.5B
• Training Approach: Full fine-tuning with FP16 mixed precision
• Domain: UK Company Law

Training Details

Training Data

• Source: "Company Law" textbook by Smith and Keenan (14th edition)
• Size: 2.3 MB plain text (550,382 tokens)
• Content: Definitions, case law, Companies Act 2006 references, exam questions
• Processing: Chunked into 1,073 examples of 1,024 tokens each with 50% overlap
• Total Training Tokens: 1,098,752 (across 3 epochs)

Training Procedure

Hyperparameters:

• Epochs: 3
• Batch size: 1 per device
• Gradient accumulation: 8 steps
• Effective batch size: 8
• Learning rate: 5e-5
• LR scheduler: Cosine with 3% warmup
• Optimizer: AdamW (weight_decay=0.01)
• Max gradient norm: 1.0
• Sequence length: 1,024 tokens
• Training regime: FP32 → FP16 mixed precision

Infrastructure:

• GPU: NVIDIA Tesla T4 (16GB)
• Training time: 20 minutes
• Total steps: 405
• Framework: HuggingFace Transformers 4.52.4

Loss Trajectory:

Step 1:   2.303
Step 100: 1.874
Step 200: 0.980
Step 300: 0.465
Step 400: 0.237
Final:    1.035 (average)

Intended Use

Primary Use Case

Educational demonstration of continued pretraining pipeline. This model is part of a three-phase learning experiment:

• Phase 1: Continued pretraining on domain text ✓ (this model)
• Phase 2: Supervised fine-tuning on Q&A pairs (planned)
• Phase 3: DPO alignment (planned)

How to Use

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "your-username/qwen-0.5b-law-phase1",
    device_map="auto",
    trust_remote_code=True,
    torch_dtype=torch.float16
)

tokenizer = AutoTokenizer.from_pretrained(
    "your-username/qwen-0.5b-law-phase1",
    trust_remote_code=True
)

# Generate (greedy decoding for stability)
prompt = "A corporation is"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=100,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Performance & Limitations

What Works

✅ Generates grammatically correct legal text
✅ Uses appropriate legal terminology
✅ References Companies Act 2006 and case names
✅ Maintains coherent sentence structure
✅ No catastrophic collapse (no repetition/gibberish)

Critical Limitations

❌ Hallucinates case citations - Invents fake cases and misattributes legal principles
❌ Confuses case law - Misrepresents what famous cases established
❌ Overfitted to textbook format - Outputs exam questions instead of explanations
❌ Factual inaccuracies - Provides incorrect legal information
❌ No fact-checking - Cannot distinguish real from invented law

Example Hallucinations

Prompt: "The doctrine of separate legal personality"
Output: "was developed by... Robert Goff and Sir Henry Price in... Bell v Lever Bros Ltd [1920]"
Reality: ❌ Completely false. The doctrine comes from Salomon v Salomon [1897], not Bell v Lever Bros.

Prompt: "In Macaura v Northern Assurance"
Output: "shareholders are entitled to the dividends..."
Reality: ❌ Wrong. Macaura is about insurable interest, not dividends.

Out-of-Scope Uses

DO NOT use this model for:

• Legal advice or consultation
• Academic citations or legal research
• Court filings or legal documents
• Any situation where accuracy matters
• Production applications
• Decision-making about legal matters

Misuse risks:

• Providing harmful legal misinformation
• Making incorrect legal decisions
• Academic misconduct if used for assignments
• Liability issues if used for actual legal work

Bias & Ethical Considerations

• Training bias: Single textbook from UK jurisdiction (not representative of all legal systems)
• Temporal bias: Training data reflects law as of textbook publication (may be outdated)
• Accessibility bias: Written in formal legal English (may exclude non-experts)
• Hallucination risk: Model confidently states false information (dangerous in legal context)

Evaluation Results

Manual Testing (35 prompts across 8 categories)

Overall Assessment: Model learned legal writing style but not factual accuracy.

Environmental Impact

• Hardware: 1x NVIDIA Tesla T4 (16GB)
• Training time: 20 minutes
• Cloud provider: Kaggle
• Estimated CO2: ~0.02 kg (20 min × 70W GPU × 0.475 kg CO2/kWh grid average)

Technical Specifications

Model Architecture

• Type: Decoder-only Transformer (Qwen 2.5 architecture)
• Parameters: 494,032,768 (all trainable)
• Hidden size: 896
• Layers: 24
• Attention heads: 14
• Vocab size: 151,665 tokens
• Context length: 131,072 tokens (training used 1,024)

Software

• Transformers: 4.52.4
• PyTorch: 2.6.0+cu124
• CUDA: 12.4
• Python: 3.11.13

Citation

If you use this model for research on continued pretraining or domain adaptation:

@misc{qwen-law-phase1,
  author = {Your Name},
  title = {Qwen 0.5B Company Law Phase 1: Continued Pretraining Experiment},
  year = {2025},
  publisher = {HuggingFace},
  howpublished = {\url{https://huggingface.co/your-username/qwen-0.5b-law-phase1}},
  note = {Experimental model - not for legal use}
}

Acknowledgments

• Base model: Qwen Team
• Training framework: HuggingFace Transformers
• Infrastructure: Kaggle (free T4 GPU access)

Model Card Authors

[Your Name] - Experimental continued pretraining for educational purposes

Contact

For questions about the training methodology (not legal questions):

• HuggingFace: @your-username
• Issues: Model repository

---

Version: 1.0 (Phase 1 only)
Last Updated: October 2025
Status: Experimental - Educational use only

This model card is honest about limitations while still documenting what was accomplished. Replace `your-username` and `[Your Name]` with your actual details before uploading.