nala-qwen3-4b

Nala — Qwen3-4B fine-tuned on Indonesian Government Documents

Nala is a QLoRA fine-tune of Qwen/Qwen3-4B trained on Indonesian government regulation Q&A pairs.

Use case: AI assistant for government contractors — reading tender documents (RKS), understanding procurement regulations, and navigating Indonesian bureaucratic law.

Part of Deflated — a personal AI research studio.

---

What it does

• Answers questions about Indonesian government regulations
• Cites specific law numbers, years, and articles correctly
• Understands formal Indonesian bureaucratic language (bahasa birokrasi)
• Designed to run on-premise (sensitive tender documents shouldn't go to a cloud API)

Training Data

• Source: 695 PDFs scraped from Indonesian law databases
  • 284 PDFs from JDIHN (national law database)
  • 411 PDFs from JDIH Kota Bandung
• Q&A Generation: Mistral Small via Mistral API
• Final dataset: 1,966 Q&A pairs after deduplication and cleaning
• Format: chat messages (user/assistant)

Training

• Base model: Qwen/Qwen3-4B (Apache 2.0)
• Method: QLoRA — 4-bit NF4 quantization, LoRA r=16, alpha=32
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable params: 33M / 4B total (0.81%)
• Epochs: 2
• Batch size: 4, gradient accumulation 4 (effective 16)
• Learning rate: 2e-4 cosine with 5% warmup
• Hardware: Kaggle free tier T4 (15.6 GB VRAM)
• Training time: ~30 minutes

How to use

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, pipeline
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
)

base_model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    quantization_config=bnb_config,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("ripkiiiii/nala-qwen3-4b")
model = PeftModel.from_pretrained(base_model, "ripkiiiii/nala-qwen3-4b")

pipe = pipeline("text-generation", model=model, tokenizer=tokenizer, max_new_tokens=256)

messages = [{"role": "user", "content": "Apa dasar hukum pemberian tunjangan kepada pegawai negeri sipil di daerah?"}]
output = pipe(messages)
print(output[0]["generated_text"][-1]["content"])

Limitations

• Trained on 1,966 pairs — narrow domain coverage
• Best for questions about Indonesian regional regulations (peraturan daerah, bupati, walikota)
• Not reliable for national-level law or specialized domains without more data
• Will hallucinate outside its training domain

About

Built by Muhammad Rifky — Digital PR student at Telkom University, building AI things with free compute and a lot of waiting.

Read the build log: deflated.xyz/blog

Framework versions

• PEFT 0.18.1
• TRL 0.19.1
• Transformers 4.51.3
• PyTorch 2.10.0+cu128
• Datasets 4.8.5
• Tokenizers 0.21.4