gemma-3-4b-tax-law-lora

Gemma 3 4B Tax Law LoRA (Korean)

🇰🇷 Korean tax-law Q/A specialized LoRA fine-tuned model based on Gemma 3 4B.

Model Details

• Developed by: hongjoong-entec
• Model type: Causal LM with LoRA fine-tuning
• Language(s): ['ko']
• License: apache-2.0
• Finetuned from: google/gemma-3-4b-it

Uses

Direct Use

• Korean tax-law Q/A and related explanations in Korean
• Tax consulting assistance (with professional review)

Out-of-Scope Use

• Legal advice without professional review
• Sensitive decisions without human validation

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Load model and tokenizer
repo = "hongjoong-entec/gemma-3-4b-tax-law-lora"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
model = AutoModelForCausalLM.from_pretrained(
    repo, 
    torch_dtype=torch.float16, 
    trust_remote_code=True
)

# Prompt template for best results
prompt_style = """Below is an instruction that describes a task, paired with an input that provides further context.
Write a response that appropriately completes the request.
Respond in Korean and provide only the final answer without chain-of-thought, analysis, section headers, or disclaimers.
If needed, include a brief basis in one concise sentence.

### Question:
{}

### Response:
{}"""

# Example usage
question = "접대비 한도는 어떻게 계산하나요?"
input_text = prompt_style.format(question, "")
inputs = tokenizer([input_text], return_tensors="pt")

# Filter unwanted outputs
bad_words = ["Disclaimer:", "Chain of Thought:", "### Chain of Thought:", "### Reasoning:", "### Analysis:"]
bad_words_ids = tokenizer(bad_words, add_special_tokens=False).input_ids

# Generate response
outputs = model.generate(
    input_ids=inputs.input_ids,
    attention_mask=inputs.attention_mask,
    max_new_tokens=1024,
    eos_token_id=tokenizer.eos_token_id,
    pad_token_id=tokenizer.pad_token_id,
    bad_words_ids=bad_words_ids,
    temperature=0.01,
    use_cache=True,
)

# Decode only generated part
gen_only = outputs[0, inputs.input_ids.shape[-1]:]
answer = tokenizer.decode(gen_only, skip_special_tokens=True).strip()
print(answer)

Training Details

Training Data

• Internal processed Korean tax-law QA subset
• Dataset format: legalqa_fino1_format.jsonl
• Focus: Korean tax regulations and Q/A pairs

Training Procedure (LoRA)

• Base model: google/gemma-3-4b-it
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Hyperparameters:
  • r=64
  • alpha=16
  • dropout=0.05
  • Learning rate: 2e-4
  • Batch size: 1 (with gradient accumulation)

Evaluation

• Qualitative evaluation on Korean tax-law prompts
• ROUGE scoring against reference answers
• Manual review for accuracy and relevance

Limitations

⚠️ Important Disclaimers:

• This model is for informational purposes only and does not constitute legal advice
• Always consult with qualified tax professionals for official guidance
• Outputs may be outdated or incomplete and require human validation
• Tax laws change frequently - verify current regulations

Model Performance

The model demonstrates good performance on Korean tax-related queries but should be used as a starting point for research rather than definitive answers.