finance-specialist-v7

Finance Specialist v7

A fine-tuned Llama 3.2 1B Instruct model specialized for finance conversations, trained with knowledge-preserving LoRA techniques using llm-forge.

Model Details

Key Design: Zero Catastrophic Forgetting

This model was carefully tuned to add finance conversational ability without destroying the base model's general knowledge. Previous versions (v1-v6) suffered from catastrophic forgetting. v7 fixes this with:

1. LoRA r=8 (minimal weight perturbation)
2. Attention-only targets (q/k/v/o_proj) — MLP reasoning layers untouched
3. Learning rate 1e-5 (5x lower than v6)
4. Data cleaning (removed 72% of noisy/duplicate training samples)
5. No NEFTune noise (amplified forgetting on small datasets)
6. Single epoch (no overfitting)

Benchmark Results

General Knowledge Preservation (v7 vs Base)

Finance Domain (v7 vs Base)

Comparison with v6 (which had catastrophic forgetting)

Usage

With Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "Venkat9990/finance-specialist-v7",
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained("Venkat9990/finance-specialist-v7")

messages = [
    {"role": "system", "content": "You are a finance specialist AI assistant."},
    {"role": "user", "content": "What is a bond yield curve inversion?"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(text, return_tensors="pt").to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256, temperature=0.1, top_p=0.9)
print(tokenizer.decode(output[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

With Ollama (GGUF)

# Download GGUF and Modelfile from this repo, then:
ollama create finance-specialist-v7 -f Modelfile
ollama run finance-specialist-v7

Training Configuration

model:
  name: unsloth/Llama-3.2-1B-Instruct
  max_seq_length: 2048
  torch_dtype: bf16

lora:
  r: 8
  alpha: 16
  target_modules: [q_proj, v_proj, k_proj, o_proj]
  use_rslora: false

training:
  mode: lora
  learning_rate: 1.0e-5
  num_epochs: 1
  per_device_train_batch_size: 2
  gradient_accumulation_steps: 8
  gradient_checkpointing: true
  assistant_only_loss: true
  completion_only_loss: true
  neftune_noise_alpha: null
  label_smoothing_factor: 0.0

data:
  train_path: Josephgflowers/Finance-Instruct-500k
  format: sharegpt
  max_samples: 20000
  cleaning:
    enabled: true
    quality_preset: permissive
    dedup_enabled: true

Training Metrics

• Train loss: 2.16 → 0.72 (avg 1.569)
• Eval loss: 1.326
• Token accuracy: 67.8% (eval)
• Masked tokens: 97.7%
• Hardware: 1x NVIDIA A100 80GB (Hopper HPC)

Built With

llm-forge — Config-driven, YAML-first open-source LLM training platform.

Author

Naga Venkata Sai Chennu (@Venkat9990) — George Mason University