smollm3-instruct-discharge-sentences-sft

SmolLM3-Instruct Discharge Sentences SFT

A fine-tuned version of SmolLM3-3B for multi-label clinical sentence classification in hospital discharge summaries.

Model Description

This model classifies individual sentences from discharge summaries into categories of follow-up actions:

• instructions: Case-specific instructions for the patient
• appointment: Appointment-related followup
• medication: Medication-related followups
• lab: Lab-related followup
• procedure: Procedure-related followup
• imaging: Imaging-related followup
• other: Other helpful contextual information

Performance

Evaluated on 5,313 test samples:

Per-Category Performance

Comparison with Base SFT Model

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

# Load base model and adapter
base_model = AutoModelForCausalLM.from_pretrained(
    "HuggingFaceTB/SmolLM3-3B",
    torch_dtype="auto",
    device_map="auto",
)
model = PeftModel.from_pretrained(base_model, "chrisvoncsefalvay/smollm3-instruct-discharge-sentences-sft")
tokenizer = AutoTokenizer.from_pretrained("chrisvoncsefalvay/smollm3-instruct-discharge-sentences-sft")

# Prepare input
messages = [
    {"role": "system", "content": "You are a clinical action item classifier..."},
    {"role": "user", "content": "Classify this sentence:\n\nPlease take aspirin 81mg daily."}
]

input_text = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)

# Generate
outputs = model.generate(**inputs, max_new_tokens=64, temperature=0.1)
response = tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
print(response)
# {"categories": ["medication", "instructions"]}

Training Details

• Base Model: HuggingFaceTB/SmolLM3-3B (instruction-tuned)
• Method: LoRA (r=64, alpha=128, dropout=0.1)
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Dataset: chrisvoncsefalvay/smol-discharge-sentences-sft
• Training Samples: 25,782
• Epochs: 3
• Learning Rate: 5e-5
• Effective Batch Size: 16
• Precision: bf16

Advantages Over Base Model

Using the instruction-tuned SmolLM3-3B as a base provides:

1. Perfect JSON validity (100%) - built-in chat template ensures reliable structured output
2. Significantly higher recall (+35%) - better at identifying action items
3. No custom template needed - uses standard chat format

Citation

If you use this model, please cite:

@misc{smollm3-instruct-discharge-sft,
  author = {von Csefalvay, Chris},
  title = {SmolLM3-Instruct Discharge Sentences SFT},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/chrisvoncsefalvay/smollm3-instruct-discharge-sentences-sft}
}

License

Apache 2.0