mistral-instruct-finetuned-iee_v0-4bit

Mistral-7B-Instruct LoRA adapter: local prototype of an LLM QA step for patient chats

Author: Vitalii Filipov · Created: October 2023 · Status: research prototype, not for clinical use

What it is

A LoRA adapter for mistralai/Mistral-7B-Instruct-v0.1, trained on a 4-bit quantized base model (QLoRA-style).

It is a local, simplified prototype of a QA step from a production clinical NLP system for patient chats. In production, this QA logic runs as an agentic workflow on OpenAI, Anthropic and Google models through their APIs. The workflow checks two things:

1. how the system interpreted the patient's answer (the extracted entities and values);
2. the reply the system is about to send. It flags answers that are non-standard, too short or context-dependent.

The adapter tested whether an open-source 7B model, run locally, could take over part of this check.

Task: given a system question, the patient's answer and the system's interpretation, return a structured verdict: is the interpretation supported by the answer, and is a clarifying question needed.

Training

Evaluation

Not formally evaluated. This was an exploratory prototype used to compare a local open-source model with the API-based workflow.

The production QA workflow (API-based) is evaluated separately against doctor-verified reference data. In production it reduced final-output errors by about 25-30% compared with the system without the QA step; the effect varies with case mix.

Example (synthetic, for illustration only)

Input
  System question: How many times a day do you take metformin now?
  Patient answer:  same as before
  Interpretation:  {"medication": "metformin", "frequency": "2 times/day"}
 
Output
  {"verdict": "needs_clarification",
   "reason": "context-dependent answer; frequency is not stated in the answer and was inferred",
   "follow_up": "Could you tell me how many times a day you take metformin now?"}

How to load

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
 
base_id = "mistralai/Mistral-7B-Instruct-v0.1"
adapter_id = "philka-ua/mistral-instruct-finetuned-iee_v0-4bit"
 
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="fp4",
                         bnb_4bit_compute_dtype=torch.float32)
tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, quantization_config=bnb, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)

Limitations

• Research prototype; not validated for clinical decision-making.
• Trained for one narrow QA step, not as a general medical assistant.
• No formal evaluation of this adapter; see the Evaluation section above.