pharma-tinyllama-instruct-lora

Pharma TinyLlama — Instruction LoRA Adapter (Stage 2)

This is the Stage 2 instruction-tuning LoRA adapter for TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T.

Training pipeline

This adapter was trained on top of the Stage 1 domain-adapted merged model (ThakrePranjal/pharma-tinyllama-instruct-merged), following a 2-stage pipeline:

TinyLlama (base)
    → Stage 1: Domain Adaptive Pretraining  [ThakrePranjal/pharma-tinyllama-domain-lora]
    → Stage 1 Merged Model
    → Stage 2: Instruction Fine-Tuning      [THIS ADAPTER]

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_name = "TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T"
adapter_repo    = "ThakrePranjal/pharma-tinyllama-instruct-lora"

base_model = AutoModelForCausalLM.from_pretrained(base_model_name)
model      = PeftModel.from_pretrained(base_model, adapter_repo)
tokenizer  = AutoTokenizer.from_pretrained(adapter_repo)
model.eval()

Inference (Alpaca-style prompt)

import torch

def generate(instruction, input_text="", max_new_tokens=150):
    if input_text.strip():
        prompt = (
            f"### Instruction:\n{instruction}\n\n"
            f"### Input:\n{input_text}\n\n"
            f"### Response:\n"
        )
    else:
        prompt = (
            f"### Instruction:\n{instruction}\n\n"
            f"### Response:\n"
        )
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        out = model.generate(
            **inputs,
            max_new_tokens=max_new_tokens,
            do_sample=True,
            temperature=0.7,
            top_p=0.9,
            repetition_penalty=1.1,
            pad_token_id=tokenizer.eos_token_id,
        )
    return tokenizer.decode(out[0], skip_special_tokens=True)

print(generate("Explain the primary mechanism of action of metformin."))

Dataset

ThakrePranjal/pharma-instruction-dataset

Limitations

Trained on a small pharma corpus. Not validated for clinical or production use. Outputs must be reviewed against authoritative sources.