HypoAgent-Mixtral-8x7B-Instruct-SFT

HypoAgent — Mixtral-8x7B-Instruct SFT (LoRA)

A LoRA adapter for mistralai/Mixtral-8x7B-Instruct-v0.1, fine-tuned to turn a description of a research study into a rigorous, prespecified hypothesis-testing and statistical-analysis plan. Given a study's objective, design, population, intervention/exposure, comparator, outcome and constraints, it produces a structured plan (estimand, hypotheses, primary method + rationale, assumptions, diagnostics, effect/uncertainty reporting, sample-size/power, missing-data handling, multiplicity, sensitivity analyses, decision rule, limitations).

• Paired dataset: AliKhajegiliM/HypoAgent (private)
• Code & pipeline: GitHub AliKhajegiliM/HypoAgent (private)
• Base model: mistralai/Mixtral-8x7B-Instruct-v0.1 (Apache-2.0)

Model details

Training

• Data: the curated HypoAgent SFT set (data/sft/curated/, ≤60k STUDY CONTEXT → PLAN examples), synthesized from structured facts drawn from ClinicalTrials.gov (66%), AEA RCT Registry (25%) and OSF Registrations (8%); grouped train/val/test split by study family (0 leakage). Task types: full-plan, method-selection, plan-completion, plan-critique, alternatives.
• Procedure: 5 epochs, 2,275 steps, per-device batch size 1, max LR 3e-4, bf16.
• Loss: train 1.06 → 0.42; eval 0.52 → 0.49 (best ≈ 0.491 at step 1820 / epoch 4; a slight rise by epoch 5 suggests mild overfitting — an epoch-4 checkpoint may generalize marginally better).

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = "mistralai/Mixtral-8x7B-Instruct-v0.1"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(model, "AliKhajegiliM/HypoAgent-Mixtral-8x7B-Instruct-SFT")

sys = ("You are an expert statistical study-design assistant. Produce rigorous, transparent, "
       "and reproducible hypothesis-testing plans.")
user = ("STUDY DESCRIPTION\nObjective: ...\nDesign: cluster_randomized\n"
        "Primary outcome: ...\n\nTask: Produce a rigorous, prespecified hypothesis-testing and "
        "statistical-analysis plan for this study.")
msgs = [{"role": "system", "content": sys}, {"role": "user", "content": user}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=1200, do_sample=False)
print(tok.decode(out[0, ids.shape[1]:], skip_special_tokens=True))

The adapter ships the Mixtral-Instruct chat_template.jinja; keep the system prompt above for in-distribution behaviour.

Intended use & limitations

Intended: a study-design drafting assistant for researchers/biostatisticians — generating, completing, critiquing, or offering alternatives to analysis plans across clinical, economics/ social-science, and behavioral domains.

Out of scope / cautions

• Not medical, clinical, or financial advice, and not a substitute for a qualified statistician. Outputs must be reviewed before use.
• Training targets are synthesized from structured registry facts, not authoritative SAPs; the model learns to produce rigorous structure, and can still hallucinate specifics (exact margins, effect sizes) — it is trained to flag these as "must be prespecified," but verify.
• Part of the training data (AEA/OSF) carries training_use_status=review_required (unclear content license); no verbatim source text is reproduced, but treat the artifact as research-use.
• Inherits Mixtral-8x7B-Instruct's biases and knowledge cutoff. Non-English performance untested.
• Held-out evaluation: a separate study-design-soundness test set lives in the GitHub repo (external_eval/); report scores against the lexical-baseline floor, not in isolation.

License

Research use. The base model is Apache-2.0; this adapter is a derivative distributed for research, and its training data has mixed provenance (ClinicalTrials.gov public domain + review-flagged AEA/OSF). Attribution: base model authors + the HypoAgent dataset (case_id/source ids therein).