endpoint-qwen3.5-4b-lora-v2

Clinical Trial Endpoint Classifier — 4B v2 (Qwen3.5-4B LoRA)

v2 update of the endpoint-qwen3.5-4b-lora model. Trained on 2x more data with broader source coverage spanning ClinicalTrials.gov, EU Clinical Trials Register, and Chinese Clinical Trial Registry (ChiCTR).

A fine-tuned LoRA adapter on Qwen3.5-4B for extracting and classifying clinical trial endpoints from outcome text. Returns structured JSON with standardized endpoint names, measurement types, methods, and more.

⚠️ Loading fix (this revision)

The earlier snapshot of this adapter was saved with Unsloth's multimodal-wrapper layer paths (base_model.model.model.language_model.layers.X.…). Vanilla Qwen/Qwen3.5-4B is text-only and stores its decoder layers at model.layers.X (no language_model nesting), so when loaded with vanilla peft / transformers, PeftModel.from_pretrained logged a Found missing adapter keys warning and silently fell back to a default-initialized LoRA — the base instruction-tuned model would still produce JSON that looked schema-conformant (the system prompt is detailed enough), so the regression was easy to miss in spot checks. Loading via unsloth.FastLanguageModel.from_pretrained was unaffected.

This revision rewrites adapter_model.safetensors and adapter_config.json so PeftModel.from_pretrained(base, "Shubh-0789/endpoint-qwen3.5-4b-lora-v2") loads with 0 warnings and applies the trained weights to all the layers it was actually fine-tuned on. The training data, hyperparameters and behaviour are unchanged — see Quick start (fixed adapter) below for the up-to-date load snippet, and What changed in this release further down for technical detail.

Quick start (fixed adapter)

The PEFT path now works out of the box:

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3.5-4B",
    dtype=torch.bfloat16,
    device_map="auto",
)
tok = AutoTokenizer.from_pretrained("Qwen/Qwen3.5-4B")
model = PeftModel.from_pretrained(base, "Shubh-0789/endpoint-qwen3.5-4b-lora-v2")
model.eval()

clinical_text = "Primary endpoints are ORR and progression-free survival (PFS) assessed by RECIST v1.1 | [Time Frame: Up to 24 months]"

# Use the full training-time system prompt for best quality (10-field schema +
# measurement_method/evaluation_criteria taxonomy rules + few-shot examples).
# Minimal prompt also works:
messages = [
    {"role": "user", "content":
        f"Extract and classify the clinical trial endpoint from the following text. "
        f"Return ONLY a JSON.\nText: {clinical_text}"},
]
prompt = tok.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True,
    enable_thinking=False,  # Qwen3.5 thinking mode is OFF for direct JSON
)
inputs = tok(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(**inputs, max_new_tokens=512, do_sample=False)
print(tok.decode(out[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True))

The unsloth.FastLanguageModel path (next section) continues to work as before and is generally faster for single-request inference.

Verifying the LoRA is actually applied

If you're integrating this into a pipeline, a quick health check:

import torch
# After loading `model` as above, lora_B for at least one trained layer
# should be non-zero. (Default-init lora_B is zero, so any non-zero value
# means the trained adapter is in effect.)
key = "base_model.model.model.layers.3.self_attn.q_proj.lora_B.default.weight"
assert dict(model.named_parameters())[key].detach().abs().max() > 0

What changed in this release

Symptom you may have seen: the adapter loaded "successfully" via PeftModel.from_pretrained(base, "Shubh-0789/endpoint-qwen3.5-4b-lora-v2") but printed UserWarning: Found missing adapter keys while loading the checkpoint: ['base_model.model.model.layers.0.mlp.gate_proj.lora_A.default.weight', …], and downstream extractions were noticeably less consistent than the README examples — because the LoRA effectively wasn't being applied. Loading via unsloth.FastLanguageModel.from_pretrained was unaffected (Unsloth re-applies its own path translations on the fly).

Root cause. Unsloth's training pipeline saves LoRA weights under a multimodal-wrapper namespace, with keys like base_model.model.model.language_model.layers.0.mlp.gate_proj.lora_A.weight. Vanilla Qwen/Qwen3.5-4B is the text-only Qwen3_5ForCausalLM, which has its decoder layers at model.layers.X (no language_model nesting). When peft >= 0.18 looked for the saved lora_A.weight keys at the model's actual layer paths, none matched — so it injected default-initialized LoRA modules (lora_B = 0, contributing nothing) and emitted the warning.

Fix in this revision (no behavior change for inference):

• adapter_model.safetensors: stripped .language_model. from every layer path. The trained tensor values are unchanged — every saved tensor at model.language_model.layers.X.<module> now lives at model.layers.X.<module>, which is exactly where Qwen3.5-4B keeps that decoder layer.
• adapter_config.json: cleared the auto_mapping field that referenced an Unsloth-internal Qwen3_5ForConditionalGeneration class (which doesn't exist in vanilla transformers).

After the fix, PeftModel.from_pretrained(...) loads with no warnings and all 256 trained tensors match the file bit-for-bit (verified with safetensors.torch.load_file + torch.allclose).

The saved adapter still reflects Unsloth's training-time module pattern: every layer (0-31) carries MLP LoRAs (gate_proj, up_proj, down_proj), and every 4th layer (3, 7, 11, 15, 19, 23, 27, 31) additionally carries self-attention LoRAs (q_proj, k_proj, v_proj, o_proj). PEFT handles this layout via the standard list-form target_modules — no further config changes needed.

Verified with: transformers==5.7.0, peft==0.19.1, torch==2.8.0+cu128 (CUDA 12.8, RTX 5090).

If you were pinning the previous snapshot, the old commit hash is still reachable via revision= — but its loading path was broken on peft >= 0.18 (when not going through Unsloth), so we recommend updating to this snapshot and re-running any cached extractions you've already produced.

What's New in v2

• 2x training data: 3,906 samples (vs 1,948 in v1)
• Multi-source diversity: EU CTR (700) + ChiCTR (700) + ClinicalTrials.gov (600) added on top of v1's CTgov data
• 12 disease categories in v2 CTgov sample: diabetes, breast cancer, cardiovascular, alzheimer's, asthma, depression, hepatitis, rheumatoid arthritis, chronic kidney disease, multiple sclerosis, obesity, parkinson's
• Better generalization to non-US trial registries (EU + China)
• Improved labeling: v2 samples labeled by GPT-OSS-120B (vs v1 by Qwen3.6-plus)

Output Format

{
  "endpoints": [
    {
      "endpoint_name_standardized": "Objective Response Rate",
      "measurement_of": "tumor response",
      "measurement_type": "binary",
      "metric_type": "proportion",
      "timeframe": "Week 24",
      "measurement_method": "RECIST v1.1",
      "evaluation_criteria": "CR or PR",
      "unit": "%",
      "population": null,
      "is_composite": false,
      "components": []
    }
  ]
}

Field Definitions

Supports multiple endpoints from a single text (e.g., safety texts with 10+ sub-endpoints).

Training Details

Data Composition

Hyperparameters

Method: LoRA (bf16, NOT 4-bit)
LoRA rank: 16, alpha: 16, dropout: 0
Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Learning rate: 2e-4 (cosine scheduler)
Batch size: 2 per device (gradient accumulation 8, effective 16)
Epochs: 3
Optimizer: adamw_8bit
Sequence length: 2048
Gradient checkpointing: unsloth
Warmup steps: 10
Weight decay: 0.01
Max grad norm: 1.0
Seed: 3407

Usage

With Unsloth (Fastest)

import json
from unsloth import FastLanguageModel
from transformers import AutoTokenizer
import torch

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="Shubh-0789/endpoint-qwen3.5-4b-lora-v2",
    max_seq_length=2048,
    load_in_4bit=False,
    load_in_16bit=True,
    dtype=torch.bfloat16,
)
text_tokenizer = AutoTokenizer.from_pretrained("Shubh-0789/endpoint-qwen3.5-4b-lora-v2")
FastLanguageModel.for_inference(model)
model.generation_config.pad_token_id = text_tokenizer.pad_token_id

clinical_text = "Primary endpoints are ORR and progression-free survival (PFS) assessed by RECIST v1.1 | [Time Frame: Up to 24 months]"

messages = [
    {"role": "user", "content": f"Extract and classify the clinical trial endpoint from the following text. Return ONLY a JSON.\nText: {clinical_text}"}
]

inputs = text_tokenizer.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True,
    return_tensors="pt", return_dict=True,
).to(model.device)

with torch.no_grad():
    outputs = model.generate(**inputs, max_new_tokens=512, temperature=0.1, do_sample=True)

result = text_tokenizer.decode(outputs[0][inputs["input_ids"].shape[1]:], skip_special_tokens=True)
endpoints = json.loads(result)
print(json.dumps(endpoints, indent=2))

Output:

{
  "endpoints": [
    {
      "endpoint_name_standardized": "Objective Response Rate",
      "measurement_of": "tumor response",
      "measurement_type": "binary",
      "metric_type": "proportion",
      "timeframe": "Up to 24 months",
      "measurement_method": "RECIST v1.1",
      "evaluation_criteria": null,
      "unit": "%",
      "population": null,
      "is_composite": false,
      "components": []
    },
    {
      "endpoint_name_standardized": "Progression-Free Survival",
      "measurement_of": "disease progression or death",
      "measurement_type": "time-to-event",
      "metric_type": "hazard ratio",
      "timeframe": "Up to 24 months",
      "measurement_method": "RECIST v1.1",
      "evaluation_criteria": null,
      "unit": null,
      "population": null,
      "is_composite": false,
      "components": []
    }
  ]
}

With PEFT/Transformers

See Quick start (fixed adapter) at the top — PeftModel.from_pretrained works directly on this revision, no Unsloth required.

Inference Tip: Disable Thinking

Qwen3.5 supports a thinking mode. For this task, disable thinking for direct JSON output (the model was trained without <think> blocks):

# When using the tokenizer's chat template:
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True,
                                 enable_thinking=False)

# When using vLLM:
# --reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'

Model Comparison

Limitations

• Trained primarily on English clinical trial text (ChiCTR data is also in English)
• Complex composite endpoints may need verification
• Minimum inference: any GPU with 10GB+ VRAM
• Best inference settings: temperature=0.1, do_sample=True, thinking disabled

Citation

@misc{endpoint-qwen3.5-4b-lora-v2,
  author = {Shubh-0789},
  title = {Clinical Trial Endpoint Classifier — 4B v2 (Qwen3.5-4B LoRA)},
  year = {2026},
  publisher = {Hugging Face},
  url = {https://huggingface.co/Shubh-0789/endpoint-qwen3.5-4b-lora-v2}
}