NeuroQwen3-0.6B

NeuroQwen3-0.6B

NeuroQwen3-0.6B is a domain-adapted version of Qwen/Qwen3-0.6B, trained on 1.2 million PubMed articles related to neuroscience. The goal was not to overwrite the model's pre-trained linguistic or instruction-following behavior, but to bias its token distribution towards a cogntiive neuroscience corpora. It was trained at a very slow learning rate (5e-6) over a single epoch. We chose a single epoch and low learning rate to reduce overfitting and catastrophic forgetting while still exposing the model to broad in-domain language.

Quickstart

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "neurovlm/NeuroQwen3-0.6B"

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
if tokenizer.pad_token is None:
    tokenizer.pad_token = tokenizer.eos_token

dtype = torch.bfloat16 if torch.cuda.is_available() else torch.float32
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=dtype,
    device_map="auto",
)
model.config.pad_token_id = tokenizer.pad_token_id

prompt = (
    "What is the default mode network?"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

with torch.no_grad():
    out = model.generate(
        **inputs,
        max_new_tokens=120,
        do_sample=False,
        repetition_penalty=1.08,
        eos_token_id=tokenizer.eos_token_id,
        pad_token_id=tokenizer.pad_token_id,
    )

generated = out[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(generated, skip_special_tokens=True).strip())

Training

The model was initialized from Qwen/Qwen3-0.6B and trained for one epoch on a PubMed neuroimaging corpus. Each training example was formatted as:

<paper title>

<paper abstract>

Training settings:

• Trainer: TRL SFTTrainer
• Objective: full-text causal language modeling
• Max sequence length: 1024
• Packing: enabled, wrapped
• Epochs: 1
• Learning rate: 5e-6
• Scheduler: cosine
• Warmup ratio: 0.10
• Weight decay: 0.01
• Batch size: 8 per device
• Gradient accumulation: 2
• Precision: bfloat16 when available
• Seed: 12345

Intended Use

This model is meant for neuroscience text generation and as a decoder backbone inside NeuroVLM. It may be useful for generating or completing descriptions of functional neuroimaging concepts, brain networks, regions, and experimental findings.

Citation

If you use this model, please cite the base Qwen model and the NeuroVLM project.