qwen2.5-1.5b-style-without-erasure-lora

Style Without Erasure LoRA for Qwen2.5-1.5B-Instruct

A LoRA adapter trained by self-distillation against a style-blind teacher: the frozen base model reading a cue-free prompt supervises the adapted model reading the cue-present version of the same prompt. It reduces how much the model's next-token distribution moves when a speaker is described differently, while the propositional content of the request is held byte-identical.

This is a research artifact accompanying Style Without Erasure: Measuring and Removing Speaker-Style Sensitivity in Language Models. It is not a product, and §"Out of scope" below is not boilerplate — please read it.

• Code, paper and full provenance: https://github.com/SamhitaK10/style-without-erasure
• Base model: Qwen/Qwen2.5-1.5B-Instruct (Apache-2.0)

Checkpoint provenance

This release uses a recovered retained backup of the dialense_lora checkpoint. Its configuration matches the paper run exactly: Qwen2.5-1.5B-Instruct, r=16, α=32, all seven Qwen attention/MLP projection targets, 392 tensors and 18,464,768 adapter parameters. The released files should match these hashes:

adapter_config.json       sha256 884371482b02bb30481ffd2c6406cff933976b0e2a9e42505a6b683f06a21159
adapter_model.safetensors sha256 d48899aab39a1d18060825b2384cafcf889b4110020ba246824c572e11d12d8c

No checksum was recorded at the moment of the original evaluation, so exact byte-for-byte identity with the evaluated checkpoint is not independently provable. The architecture and configuration match the reported run exactly.

---

What it does

The base model answers the same clinical question differently depending on how the person asking is described. Measured against a placebo contrast matched on exact token length and input-embedding distance, that sensitivity runs 3.6×–7.4× the matched control across five speaker-style dimensions — fluency, health literacy, confidence, emotional expressiveness, communication style.

This adapter is trained to produce, in the presence of a style cue, the distribution the base model would have produced without one. The teacher is the model itself, so no external labels and no judgment about how a system ought to adapt to a speaker enter the objective.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

BASE = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(BASE, dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(model, "SamhitaK10/qwen2.5-1.5b-style-without-erasure-lora")
model.eval()

msgs = [
    {"role": "system", "content": "You are a physician taking a patient history. Ask one follow-up question."},
    {"role": "user", "content": "The patient speaks limited, ungrammatical English as a second language. "
                                "Reported history: chest pressure for 3 days; radiates to the left arm; "
                                "father had a heart attack at 55."},
]
ids = tok.apply_chat_template(msgs, return_tensors="pt", add_generation_prompt=True).to(model.device)
print(tok.decode(model.generate(ids, max_new_tokens=48)[0][ids.shape[-1]:], skip_special_tokens=True))

The system prompt above is the one used throughout training and evaluation. Behaviour under other system prompts was not measured.

model.disable_adapter() restores base behaviour — which is also the point made under "Suppression, not erasure" below.

Training

Forward KL is mass-covering, so the teacher's entropy acts as a floor on the student's. That is the mechanism that separates this from an objective which merely makes the model confidently insensitive to everything — see the ablation below.

Evaluation

Held-out phrasings crossed with held-out scenarios; the model saw neither. Divergence is Jensen–Shannon in bits over a fixed 24-token reference continuation.

Rows marked raw output not preserved are reported in the paper but their raw experiment output was lost; the repository documents this openly in results/MISSING.md. They are listed here for completeness, not as verified claims.

Ablation worth knowing about. The same objective with hard one-hot targets, every other hyperparameter fixed, collapses the output distribution: entropy 0.278 bits, 68.8% of positions above p = .99, and the sensitivity metric flips sign between two learning rates while the damage reproduces. A model can satisfy a "reduced style sensitivity" metric by becoming insensitive to everything, and perplexity does not catch it. This adapter is the soft-target arm.

Out of scope

Not for clinical use. No patient, clinician, or care setting was involved at any point. The clinical framing is a testbed chosen because it supplies matched scenarios with verified content identity — it is not a claim about medicine. No evaluation of clinical accuracy, safety, or patient outcomes was performed.

The removal is not style-specific. After training, the matched placebo contrast also falls 75.4%. The model did not learn to disregard communication style specifically; it learned to disregard non-content framing in general, while retaining sensitivity to clinical content. Do not describe this adapter as "debiased".

Suppression, not erasure. A linear probe still recovers the style dimension from the residual stream after training, with no layer showing a reduction that survives both a cluster-corrected bootstrap interval and a cluster permutation test. The design can exclude drops larger than 0.107 AUC at 15 held-out units; it cannot show the representation is unchanged. Behaviourally the adapter does what was asked, but the information is still there, the effect is one disable_adapter() away from reverting, and robustness to further fine-tuning was not tested.

One model, one scale. Everything above is Qwen2.5-1.5B-Instruct. Nothing establishes that the method transfers to other families or sizes. The one attempt at a second scale — activation patching on the 0.5B model — did not replicate and is reported as a failed replication.

Stated style, not enacted style. The cues are third-person descriptors of how a speaker communicates, not speech by people who actually communicate that way. A model may respond differently to enacted dysfluency than to a description of it.

Next-token measurement. Divergence is measured over a fixed 24-token continuation. No human evaluation of generation quality was performed.

Training data

The training prompts combine author-written style descriptors with clinical content blocks derived from the 50 scenarios in DiaLense, whose clinical content is grounded in the MTS-Dialog corpus.

MTS-Dialog is published under CC BY 4.0 and its attribution requirement passes to work derived from it:

Ben Abacha A, Yim W, Fan Y, Lin T. An Empirical Study of Clinical Note Generation from Doctor-Patient Encounters. Proceedings of EACL 2023, pp. 2291–2302. https://aclanthology.org/2023.eacl-main.168

Scenario content follows a source note but is restructured into a fixed ten-fact schema with identifying specifics generalised; the scenarios are not verbatim copies. Forty of the fifty are traceable to specific MTS-Dialog records; ten predate the provenance table and have no recorded source. The upstream corpus states no de-identification procedure, so no de-identification claim is made here. Full chain: data/README.md.

No corpus text is redistributed in this repository — only adapter weights.

Licence

The adapter weights are released under Apache-2.0, matching the base model they are a derivative of. Using them requires the base model, which carries its own Apache-2.0 terms held by the Qwen Team.

The attribution obligation on the MTS-Dialog-derived training data (CC BY 4.0) is satisfied by the citation above and travels with any redistribution.

Citation

@misc{kondareddy_style_without_erasure,
  title  = {Style Without Erasure: Measuring and Removing Speaker-Style
            Sensitivity in Language Models},
  author = {Kondareddy, Samhita},
  note   = {Manuscript},
  url    = {https://github.com/SamhitaK10/style-without-erasure}
}