cca-qwen2.5-3b-lora

Cross-Consistency Assessment — Qwen2.5-3B QLoRA adapter

A rank-32 LoRA adapter that decides whether two options from different facets of one scenario can coexist.

Given two values with what each requires of a scenario and what each provides to one, it answers a single question: could one real configuration contain both? The verdict is read from one token's logits — no generation.

Trained during an internship at the Institute for Systems Studies & Analyses (ISSA), Defence Research & Development Organisation (DRDO).

Code, corpus and the Java deployment: https://github.com/Onecombatboot/morphological-analyzer

---

Results

Measured on 1,191 held-out cells from 10 domains never seen in training, with the decision threshold fitted leave-one-domain-out.

Inter-annotator agreement from a blind re-labelling of 289 cells: 88.9%, Cohen's κ = 0.730. At its operating point the adapter reproduces the corpus judgements at a rate comparable to an independent human analyst.

Selective prediction

Intended to be used with a coverage threshold — deciding the cells it is confident about and referring the rest, which is the normal mode for decision support.

---

Usage

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel

BASE = "Qwen/Qwen2.5-3B"
ADAPTER = "mangiis/cca-qwen2.5-3b-lora"

tok = AutoTokenizer.from_pretrained(BASE)
model = AutoModelForCausalLM.from_pretrained(
    BASE,
    quantization_config=BitsAndBytesConfig(
        load_in_4bit=True, bnb_4bit_quant_type="nf4",
        bnb_4bit_compute_dtype=torch.bfloat16, bnb_4bit_use_double_quant=True),
    dtype=torch.bfloat16, device_map={"": 0})
model = PeftModel.from_pretrained(model, ADAPTER).eval()

SYSTEM = (
    "You are a defence and security analyst carrying out cross-consistency "
    "assessment for General Morphological Analysis. You are given two options "
    "from different facets of ONE scenario, with what each requires and "
    "provides. Decide whether a single real scenario could contain both. "
    "Answer N only when you can name the clause that makes it impossible or "
    "means it never occurs; ineffective or unwise is still Y.")
PREFIX = "VERDICT:"

# the verdict must be exactly one token past the prefix, or the logits are
# being read at the wrong position
y_id = tok(PREFIX + " Y", add_special_tokens=False)["input_ids"][-1]
n_id = tok(PREFIX + " N", add_special_tokens=False)["input_ids"][-1]


def score_one(pa, a, a_req, a_prov, pb, b, b_req, b_prov):
    """P(inconsistent) for one presentation order."""
    user = (f"Facet A — {pa}\n{a} | REQUIRES: {a_req} | PROVIDES: {a_prov}\n\n"
            f"Facet B — {pb}\n{b} | REQUIRES: {b_req} | PROVIDES: {b_prov}\n\n"
            "Can one scenario contain both?")
    msgs = [{"role": "system", "content": SYSTEM},
            {"role": "user", "content": user}]
    prompt = tok.apply_chat_template(
        msgs, tokenize=False, add_generation_prompt=True) + PREFIX
    enc = tok(prompt, return_tensors="pt").to(model.device)
    with torch.no_grad():
        lg = model(**enc, logits_to_keep=1).logits[0, -1, :]
    pair = torch.stack([lg[y_id], lg[n_id]]).float()
    return torch.softmax(pair, dim=-1)[1].item()


def score(pa, a, a_req, a_prov, pb, b, b_req, b_prov):
    """Both presentation orders averaged. Do not skip this -- see below."""
    return 0.5 * (score_one(pa, a, a_req, a_prov, pb, b, b_req, b_prov)
                  + score_one(pb, b, b_req, b_prov, pa, a, a_req, a_prov))


p = score("OPERATOR", "Private security contractor",
          "a commercial engagement", "personnel without sovereign powers",
          "LEGAL BASIS", "Territorial waters enforcement",
          "exercise of sovereign enforcement authority",
          "jurisdiction inside the territorial sea")
print(p)          # ~0.99 -- the contractor has no sovereign authority

---

Three things that matter

Supply the premises. This model judges from REQUIRES and PROVIDES. Given bare value names it scores at chance. An early formulation asked the model to generate its own characterisations before judging and failed for exactly this reason: under teacher forcing it learns to condition on reference premises, and at inference it conditions on its own generated text, ~250 tokens of drift away.

The prompt wording is not editable. The template above, including the em dash in Facet A —, is byte-identical to the training format. Drift degrades the model silently — verdicts stay fluent and plausible while the numbers stop meaning what they used to.

Average both presentation orders. Cross-consistency is symmetric and this is not an optional refinement. Dropping it collapses the selective-prediction curve to F1 0.000 at 50% coverage: ranking survives, but the confidence signal inverts, so the model becomes least reliable exactly where it is most certain.

---

Training

Two departures from a default QLoRA recipe were necessary, not optional:

• Verdict-token weighting. The verdict is one token in a target that also carries reason prose. Under uniform cross-entropy most of the gradient trains explanation rather than the decision being scored.
• Chunked loss over supervised positions only. The full logits tensor (batch × sequence × 151,936) exceeds the budget outright. Never materialising logits at masked positions is what permitted rank 32 at all.

Data. 96 morphological domains, 648 parameter pairs, 10,286 assessed cells, 1,419 exclusions — each carrying a typed reason code and two verbatim citations, re-verified against source text on every build. Domains span far beyond defence (organ transplant allocation, patent litigation, coastal erosion, microfinance) so the only consistently predictive feature is the structure of the conflict itself rather than any field's vocabulary.

Split by domain, never by cell. Cells within a box share characterisations, so a cell-level split leaks premises and inflates every figure.

---

Limitations

• Requires characterisations. Not usable from value names alone.
• Full-coverage F1 is 0.583. The 0.811 headline is at 50% coverage. Neither number is interpretable without the other.
• Remaining errors are world-knowledge errors. Strong where the conflict is visible in the supplied clauses; weaker where the answer depends on whether a combination occurs in practice. That is a limit of the base model's knowledge.
• English only, and the criterion is one of possibility, not desirability — a combination that is feasible but unwise is judged consistent by design.

Citation

@misc{cca-qwen-lora,
  title  = {Automated Cross-Consistency Assessment for General Morphological Analysis},
  author = {Shourya, Aditya},
  year   = {2026},
  note   = {QLoRA adapter on Qwen2.5-3B. ISSA, DRDO.}
}

License

MIT. The base model carries its own licence (Qwen).