Qwen-3.5-9B-CVE-to-CWE

Qwen3.5-9B CVE-to-CWE Classifier (LoRA)

Given the free-text description of a published CVE, emits the single CWE identifier that best characterises the root-cause weakness, as one line of text (CWE-79, CWE-787, ...), over the 117-class label space of exploitintel/cve-cwe-consensus.

This is a LoRA adapter for Qwen/Qwen3.5-9B, trained with QLoRA (4-bit NF4 base, bf16 compute) via TRL SFT.

The headline number is one draw from a noisy measurement. This checkpoint scored macro-F1 0.473 on 300 held-out rows, but the identical configuration re-run later in the same search scored 0.395. Read How these values were chosen and Evaluation before quoting anything, and do not report 0.473 as the model's expected performance.

Model details

Trainable parameters: 320,864,256 across 200 modules, 3.2975% of the 9,730,678,000 parameters PEFT counted with the adapter attached. The adapter file is 1,283,518,408 bytes (400 tensors, lora_A + lora_B per module, all float32). Every tensor sits under base_model.model.model.language_model.

One Qwen 3.5 structural fact shapes the module list. Confirmed against the base model's config.json: the 32 layers follow a 3-linear / 1-full pattern (full_attention_interval: 4), so 24 layers are Gated DeltaNet linear attention and expose in_proj_qkv / in_proj_z / out_proj, while the 8 full-attention layers (3, 7, 11, ..., 31) are grouped-query attention with 16 heads over 4 KV heads and expose q_proj / k_proj / v_proj / o_proj. The adapter's tensor counts match exactly: 48 per linear-attention projection (24 layers x A/B), 16 per attention projection (8 layers x A/B), 64 per MLP projection (32 layers x A/B). There are no MoE experts in this base.

Intended use

Direct use. Map a CVE description to one CWE id, for triage and labelling pipelines that already consume CWE ids. The 117-class label space is the dataset's; ids outside it were never seen in training.

The model was trained on a specific prompt shape and that shape is part of the contract:

• System prompt (verbatim): "You are a CVE-to-CWE classifier. Given a CVE vulnerability description, identify the single root-cause CWE weakness class that best characterizes the flaw. Output exactly one CWE identifier (e.g., CWE-79) on a single line with no explanation or additional text."
• User turn: the instruction "Classify the following CVE description into exactly one CWE weakness class. Reply with the CWE ID only, for example CWE-79.", a blank line, then the CVE description inside a bare ``` fence. This is the exact string the evaluator rendered (instruction + "\n\n```\n" + description + "\n```").
• Applied through the tokenizer's chat template (chat_template.jinja, shipped in this repo) with add_generation_prompt=True and enable_thinking=False. Do not concatenate strings by hand.
• The output is one line, CWE-<n>. In evaluation all 300 generations were a single well-formed id; the first line of the output is the prediction.
• Decode greedily (do_sample=False) with a small budget; 64 new tokens is what the metric was scored with.

How to get started

import torch
from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoTokenizer

BASE = "Qwen/Qwen3.5-9B"
ADAPTER = "SASVAAI/Qwen-3.5-9B-CVE-to-CWE"

# NOTE: AutoModelForImageTextToText, not AutoModelForCausalLM. Qwen 3.5's
# CausalLM mapping raises AttributeError: 'Qwen3_5Config' object has no
# attribute 'vocab_size' on the transformers build used here; the
# image-text-to-text mapping resolves to Qwen3_5ForConditionalGeneration and
# loads the text model correctly. Training used the same class.
tokenizer = AutoTokenizer.from_pretrained(ADAPTER, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="auto", trust_remote_code=True
)
model = PeftModel.from_pretrained(model, ADAPTER)
model.eval()

# Verbatim from the training invocation. Do not paraphrase.
SYSTEM = (
    "You are a CVE-to-CWE classifier. Given a CVE vulnerability description, "
    "identify the single root-cause CWE weakness class that best characterizes "
    "the flaw. Output exactly one CWE identifier (e.g., CWE-79) on a single line "
    "with no explanation or additional text."
)
INSTRUCTION = (
    "Classify the following CVE description into exactly one CWE weakness class. "
    "Reply with the CWE ID only, for example CWE-79."
)
description = (
    "A stored cross-site scripting flaw in the FAQ page lets an attacker inject "
    "script that runs in other users' browsers."
)
messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": f"{INSTRUCTION}\n\n```\n{description}\n```"},
]
inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True,
).to(model.device)

out = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(out[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True).strip())
# -> CWE-79

The base model is ~19 GB in bfloat16; one 24 GB-class GPU is enough for inference with the adapter. Loading the base in 4-bit with bitsandbytes (load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16) reproduces the training-time numerics and needs about 7 GB.

Decoding matters. The metric was scored greedily with max_new_tokens=64 through the chat template with thinking disabled. No sampling setting was validated, and enabling thinking changes the prompt the model sees.

Training details

Data. 700 training rows and 300 validation rows built from exploitintel/cve-cwe-consensus at revision 606cef101302fc2e0f69fd298f0de28b20a2aacf by a deterministic script (autocatalyst.datagen.cve_cwe_consensus, seed 0). No model generated any training content.

Row selection, as recorded in the builder's manifest:

• Source rows are {"messages": [system, user, assistant]}; the user turn is the CVE description, the assistant turn the label.
• Rows whose label is not exactly one CWE-<n> id were dropped (about 9% of the dataset are comma-separated multi-label lists). Rows whose id is outside the dataset's 117-class set, or whose description is empty, were dropped. Duplicate descriptions within a split were collapsed to the first.
• Train: from the dataset's train split (50,074 raw rows, 44,337 kept after dropping 4,390 multi-label or malformed and 1,347 duplicate rows), 700 rows were sampled with one row guaranteed per class and the remainder weighted by the square root of class frequency, so the head classes do not crowd out the tail that macro-F1 scores. All 117 classes are present.
• Validation: from the dataset's own validation split (11,052 raw rows, 8,878 kept), 300 rows sampled at the natural class distribution. 68 classes appear; every one of them is in the training set; no description text is shared with the 700 training rows (0 overlaps measured).

The dataset's head is heavy: in the 300 validation rows CWE-79 appears 49 times, CWE-862 22, CWE-89 17, CWE-200 and CWE-22 13 each, and 68 classes share the rest.

An LLM (Claude Opus 4.6, claude-opus-4-6, via an internal inference gateway) proposed the hyperparameters the search tried. It generated no training content and computed no metric.

Method

The project allowed two methods for this run (bf16_lora, qlora) and the search tried both; see the trial table below.

No refinement stage ran; the published weights are the SFT adapter.

Final hyperparameters

Effective batch size: 64 (2 x 4 x 8). Optimizer steps: 55 (11 per epoch).

neftune_noise_alpha (0.0), use_liger_kernel, use_sample_packing and lora_init (default) were left at their no-op defaults. KD parameters are omitted deliberately: this is a qlora run, not a distillation run.

Provenance note: every value above was recovered from the platform database (runs, experiments, events tables for run 3) and cross-checked against the literal [TRAIN] command line recorded in the run log and against the shipped adapter_config.json.

How these values were chosen

These hyperparameters were selected by an automated search (autocatalyst.cli.run_autoresearch): an agent proposes one change at a time, runs train then eval, and keeps or discards on f1_macro (higher is better).

Run 3 ran 46 trials in 7 h 17 m (2026-09-08 20:29 to 2026-09-09 03:46); 35 scored and 11 errored. This checkpoint is trial 23, the run's best.

Every trial is comparable. The search's five "data strategies" all re-adapted the same fixed 700-row training file (the train_v1 ... train_v5 files below hold the same 700 rows) and every trial was scored on the same 300 validation rows, so the whole table is one comparison.

All trials: lora_alpha = 2 x r, cosine schedule, batch 2 per device, max_seq_len 2048.

What the search actually established.

• Rank dominates, up to a point. Holding QLoRA, 5 epochs, dropout 0.15, LR 2e-4, weight decay 0.01, LoRA+ 2.0: r 32 → 64 → 128 moved macro-F1 0.379137 → 0.411651 → 0.440878 (#9, #10, #14).
• LoRA+ helped. The one clean A/B at r=32 (#8 → #9) moved 0.358649 → 0.379137 by putting the B matrices at 2x the learning rate.
• More epochs hurt. 8 epochs lost to 5 every time it was tried at r=128 (#15, #20, #30, #35, #41), by 0.01 to 0.06.
• 4-bit versus bf16 base: no measurable difference. At r=16 the 4-bit base beat bf16 (#2 at 0.323589 against #1 and #3 at 0.271); at r=128 with weight decay 0.05 the three bf16 runs (#31, #36, #42: 0.404 to 0.429) sit inside the two 4-bit runs of the same configuration (#38 at 0.395, #23 at 0.473). Quantising the base cost nothing this search could detect.
• rsLoRA at r=128 diverges. All 11 rsLoRA trials at r=128 (#16, #18, #22, #25, #27, #28, #29, #37, #40, #44, #45) were aborted by the training watchdog at the third logging step with gradient norm above 10 (the configured limit; observed 12.3 in the first). The likely cause is the scaling rule: rsLoRA scales updates by alpha / sqrt(r) instead of alpha / r, which with alpha = 2r is 2 sqrt(r), about 22.6 at r=128 against 2.0 without it, an 11x larger effective update. The one rsLoRA trial that finished (#11, r=64) scored below its non-rsLoRA sibling (#10). The 11 aborted trials are why the search shows 46 trials but 35 scores.

What it did not establish: the winning margin. The search's decision "weight decay 0.01 → 0.05 improved 0.440878 → 0.473288" (#14 → #23) does not survive the re-runs. Identical configurations were run more than once because each "data strategy" restarted the inner loop on the same data:

Seeds were not pinned, so these differ only in initialisation and data order. The 0.078 gap between #23 and #38 is larger than the 0.032 "improvement" the search kept, and larger than most differences in the table. Read 0.473 as the high draw of a configuration whose expected score is somewhere in the low-to-mid 0.4s, and treat any two trials within about 0.05 of each other as tied.

Search space. Six knobs were varied (LORA_R, LORA_DROPOUT, LEARNING_RATE, EPOCHS, WEIGHT_DECAY, LORAPLUS_LR_RATIO) plus one GRAD_ACCUM probe (#13, #46), one warmup probe (#34), the bf16-vs-4-bit method switch, and the rsLoRA attempts. LR_SCHEDULER, MAX_SEQ_LEN, BATCH_SIZE, NEFTUNE_NOISE_ALPHA, USE_DORA, LORA_INIT, USE_LIGER_KERNEL and USE_SAMPLE_PACKING were never moved.

Observed training metrics (this checkpoint).

55 optimizer steps ran. The logged train loss fell from 1.2962 at step 10 (grad norm 1.06) to the run mean of 0.6052; the reported train loss is the mean over the run, not a converged value.

Evaluation

Protocol. All 300 validation rows, greedy decoding, max_new_tokens=64, prompts rendered through the chat template with enable_thinking=False. The project's classification evaluator takes the first line of the generation as the predicted label and compares it to the gold id. Generation ran through Hugging Face generate in batches of 4 (max_input_len 4096) after vLLM 0.19.1 refused the LoRA on this architecture and the evaluator fell back; the whole pass took 63 s on 8 GPUs. Every one of the 300 outputs was a single well-formed CWE-<n> id, so no prediction was lost to formatting, and the 64-token budget cut nothing (the longest output is one id).

Two macro-F1 numbers, one convention. The evaluator averages F1 over the union of gold and predicted classes, so the 13 classes the model predicted that never occur in the 300 gold rows each contribute an F1 of 0 and pull the macro average down from 0.5638 to 0.4733. Both are reported; the union convention is the one the search optimised and the one the published baselines below appear to use, but check before comparing.

Published comparison points. The dataset authors report, on their own evaluation of the same dataset (a much larger split than these 300 rows):

Those models trained on the full ~44K-row training split; this adapter saw 700 rows. The micro-F1 is in the same range; the macro-F1 is 0.04 to 0.12 lower, which is the long tail this adapter had one to twelve examples of per class to learn from. The evaluation sets also differ in size and composition, so read this as context, not a controlled comparison.

Sibling run on the same 700 / 300 split. A Gemma 4 E4B adapter trained by the same platform on this split reached accuracy 0.697 and union macro-F1 0.4587, essentially tied with this checkpoint given the spread above.

Baseline for comparison. Not measured. The untuned Qwen/Qwen3.5-9B was never scored on these 300 rows, so nothing here quantifies how much of the score the fine-tuning is responsible for. This is the most important gap in this card.

This is a validation split the search selected against. 35 trials were scored on these same 300 rows and the best was kept, so expect optimistic bias on top of the re-run spread already described. The rows are drawn from the dataset authors' own validation split, so they are unseen CVEs from the same period as training, not future CVEs.

The evaluation set is reproducible. predictions.jsonl in this repo holds every one of the 300 rows: instruction, description, prediction and gold. The builder's manifest (dataset revision, seed, drop counts, per-class counts) is summarised under Training details.

Limitations and bias

One number, wide error bars. The same configuration scored 0.473 and 0.395 in two runs. Anyone deploying this should re-evaluate on their own data rather than trust either figure.

No baseline, so no established gain. See Evaluation.

Head classes dominate what accuracy measures. CWE-79 alone is 16% of the validation rows. A model that got only the top ten classes right would post a respectable accuracy and a poor macro-F1; the two numbers here disagree by 0.24 for that reason.

Tail classes were barely trained. 117 classes over 700 rows means many classes had a single training example. Expect the model to fall back to a frequent neighbour (CWE-20 for input validation issues, CWE-200 for disclosure) when the description is ambiguous, and to emit ids it saw rarely with low reliability. It also produced 13 ids in evaluation that never occur in the 300 gold rows; some may be reasonable alternative labels, some are wrong.

Only one label. Real CVEs often carry two CWE ids (about 9% of the source dataset). The training data dropped those rows, so the model always commits to one.

Prompt shape is the contract. Change the system prompt, the instruction sentence, the code fence, or enable thinking, and you are evaluating a model nobody measured.

Domain narrowness. English CVE descriptions in NVD / CNA style. Advisories in other formats, other languages, source-code inputs and exploit write-ups are unmeasured.

Inherits all biases and limitations of the base model. This adapter changes 3.3% of the parameters and was not evaluated for safety or fairness. The base model's own card governs those properties.

Environmental impact

Covers this trial only. The full 46-trial search that selected it took 7 h 17 m on the same hardware.

Framework versions

• PEFT 0.18.1
• TRL: 1.0.0
• Transformers: 5.7.0.dev0 (git main)
• Pytorch: 2.5.1+cu121
• bitsandbytes: 0.49.2
• flash-attn: 2.8.3
• Python: 3.12

PEFT's version is the one recorded in adapter_config.json at save time; the rest are the pinned versions of the training environment. transformers is a git-main build: the qwen3_5 architecture is not in the stable PyPI release.

Citation

@misc{qwen35_cve_cwe_lora_2026,
  title  = {Qwen3.5-9B CVE-to-CWE Classifier LoRA},
  author = {Banerjee, Aaron and Anbuselvan, Pooja and Jodhpurkar, Om},
  year   = {2026},
  url    = {https://huggingface.co/SASVAAI/Qwen-3.5-9B-CVE-to-CWE}
}