Qwen2.5-1.5B-QA360

Model Card for QA360-Qwen2.5-1.5B

Qwen2.5-1.5B-Instruct fine-tuned with QLoRA / LoRA SFT to turn a single software requirement into a QA360 test-analysis JSON object: risk, automation fit, affected modules, and functional / negative / security / accessibility / API / regression tests.

This card describes the merged checkpoint (merge_and_unload of the LoRA adapter into the base weights). Publish the adapter repo separately if you also want a PEFT-only artifact.

Model Details

Model Description

QA360-Qwen2.5-1.5B is a domain-specialized instruction model for requirements-to-test-design. Given:

1. a fixed system instruction that defines the QA360 schema, and
2. a short software requirement (often prefixed with a module tag such as [Authentication]),

the model is trained to emit only a JSON object with these exact keys:

It is not a general chatbot and is not a substitute for a human QA lead, threat model, or accessibility audit. Outputs are draft test ideas for analysts to edit.

• Developed by: Shankar Subramanyam
• Model type: Causal decoder-only LLM (Qwen2 architecture), instruction-tuned then task-SFT'd
• Language(s): English (requirement text and JSON string values)
• License: Apache-2.0 (inherits from Qwen/Qwen2.5-1.5B-Instruct)
• Finetuned from: Qwen/Qwen2.5-1.5B-Instruct

Base model facts (unchanged by this SFT):

This fine-tune was trained at max_length=2048. Keep inference prompts + completions inside that budget for best schema fidelity. Long requirement specs should be summarized before calling the model.

Model Sources

• Base model: https://huggingface.co/Qwen/Qwen2.5-1.5B-Instruct
• Base paper: Qwen2 Technical Report
• Training code: Finetuning_of_Qwen_Qlora_with_merge.py (TRL SFTTrainer + PEFT LoRA + optional bitsandbytes 4-bit)
• Adapter dir (if published): qwen_qa360_qlora_adapter
• Merged dir (this card): qwen_qa360_qlora_merged_model

Uses

Direct Use

• Draft a 360° test analysis from a one-line or short-paragraph requirement
• Seed test-case writing in ALM / Xray / TestRail / Azure DevOps
• Suggest regression blast radius for a story or change request
• Produce a first-pass JSON payload for a QA agent or RAG pipeline

Intended operators: QA engineers, SDETs, business analysts, and agentic tools that already validate JSON.

Downstream Use

• Tool-calling / structured-output node inside a multi-agent SDLC stack
• Further SFT or DPO on a private requirements corpus
• Constrained decoding (outlines, xgrammar, lm-format-enforcer) against the JSON schema
• Distillation teacher for a smaller on-prem classifier + template system

Out-of-Scope Use

Do not use this model as:

• An automated sign-off for security, privacy, or accessibility compliance
• A source of executable test code, exploits, or attack payloads
• A general assistant, code generator, or policy engine
• An analyzer of non-software text (legal contracts, medical notes, etc.)
• A production API without JSON parse checks, schema validation, and human review

The training targets are English software requirements. Other languages and free-form chat will degrade.

Bias, Risks, and Limitations

Task limitations

• 1.5B is small. It will invent plausible-but-wrong module names, status codes, and SLA numbers.
• Risk labels are learned priors from the SFT corpus, not a calibrated risk model. Auth and payments examples in the data are often High; the model will over-index on that pattern.
• Arrays are often 6–8 items because that is how the dataset was written. Real stories may need 2 items or 20.
• Accessibility and security lists are generic templates (keyboard, ARIA, HTTPS, lockout…). They are not WCAG or OWASP audits.
• The model was trained to copy a single system prompt. Changing the instruction mid-flight reduces JSON validity.

Data limitations

• Private/synthetic qa360_sft.jsonl (~4.8k rows). Coverage is only as broad as the requirements someone labeled.
• No published inter-annotator agreement. Style of test wording is that of the corpus authors.
• Module names and API paths (POST /api/auth/login) reflect the labeling convention, not your system.

Safety

• Can emit security test ideas (brute force, injection, lockout). That is intended. It should not be used to generate working exploit code.
• Do not send production secrets, customer PII, or unpublished vulnerability details into a hosted endpoint.

Technical

• Merged weights are fp16/bf16, not the 4-bit training quant.
• First-token and JSON-close failures still happen. Always json.loads and retry or repair.

Recommendations

1. Validate output against the schema before storing it.
2. Keep the exact system prompt used in training.
3. Cap max_new_tokens at 1024 (3080 is unnecessary and invites rambling).
4. Use greedy or low-temperature decoding (do_sample=False or temperature=0.1) for JSON.
5. Human-review risk_level and security_tests on High-risk domains (auth, payments, PHI, admin).
6. Log prompt, raw completion, parse success, and reviewer edits if you want a v2 dataset.

How to Get Started

Transformers (merged model)

import json
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "shankarblr/qwen2.5-1.5b-qa360"
device = "cuda" if torch.cuda.is_available() else "cpu"

tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(
    repo,
    torch_dtype=torch.bfloat16 if device == "cuda" else torch.float32,
    device_map="auto" if device == "cuda" else None,
)

SYSTEM = (
    "Perform a QA360 analysis for the following software requirement. "
    "Return a structured analysis covering risk_level (High/Medium/Low), "
    "automation_candidate (true/false), affected_modules (array), "
    "functional_tests (array), negative_tests (array), security_tests (array), "
    "accessibility_tests (array), api_tests (array), and regression_scope (array). "
    "Return only valid JSON with these exact keys."
)

requirement = "[Authentication] User can login with email and password"

prompt = (
    f"<|im_start|>system\n{SYSTEM}<|im_end|>\n"
    f"<|im_start|>user\n{requirement}<|im_end|>\n"
    f"<|im_start|>assistant\n"
)

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
out = model.generate(
    **inputs,
    max_new_tokens=1024,
    do_sample=False,
    pad_token_id=tokenizer.eos_token_id,
)
text = tokenizer.decode(out[0], skip_special_tokens=False)
response = text.split("<|im_start|>assistant\n")[-1].replace("<|im_end|>", "").strip()

print(json.dumps(json.loads(response), indent=2))

Chat-template variant (preferred if the tokenizer still ships Qwen's template):

messages = [
    {"role": "system", "content": SYSTEM},
    {"role": "user", "content": requirement},
]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)

Load adapter instead of merged weights

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_id = "Qwen/Qwen2.5-1.5B-Instruct"
adapter_id = "shankarblr/qwen2.5-1.5b-qa360-adapter"

tokenizer = AutoTokenizer.from_pretrained(adapter_id)
base = AutoModelForCausalLM.from_pretrained(base_id, torch_dtype=torch.bfloat16, device_map="auto")
model = PeftModel.from_pretrained(base, adapter_id)

Expected shape

{
  "risk_level": "High",
  "automation_candidate": true,
  "affected_modules": ["Authentication", "Session Management"],
  "functional_tests": ["Verify user can login successfully with valid email and password"],
  "negative_tests": ["Verify error message displayed for incorrect password"],
  "security_tests": ["Verify HTTPS is enforced for login requests"],
  "accessibility_tests": ["Verify login form is navigable using keyboard only"],
  "api_tests": ["Verify POST /api/auth/login returns 200 with a token on valid credentials"],
  "regression_scope": ["Password reset and recovery flow"]
}

Training Details

Training Data

Private JSONL corpus qa360_sft.jsonl, one object per line:

{
  "instruction": "Perform a QA360 analysis for the following software requirement. Return a structured analysis covering risk_level (High/Medium/Low), automation_candidate (true/false), affected_modules (array), functional_tests (array), negative_tests (array), security_tests (array), accessibility_tests (array), api_tests (array), and regression_scope (array). Return only valid JSON with these exact keys.",
  "input": "[Authentication] User can login with email and password",
  "output": { "...QA360 object..." }
}

Rows were wrapped in Qwen ChatML before SFT:

<|im_start|>system
{instruction}<|im_end|>
<|im_start|>user
{input}<|im_end|>
<|im_start|>assistant
{json.dumps(output, indent=2)}<|im_end|>

Training Procedure

Supervised fine-tuning with TRL SFTTrainer and PEFT LoRA. On CUDA the base is loaded in QLoRA (bitsandbytes 4-bit NF4, double quant, bf16 compute), then the adapter is merged back into an fp16/bf16 copy of the base.

Preprocessing

• tokenizer.pad_token = tokenizer.eos_token (Qwen EOS 151645)
• padding_side = "right"
• dataset_text_field = "text"
• Truncation at 2048 tokens

Hyperparameters

Speeds, Sizes, Times

Evaluation

Testing Data, Factors & Metrics

Held-out slice: 479 ChatML examples from the same qa360_sft.jsonl distribution (not a separately authored benchmark).

Suggested metrics — compute on the eval set and paste numbers; do not invent them:

Qualitative checks used in the training script (not a benchmark):

• Admin can force password reset
• User I can upload a profile picture
• user can lock account after failed attempts

Results

eval_loss:          TBD   # from trainer.evaluate()
json_parse_rate:    TBD
schema_valid_rate:  TBD
risk_level_acc:     TBD

Summary

This is a small specialized SFT, not a frontier model. Expect usable drafts on requirements that look like the corpus (auth, profile, account lockout, CRUD-style stories) and more hallucination on novel domains (embedded, data platform, ML ops).

Environmental Impact

Unknown until the GPU run is logged. Estimate with the MLCO2 calculator using hardware, hours, and region, then add:

• Hardware Type:
• Hours used:
• Cloud Provider / Region:
• Carbon Emitted:

QLoRA on 1.5B for ~1k steps is a small training job relative to pretraining.

Technical Specifications

Model Architecture and Objective

• Architecture: Qwen2ForCausalLM
• Objective: causal LM SFT on ChatML strings (next-token prediction over the full formatted example)
• Post-train artifact: LoRA adapter merged into base weights with PeftModel.merge_and_unload()

Compute Infrastructure

• OS / Python: Linux, Python 3.12
• Key libraries: transformers, peft, trl, bitsandbytes, torch, datasets

Glossary

• QA360: Internal name for a nine-field, full-stack test-analysis schema (risk + automation + six test views + regression).
• QLoRA: 4-bit quantized base + low-rank adapters during training.
• Merged model: Adapter baked into the base; loads as a normal transformers checkpoint, no PEFT required at inference.

Model Card Authors

Shankar Subramanyam

---