xiq-1

xiq-1 (LFM2.5-2.6B Security Reviewer)

xiq-1 is an experimental LoRA fine-tune of ansulev/LFM2.5-2.6B-Uncensored trained on a filtered subset of the ansulev/claude-mythos-distilled-25k dataset. The model was trained using Unsloth for cybersecurity analysis, secure code review, and vulnerability mitigation planning.

Model Details

• Model Name: xiq-1
• Repository: xiq/xiq-1
• Base Model: ansulev/LFM2.5-2.6B-Uncensored (2.6B parameters)
• Adapter Type: LoRA (PEFT)
• Trained By: xiq
• Training Framework: Unsloth + Hugging Face TRL (SFTTrainer)

Training Configuration

• Dataset Size: 14,359 samples (13,641 train / 718 validation)
  • Focused on cybersecurity and advanced_coding categories.
• LoRA Parameters:
  • Rank ($r$): 16
  • Alpha ($\alpha$): 32
  • Dropout: 0.05
  • Target Modules: q_proj, k_proj, v_proj, out_proj, in_proj, down_proj, up_proj, gate_proj
  • Trainable Parameters: 1,703,936 (~0.063% of total model weights)
• Hyperparameters:
  • Optimizer: adamw_8bit
  • Learning Rate: 2e-4 (Linear schedule)
  • Batch Size: 2 per device (Gradient accumulation: 4; Effective batch size: 8)
  • Epochs: 2 (3,412 global steps)
  • Sequence Length: 4,096 tokens
  • Final Training Loss: 0.0516

Known Limitations & Observations

• Overfitting & Hallucination: In benchmark evaluations on specific code snippets (e.g., standard SQL injection, file uploads, command injection), the adapter exhibits template memorization and can hallucinate unrelated security topics rather than analyzing the supplied code.
• Artifact Leakage: Responses may occasionally output closing reasoning tags (</think>) inherited from distilled dataset preambles.
• Experimental Status: This checkpoint represents an initial training run. Further calibration (prompt loss masking, lower learning rates, and generalized evaluation) is advised before deploying for automated security triage.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base_model_name = "ansulev/LFM2.5-2.6B-Uncensored"
adapter_name = "xiq/xiq-1"

tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    torch_dtype=torch.bfloat16 if torch.cuda.is_bf16_supported() else torch.float16,
    device_map="auto"
)
model = PeftModel.from_pretrained(base_model, adapter_name)

messages = [
    {"role": "user", "content": "Review this function for potential vulnerabilities:\n\n```python\[email protected]('/user')\ndef get_user():\n    uid = request.args.get('id')\n    return db.query(f'SELECT * FROM users WHERE id = {uid}')\n```"}
]

inputs = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=512,
    temperature=0.2,
    top_p=0.95,
    repetition_penalty=1.1
)

response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print(response)