Qwen3-4B-Instruct-2507-GRPO-CTI

CyberSentinel CTI — Qwen3-4B-Instruct-2507 (GRPO)

CyberSentinel CTI is a 4-billion-parameter language model for cyber threat intelligence (CTI) tasks. It is a fine-tune of Qwen/Qwen3-4B-Instruct-2507 trained with GRPO (Group Relative Policy Optimization) using Unsloth and TRL, with reward functions that shape both the format of the output and the correctness of the answer.

This repository ships the model in two forms:

ℹ️ Before publishing, replace your-username/cybersentinel-cti in the examples below with the actual Hugging Face repository id.

---

Model Details

• Developed by: Emul Sazib
• Model type: Decoder-only causal language model (Qwen3 architecture)
• Language: English
• License: Apache 2.0
• Finetuned from: Qwen/Qwen3-4B-Instruct-2507 (via the unsloth/Qwen3-4B-Instruct-2507 4-bit build)
• Fine-tuning method: GRPO reinforcement learning with LoRA adapters (rank 16), merged back to 16-bit
• Parameters: 4.05B total (LoRA trainable: 33.0M, ~0.81% of the model)
• Context length: up to 262,144 tokens (inherited from the base model)
• Prompt format: Qwen3 ChatML (<|im_start|>role … <|im_end|>)

Architecture (from config.json)

---

Intended Use

Direct use

CyberSentinel CTI is intended to assist defensive security and threat-intelligence workflows, for example:

• Summarizing and triaging threat-intelligence reports and security advisories
• Reasoning over indicators of compromise (IOCs), TTPs, and adversary behavior
• Producing structured answers for CTI analysis tasks
• Assisting SOC analysts, threat researchers, and blue teams with drafting and enrichment

Out-of-scope / prohibited use

This model is released for defensive and educational purposes. It must not be used to develop malware, plan or carry out attacks, target individuals or organizations, evade detection, or otherwise cause harm. As with any LLM, outputs can be incorrect, incomplete, or fabricated — treat everything it produces as a draft to be verified by a qualified human analyst before it informs any security decision.

---

How to Get Started

🤗 Transformers (full precision)

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "your-username/cybersentinel-cti"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")

messages = [
    {"role": "system", "content": "You are CyberSentinel, a cyber threat intelligence analyst."},
    {"role": "user", "content": "Summarize the TTPs described in this report and map them to MITRE ATT&CK: ..."},
]

inputs = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt"
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=512, temperature=0.7, top_p=0.8)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

🦙 llama.cpp (GGUF, Q4_K_M)

./llama-cli -m cybersentinel-cti-q4_k_m.gguf \
  -p "You are CyberSentinel, a cyber threat intelligence analyst." -cnv

🐳 Ollama (GGUF)

# Modelfile
FROM ./cybersentinel-cti-q4_k_m.gguf

ollama create cybersentinel-cti -f Modelfile
ollama run cybersentinel-cti

---

Training

Procedure

The model was trained with GRPO (introduced in DeepSeekMath) on top of the base Qwen3-4B-Instruct-2507. LoRA adapters were attached to the attention and MLP projections (q/k/v/o_proj, gate/up/down_proj), trained, then merged into the 16-bit base weights and additionally exported to GGUF Q4_K_M via llama.cpp.

Three reward functions shaped the policy:

• format_reward_func — binary (0/1) reward for producing the required output structure.
• correctness_reward_func — task reward in the range −1 … +2 for a correct answer.
• soft_format_reward_func — partial-credit reward in 0 … 0.5 for near-correct formatting.

Hyperparameters

Data

The training set contained 14,923 CTI prompts after filtering (13 over-long prompts dropped from 14,936), with additional validation (2,630) and test (3,170) splits. Each example pairs a CTI prompt with a reference answer used by the correctness reward.

Training signal

Over training the format reward climbed toward ~0.9 and the correctness reward lifted off its −1.0 floor, indicating the policy learned both the required output structure and improved task correctness. These are training rewards, not a held-out benchmark score — see Limitations.

Software / environment

• Unsloth 2026.8.19, TRL 0.24.0, Transformers 4.57.6, PyTorch 2.11.0, vLLM 0.23.0
• Hardware: 1× NVIDIA RTX 3090 (24 GB), CUDA 13.0
• Training runtime: ~6.5 hours

---

Bias, Risks, and Limitations

• No formal held-out evaluation is reported here. The evidence of learning is the training reward curve; downstream accuracy on your own CTI benchmark should be measured before relying on the model.
• The model can hallucinate IOCs, CVE identifiers, ATT&CK mappings, and other security facts. Verify all outputs against authoritative sources.
• Trained for ~0.54 epochs; it is a lightweight domain adaptation, not an exhaustive fine-tune.
• Inherits the biases and limitations of the base Qwen3-4B-Instruct-2507 model.
• English-centric; performance on other languages is not characterized.

Recommendation: Keep a qualified human in the loop. Do not use model output as the sole basis for any security or operational decision.

---

License

Released under the Apache License 2.0 (see LICENSE), consistent with the Apache-2.0 license of the base Qwen3-4B-Instruct-2507 model. You are free to use, modify, and redistribute the model, including commercially, provided you retain the license and attribution.

---

Citation

If you use this model, please cite it along with the base model and methods:

@misc{cybersentinel_cti_2026,
  title        = {CyberSentinel CTI: A Qwen3-4B Model for Cyber Threat Intelligence},
  author       = {Sazib, Emul},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/your-username/cybersentinel-cti}},
  note         = {Fine-tuned from Qwen3-4B-Instruct-2507 with GRPO}
}

Base model — Qwen3:

@misc{qwen3,
  title  = {Qwen3 Technical Report},
  author = {Qwen Team},
  year   = {2025}
}

Method — GRPO:

@article{shao2024deepseekmath,
  title   = {DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models},
  author  = {Zhihong Shao and Peiyi Wang and Qihao Zhu and Runxin Xu and Junxiao Song and Mingchuan Zhang and Y. K. Li and Y. Wu and Daya Guo},
  year    = {2024},
  eprint  = {arXiv:2402.03300}
}

Tooling — TRL & Unsloth:

@misc{vonwerra2022trl,
  title        = {TRL: Transformer Reinforcement Learning},
  author       = {Leandro von Werra and Younes Belkada and Lewis Tunstall and Edward Beeching and Tristan Thrush and Nathan Lambert and Shengyi Huang and Kashif Rasul and Quentin Gallou{\'e}dec},
  year         = {2020},
  howpublished = {\url{https://github.com/huggingface/trl}}
}

Trained with Unsloth for 2× faster fine-tuning.