RedSage-Qwen3-8B-Base

RedSage-Qwen3-8B-Base

<div align="center"> <img src="https://img.shields.io/badge/Task-Cybersecurity-red" alt="Cybersecurity"> <img src="https://img.shields.io/badge/Stage-Targeted_Pretraining-blue" alt="Targeted Pretraining"> </div>

Model Summary

RedSage-Qwen3-8B-Base is a cybersecurity-specialized Large Language Model (LLM) developed by RISys-Lab. It represents the second stage of the RedSage pre-training pipeline.

This model builds upon RedSage-Qwen3-8B-CFW by undergoing Targeted Pre-Training on high-quality, curated cybersecurity resources (RedSage-Seed and RedSage-Dump). While the previous stage focused on breadth using web data, this stage focuses on depth, technical standards, and verified skills.

• Paper: RedSage: A Cybersecurity Generalist LLM (arXiv)
• Repository: GitHub
• Base Model: RISys-Lab/RedSage-Qwen3-8B-CFW
• Variant: Base (Final Pre-trained Checkpoint)

Intended Use

This model is a base model intended for:

1. Fine-tuning: Serving as a high-quality foundation for downstream cybersecurity tasks (e.g., incident response, malware analysis).
2. Research: Investigating the impact of curated versus web-scale data in domain adaptation.
3. Completion: Code completion and technical writing in cybersecurity contexts.

Note: As a base model, this checkpoint has not been instruction-tuned (SFT) or aligned (DPO). It behaves like a completion engine. For a chat-ready assistant, please see RISys-Lab/RedSage-Qwen3-8B-DPO.

Training Lineage

RedSage employs a multi-stage training pipeline. This model represents the output of Stage 2.

1. Stage 1: Continual Pre-Training (CPT) -> RedSage-Qwen3-8B-CFW (CyberFineWeb data)
2. Stage 2: Targeted Pre-Training -> RedSage-Qwen3-8B-Base (Current Model)
   • Data: RedSage-Seed (~150M Tokens) + RedSage-Dump (~700M Tokens)
3. Stage 3: Supervised Fine-Tuning (SFT) -> RedSage-Qwen3-8B-Ins
4. Stage 4: Direct Preference Optimization (DPO) -> RedSage-Qwen3-8B-DPO

Training Data: RedSage-Seed & Dump

This model was trained on approximately 850 million tokens of curated data, split into two collections:

1. RedSage-Seed (~150M Tokens): A highly curated collection of 28,637 samples converted to structured Markdown.

   • Knowledge: General concepts and Frameworks (MITRE ATT&CK, CAPEC, CWE, OWASP).
   • Skills: Offensive security resources including write-ups, hacking techniques, and payload examples.
   • Tools: Manuals and cheat sheets for CLI tools and Kali Linux.

2. RedSage-Dump (~700M Tokens): A larger aggregation of 459K technical documents.

   • Sources: Computer education portals, cybersecurity news, RFC entries, NIST publications, and the National Vulnerability Database (NVD).

Performance

RedSage-8B-Base achieves state-of-the-art performance among 8B models, showing significant improvements over the general-purpose Qwen3-8B-Base. It achieves the highest mean score on external benchmarks among all 8B base models tested.

RedSage-Bench (0-shot Accuracy)

External Cybersecurity Benchmarks (5-shot)

Training Procedure

The model was trained using the Axolotl framework.

• Learning Rate: 2.5e-6 (constant with linear warmup)
• Optimizer: AdamW
• Epochs: 1

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "RISys-Lab/RedSage-Qwen3-8B-Base"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

text = "The primary difference between a firewall and an IDS is"
inputs = tokenizer(text, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model or dataset, please cite our paper:

@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}