RedSage-Qwen3-8B-CFW

RedSage-Qwen3-8B-CFW

<div align="center"> <img src="https://img.shields.io/badge/Task-Cybersecurity-red" alt="Cybersecurity"> <img src="https://img.shields.io/badge/Stage-Continual_Pretraining-blue" alt="CPT"> </div>

Model Summary

RedSage-Qwen3-8B-CFW is a cybersecurity-specialized Large Language Model (LLM) developed by [RISys-Lab]. It is the result of Continued Pre-training (CPT) on the CyberFineWeb corpus.

This model serves as the foundational stage of the RedSage pipeline. It takes the general-purpose Qwen3-8B-Base and adapts it to the cybersecurity domain using ~11.7 billion tokens of filtered, high-quality cybersecurity web data. To maintain general reasoning capabilities, it utilizes a data replay strategy with educational content.

• Paper: RedSage: A Cybersecurity Generalist LLM (arXiv)
• Repository: GitHub
• Base Model: Qwen/Qwen3-8B-Base
• Variant: CFW (CyberFineWeb Continued Pre-training)

Intended Use

This model is a base model intended for:

1. Further fine-tuning on downstream cybersecurity tasks.
2. Research into domain adaptation and continual pre-training dynamics.
3. Cybersecurity text completion and generation.

Note: As a base model, this checkpoint has not been instruction-tuned (SFT) or aligned (DPO). It behaves like a completion engine. For a chat-ready assistant, please see RISys-Lab/RedSage-Qwen3-8B-DPO.

Training Lineage

RedSage employs a multi-stage training pipeline. This model represents the output of Stage 1.

1. Stage 1: Continual Pre-Training (CPT) -> RedSage-Qwen3-8B-CFW (Current Model)
   • Data: CyberFineWeb (11.8B tokens)
2. Stage 2: Targeted Pre-Training -> RedSage-Qwen3-8B-Base
3. Stage 3: Supervised Fine-Tuning (SFT) -> RedSage-Qwen3-8B-Ins
4. Stage 4: Direct Preference Optimization (DPO) -> RedSage-Qwen3-8B-DPO

Training Data: CyberFineWeb

This model was trained on CyberFineWeb, a large-scale cybersecurity corpus constructed by filtering the FineWeb dataset (2013–2024).

1. Filtering: A ModernBERT-base classifier was trained on the Cybersecurity Topic Classification dataset to identify cybersecurity content within Common Crawl.
2. Dataset Size: The filtering process yielded ~125M documents (~89.8B tokens). We select the latest subset of ~11.7B tokens for this training stage.
3. General Knowledge Replay: To prevent catastrophic forgetting, we mixed the cybersecurity data with a 30% replay ratio of FineWeb-Edu samples.

Performance

RedSage-8B-CFW demonstrates improved performance over the general-purpose Qwen3-8B-Base on cybersecurity benchmarks while maintaining general capabilities.

RedSage-Bench (0-shot Accuracy)

External Cybersecurity Benchmarks (5-shot)

Training Procedure

The model was trained using the Axolotl framework

• Learning Rate: 2.5e-6 (constant with linear warmup)
• Optimizer: AdamW
• Epochs: 1

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "RISys-Lab/RedSage-Qwen3-8B-CFW"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

text = "The primary difference between a firewall and an IDS is"
inputs = tokenizer(text, return_tensors="pt").to("cuda")

outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Citation

If you use this model or dataset, please cite our paper:

@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}