RedSage-Qwen3-8B-DPO

RedSage-Qwen3-8B-DPO

<div align="center"> <img src="https://img.shields.io/badge/Task-Cybersecurity-red" alt="Cybersecurity"> <img src="https://img.shields.io/badge/Stage-DPO_Alignment-blue" alt="DPO"> </div>

Model Summary

RedSage-Qwen3-8B-DPO is the final, aligned version of the RedSage cybersecurity LLM series developed by RISysLab. It represents the fourth and final stage of the RedSage training pipeline.

This model is fine-tuned from RedSage-Qwen3-8B-Ins using Direct Preference Optimization (DPO) on the AllenAI Tulu 3 Preference Mixture. This alignment stage significantly enhances the model's general reasoning capabilities and safety behaviors while maintaining the deep cybersecurity domain expertise acquired during previous stages.

• Developed by: RISysLab
• Repository: GitHub
• Base Model: RISys-Lab/RedSage-Qwen3-8B-Ins
• Paper: RedSage: A Cybersecurity Generalist LLM (arXiv)

Training Lineage

RedSage employs a multi-stage training pipeline. This model represents the output of Stage 4.

1. Stage 1: Continual Pre-Training (CPT) -> RedSage-Qwen3-8B-CFW
2. Stage 2: Targeted Pre-Training -> RedSage-Qwen3-8B-Base
3. Stage 3: Supervised Fine-Tuning (SFT) -> RedSage-Qwen3-8B-Ins
4. Stage 4: Direct Preference Optimization (DPO) -> RedSage-Qwen3-8B-DPO (Current Model)
   • Data: Tulu 3 Preference Mixture

Dataset: Preference Alignment

The model was aligned using the following high-quality preference dataset to ensure robust instruction following and general reasoning:

• Dataset: allenai/llama-3.1-tulu-3-8b-preference-mixture
• Description: A comprehensive collection of preference data used to align the Tulu 3 models, focusing on helpfulness, factuality, and safety.

Performance & Evaluation

RedSage-Qwen3-8B-DPO achieves the best balance between specialized domain knowledge and general capability among all RedSage variants.

1. RedSage-Bench (0-shot)

2. External Cybersecurity Benchmarks (0-shot)

3. OpenLLM Leaderboard (General Benchmark)

Usage

Use the standard chat template for inference.

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_id = "RISys-Lab/RedSage-Qwen3-8B-DPO"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    torch_dtype=torch.bfloat16, 
    device_map="auto"
)

# Define the chat messages
messages = [
    {"role": "system", "content": "You are RedSage, a helpful cybersecurity assistant."},
    {"role": "user", "content": "Analyze the following log entry for potential indicators of compromise: 'POST /cgi-bin/test-cgi?* HTTP/1.1'"}
]

# Apply chat template
text = tokenizer.apply_chat_template(
    messages, 
    tokenize=False, 
    add_generation_prompt=True
)

inputs = tokenizer(text, return_tensors="pt").to(model.device)

outputs = model.generate(**inputs, max_new_tokens=512)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Intended Use

• Primary Use: General-purpose cybersecurity assistance, log analysis, threat intelligence summarization, and educational queries.
• Benefits: Better instruction adherence based on human preference compared to the SFT-only version.
• Limitations: While aligned, the model may still produce incorrect information. Always verify outputs in critical security environments.

Citation

If you use this model or dataset, please cite our paper:

@inproceedings{suryanto2026redsage,
  title={RedSage: A Cybersecurity Generalist {LLM}},
  author={Naufal Suryanto and Muzammal Naseer and Pengfei Li and Syed Talal Wasim and Jinhui Yi and Juergen Gall and Paolo Ceravolo and Ernesto Damiani},
  booktitle={The Fourteenth International Conference on Learning Representations},
  year={2026},
  url={https://openreview.net/forum?id=W4FAenIrQ2}
}