Qwen3-14B-why

Qwen3-14B-Pentest-GRPO

A penetration-testing decision-planning model based on Qwen3-14B, trained with GRPO (Group Relative Policy Optimization) via the verl framework.

一个基于 Qwen3-14B 的渗透测试决策规划模型，使用 GRPO（Group Relative Policy Optimization） 强化学习方法、通过 verl 框架训练而成。

Given the current state of a penetration-testing engagement (target-machine info, the Penetration Testing Tree / PTT progress, and the previous step and its result), the model outputs a structured next-action decision: a high-level strategy, a concrete step, and the MCP tool tasks to invoke.

给定一次渗透测试过程中的当前状态（目标机器信息、Penetration Testing Tree(PTT) 进展、上一步动作及其结果），模型输出结构化的下一步决策：高层策略、具体步骤，以及需要调用的 MCP 工具任务。

⚠️ Safety & Compliance Notice / 安全与合规声明 EN: This model is intended for authorized security testing, red-teaming, CTF, and research only. Use it exclusively on systems for which you have explicit written authorization. Any unauthorized intrusion, attack, or misuse is illegal; the authors accept no liability. See Bias, Risks, and Limitations. 中文: 本模型面向授权的安全测试、红队演练、CTF 与安全研究场景。仅可在你拥有明确书面授权的系统上使用。任何未授权的入侵、攻击或滥用均属违法，作者不承担任何责任。详见 Bias, Risks, and Limitations。

---

Model Details / 模型详情

EN

• Model type: Causal decoder-only LLM (Qwen3 architecture), GRPO-finetuned
• Base model: Qwen/Qwen3-14B
• Parameters: ~14B, full-parameter fine-tuning (not LoRA)
• Language: English prompts and outputs
• License: Apache-2.0 (inherited from Qwen3-14B)
• Training framework: verl (verl.trainer.main_ppo, adv_estimator=grpo)
• Ability tag: pentest_planning

中文

• 模型类型: 因果解码器 LLM（Qwen3 架构），经 GRPO 强化学习微调
• 基座模型: Qwen/Qwen3-14B
• 参数量: 约 14B，全参数微调（非 LoRA）
• 语言: 英文提示词与输出
• 许可证: Apache-2.0（继承自 Qwen3-14B）
• 训练框架: verl（verl.trainer.main_ppo，adv_estimator=grpo）
• 能力标签: pentest_planning

Task / 任务

• EN — Input: engagement state snapshot (Machine / PTT / Previous strategy / Previous step / Previous step result). Output: a fixed-order XML structured decision.
• 中文 — 输入: 渗透测试引擎的状态快照（Machine / PTT / Previous strategy / Previous step / Previous step result）。输出: 固定顺序的 XML 结构化决策。

---

Prompt Format / 提示词格式

EN: The model relies on a fixed system prompt to constrain its structured output. At inference time, reuse the training system prompt, otherwise the reward-aligned structured output degrades.

中文: 模型使用固定的 system prompt 约束输出结构。推理时请复用训练时的 system prompt，否则奖励对齐的结构化输出会退化。

System prompt:

You are an expert penetration-testing planning agent. Given the current state of an
engagement against a target machine — including the Penetration Testing Tree (PTT) that
tracks completed and pending tasks, and the outcome of the previous step — decide the
single best next action.

Your response MUST strictly use the following XML tags in order:
<think>
[step-by-step reasoning: strategy analysis then step analysis]
</think>
<new_strategy>[a concise high-level strategy]</new_strategy>
<new_step>[the concrete next step to execute]</new_step>
<mcp_tasks>[a JSON object mapping tool -> task, e.g. {"Metasploit": "Exploit CVE-XXXX."}]</mcp_tasks>

User message (state template) / 用户消息（状态模板）:

## Target machine
{machine}

## Penetration Testing Tree (current progress)
{ptt}

## Previous strategy
{prev_strategy}

## Previous step
{prev_step}

## Previous step result
{prev_result}

Decide the best next action now.

Expected output (example) / 期望输出（示例）:

<think>
...strategy analysis...
...step analysis...
</think>
<new_strategy>Enumerate exposed web service for known CVEs</new_strategy>
<new_step>Run a targeted vulnerability scan against the web port identified in the PTT</new_step>
<mcp_tasks>{"Nmap": "Scan http service for version and known CVEs", "Metasploit": "Search matching exploit modules"}</mcp_tasks>

---

How to Get Started / 快速开始

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "path/to/Qwen3-14B-Pentest-GRPO"  # this model dir / 本模型目录
tok = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="auto", device_map="auto")

SYSTEM = (
    "You are an expert penetration-testing planning agent. ..."  # full system prompt above / 使用上文完整 system prompt
)
USER = (
    "## Target machine\n<machine>\n\n"
    "## Penetration Testing Tree (current progress)\n<ptt>\n\n"
    "## Previous strategy\n<prev_strategy>\n\n"
    "## Previous step\n<prev_step>\n\n"
    "## Previous step result\n<prev_result>\n\n"
    "Decide the best next action now."
)

messages = [{"role": "system", "content": SYSTEM}, {"role": "user", "content": USER}]
inputs = tok.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(inputs, max_new_tokens=1024, temperature=0.7, top_p=0.95)
print(tok.decode(out[0][inputs.shape[1]:], skip_special_tokens=True))

vLLM (recommended for deployment / 推荐部署方式)

python -m vllm.entrypoints.openai.api_server \
    --model path/to/Qwen3-14B-Pentest-GRPO \
    --max-model-len 12288

EN: Trained with max_prompt_length=6144, max_response_length=4096. The PTT context can be long, so set --max-model-len ≥ 10k when serving. 中文: 训练时 max_prompt_length=6144、max_response_length=4096。PTT 上下文可能很长，部署时 --max-model-len 建议 ≥ 10k。

---

Training Details / 训练细节

Training Data / 训练数据

EN: Sourced from a penetration-testing decision CSV where each row is one "decision point" in an engagement. Input columns: Machine, PTT, Previous strategy, Previous step, Previous step result. Reference-decision columns: New strategy, Strategy explanation, New step, Step explanation, MCP_tasks. Converted by convert_to_grpo.py into verl parquet schema (data_source / prompt / ability / reward_model / extra_info); the reference decision is packed as JSON into reward_model.ground_truth.

中文: 来源于渗透测试决策 CSV，每行是一次交战过程中的一个「决策点」。输入列：Machine, PTT, Previous strategy, Previous step, Previous step result；参考决策列：New strategy, Strategy explanation, New step, Step explanation, MCP_tasks。由 convert_to_grpo.py 转为 verl parquet schema（data_source / prompt / ability / reward_model / extra_info），参考决策打包为 JSON 存入 reward_model.ground_truth。

EN: Raw CSVs have ~27.7k / ~7.4k decision points; filter_overlong_prompts=True + truncation='error' filtered out samples exceeding 6144 tokens. GRPO needs no (prompt, response) supervised pairs — it samples multiple responses per prompt and estimates advantage from group-relative rewards. 中文: 原始 CSV 约 27.7k / 7.4k 条决策点；filter_overlong_prompts=True + truncation='error' 过滤掉超过 6144 token 的样本。GRPO 无需 (prompt, response) 监督对，它对同一 prompt 采样多个回答并用组内相对奖励估计优势。

Reward Function / 奖励函数

EN: Rule-based reward (offline, no external LLM), normalized to [0, 1] (see pentest_reward.py). 中文: 规则型奖励（离线、无需外部 LLM），归一化到 [0, 1]（见 pentest_reward.py）。

EN/中文: Can be replaced with an LLM-as-judge for stronger semantic scoring. / 可替换为 LLM-as-judge 以获得更强的语义判分。

Training Procedure / 训练过程

EN: GRPO (adv_estimator=grpo, no critic) on verl + FSDP (full-param) + vLLM async rollout. Rollouts per prompt rollout.n=8; KL regularization use_kl_loss=True, kl_loss_coef=0.001, kl_loss_type=low_var_kl.

中文: GRPO（adv_estimator=grpo，无 critic），基于 verl + FSDP（全参数）+ vLLM 异步 rollout。每 prompt 采样 rollout.n=8；KL 正则 use_kl_loss=True、kl_loss_coef=0.001、kl_loss_type=low_var_kl。

Hyperparameters / 超参数

Compute / 算力

---

Evaluation / 评估

EN: Evaluated on the validation set (268 samples) using the custom reward reward/mean@4 (mean over 4 samples per prompt). 中文: 在验证集（268 样本）上以自定义奖励 reward/mean@4（每 prompt 4 次采样均值）评估。

EN — Summary: From a starting point of ≈0.353, validation reward rose to ≈0.426 (~+20% relative gain) and plateaued around step ~1500. Training stayed stable throughout: train/inference probability agreement rollout_actor_probs_pearson_corr ≈ 0.999, pg_clipfrac ≈ 1e-3, no divergence.

中文 — 结论: 相较起点（≈0.353），验证奖励提升到 ≈0.426（约 +20% 相对提升），并在训练中段（~step 1500）趋于饱和。训练全程稳定：训练/推理概率一致性 rollout_actor_probs_pearson_corr ≈ 0.999，pg_clipfrac ≈ 1e-3，无发散。

EN: This metric is a rule-based reward (format compliance + tool-name hit + strategy word overlap), NOT real-world attack success on live targets. Real efficacy must be evaluated separately in a controlled lab. 中文: 该指标由规则型奖励度量（结构合规 + 工具命中 + 策略词重叠），并非真实靶机上的攻击成功率。真实效果需在受控靶场中另行评估。

---

Bias, Risks, and Limitations / 偏差、风险与局限

EN

• ⚠️ Dual-use / security risk: The model generates offensive-security actions and tool-invocation suggestions. Use it only in explicitly authorized environments for lawful security testing, red-teaming, CTF, and research. Never use it on unauthorized systems. Users bear full compliance and legal responsibility.
• Reward-proxy bias: The training reward is a rule-based proxy (tag completeness / tool-name hit / strategy word overlap), which may encourage "gaming the format and keywords" rather than truly optimal tactical decisions (reward-hacking risk).
• No real-world execution validation: Outputs are not validated for success on live targets; tools and CVEs in <mcp_tasks> may be inaccurate or outdated and must be reviewed by a human expert before execution.
• Data limitations: Training data comes from a specific pentest decision corpus with limited coverage of machine types, vulnerabilities, and tools; out-of-distribution generalization is unknown.
• Hallucination: May fabricate non-existent CVEs, tools, or paths.
• Language: Optimized for English prompts only.

中文

• ⚠️ 双重用途 / 安全风险: 模型会生成攻击性安全动作与工具调用建议。仅限在获得明确授权的环境中用于合法的安全测试、红队、CTF 与研究，禁止用于任何未授权系统。使用者须自行承担合规与法律责任。
• 奖励代理偏差: 训练奖励是规则型代理指标（标签齐全度 / 工具名命中 / 策略词重叠），可能激励模型"迎合格式与关键词"而非产出真正最优的战术决策（reward hacking 风险）。
• 无真实执行验证: 模型输出未在真实靶机上验证成功率；<mcp_tasks> 中的工具与 CVE 可能不准确或过时，必须由人工安全专家审核后再执行。
• 数据局限: 训练数据来自特定渗透测试决策语料，覆盖的机器类型、漏洞与工具有限；对分布外目标的泛化能力未知。
• 幻觉: 可能编造不存在的 CVE、工具或路径。
• 语言: 仅针对英文提示优化。

Recommendations / 使用建议

• EN: Always have a qualified security professional review every output before execution; use only in isolated/authorized environments; reuse the training system prompt for structured output; use an LLM-as-judge or real lab regression for more reliable evaluation.
• 中文: 始终由具备资质的安全人员人工复核每一步输出后再执行；仅在隔离/授权的实验环境中使用；推理时复用训练 system prompt以保持结构化输出；如需更可靠评估，替换为 LLM-as-judge 或真实靶场回归测试。

---

Framework / Reproduction / 框架与复现

• Base / 基座: Qwen/Qwen3-14B
• RL: verl main_ppo with adv_estimator=grpo
• Rollout: vLLM (FLASH_ATTN backend)
• EN: Training script, data conversion, reward function, and end-to-end reproduction steps are in the project's run_grpo_qwen3_14b.sh, convert_to_grpo.py, pentest_reward.py, GRPO_TRAIN_README.md.
• 中文: 训练脚本、数据转换、奖励函数与端到端复现步骤见项目内 run_grpo_qwen3_14b.sh、convert_to_grpo.py、pentest_reward.py、GRPO_TRAIN_README.md。

Software / 软件版本

---

Citation / 引用

EN/中文: Please also cite Qwen3, verl, and GRPO. / 请同时引用 Qwen3、verl 与 GRPO。

@misc{qwen3,
  title  = {Qwen3 Technical Report},
  author = {Qwen Team},
  year   = {2025}
}

@article{shao2024deepseekmath,
  title   = {DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models},
  author  = {Shao, Zhihong and others},
  journal = {arXiv preprint arXiv:2402.03300},
  year    = {2024},
  note    = {Introduces GRPO}
}

@misc{verl,
  title  = {verl: Volcano Engine Reinforcement Learning for LLMs},
  author = {ByteDance Seed and verl Community},
  url    = {https://github.com/volcengine/verl}
}