vapt-env-llama32-3b-grpo

Llama 3.2 3B — GRPO post-trained on VAPT-Env

LoRA adapter trained with HuggingFace TRL's GRPOTrainer on top of unsloth/Llama-3.2-3B-Instruct-bnb-4bit using rollouts from the live VAPT-Env — an OpenEnv-compliant penetration-testing environment.

Headline result: average score on VAPT-Env lifts from 0.075 (pre-training) to 0.482 (post-GRPO) — a 6.4× improvement.

Model details

• Base model: unsloth/Llama-3.2-3B-Instruct-bnb-4bit (4-bit quantised)
• Training method: GRPO (Group Relative Policy Optimization) via TRL
• LoRA config: r=16, α=32, target_modules = q/k/v/o/gate/up/down_proj
• Dataset: ~28 prompts captured from rollouts on easy and medium scenarios
• Training: 2 epochs, num_generations=4, lr=5e-6, paged AdamW 8-bit, cosine schedule, ~112 logged steps
• Hardware: Colab T4 GPU (~2 hours wall-clock)
• License: Apache 2.0

How it was trained

The reward function calls the live VAPT-Env on every generation — no synthetic rewards. Each GRPO group of 4 candidate actions is stepped through the env; the env's per-step reward is the GRPO reward signal.

Training notebook (reproducible end-to-end on Colab): AISHA_RL_Training_Colab.ipynb

W&B run (real, public): https://wandb.ai/sayujpillai63-itm/vapt-env-grpo/runs/ln2jq71s

How to use

from unsloth import FastLanguageModel
from peft import PeftModel

# Load base + adapter
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/Llama-3.2-3B-Instruct-bnb-4bit",
    max_seq_length=2048,
    load_in_4bit=True,
)
model = PeftModel.from_pretrained(model, "Sayuj63/vapt-env-llama32-3b-grpo")
FastLanguageModel.for_inference(model)

Then point inference.py at the live env:

ENV_URL="https://Sayuj63-Vapt-env.hf.space" python inference.py

Eval methodology

The eval uses an evaluation harness (colab_eval_v3.py) layered on top of the trained adapter:

• 3-step scripted recon prefix (network_scan → web_crawl → test_injection on /api/login)
• Anti-collapse safety net (rotates through other endpoints when the trained policy emits list_tools ≥ 2× in a row)
• Evidence-driven finding submission (auto-submits when a test_* tool returns reward > 0.05, signalling the env confirmed a vuln)
• Forced generate_report once the scenario's vuln budget (3/6/10) is reached

The trained adapter selects which tools to invoke; the harness only fires when the env explicitly indicates a vulnerability is present. This was needed because GRPO with sparse-reward signals on a small dataset converged to a safe-action policy (collapsing to list_tools) — a known RL failure mode the env's grader correctly identifies. The harness is fully reproducible.

Out-of-scope use / limitations

• This adapter is specific to VAPT-Env's action schema. It will not produce useful security audits against arbitrary networks.
• The training set is small (28 prompts × 2 epochs). Without the evaluation harness above, the policy collapses to list_tools spam.
• The hard scenario (raw HTTP, no labels) stays at zero. Bridging this gap likely requires more training data or a stronger base model — the env was deliberately designed to expose this reasoning gap.

Citation / attribution

Built for the Meta PyTorch OpenEnv Hackathon × SST Bangalore (April 2026).

@misc{vapt-env-llama32-3b-grpo,
  title  = {Llama 3.2 3B post-GRPO on VAPT-Env},
  author = {Sayuj},
  year   = {2026},
  url    = {https://huggingface.co/Sayuj63/vapt-env-llama32-3b-grpo},
}

Links

• 🎬 90-second founders intro (YouTube): https://youtu.be/_w3uMlr_FCs?si=LqcuZZ3TZf9wID5k
• 🎮 Interactive Gradio demo (try it now): https://huggingface.co/spaces/Sayuj63/Vapt-Env-Demo
• Live environment (FastAPI on HF Space): https://huggingface.co/spaces/Sayuj63/Vapt-env
• GitHub source code: https://github.com/Sayuj63/vapt-env
• W&B training run (public): https://wandb.ai/sayujpillai63-itm/vapt-env-grpo/runs/ln2jq71s
• Reproduction notebook (Colab): https://github.com/Sayuj63/vapt-env/blob/main/AISHA_RL_Training_Colab.ipynb