sangue-e-grafi-nemotron-nano-grpo

🩸 Sangue e Grafi — Nemotron Mini 4B GRPO Adapter

GRPO-reinforced LoRA adapter for Nemotron — the RL stage that aligns tool-use with ontology constraints.

<p align="center"> <img src="https://huggingface.co/spaces/cyberandy/sangue-e-grafi/resolve/main/banner.png" alt="Sangue e Grafi banner" width="700"/> </p>

Model Description

This is a LoRA (PEFT) adapter trained via Group Relative Policy Optimization (GRPO) on top of the Nemotron SFT adapter (already merged into nvidia/Nemotron-Mini-4B-Instruct). It is the second training stage for the Nemotron-based pipeline in the Sangue e Grafi project.

Training Pipeline

nvidia/Nemotron-Mini-4B-Instruct (4B)
        │
        ▼
   ┌─────────┐     500 adversarial scenarios
   │   SFT   │◄─── teacher traces (Gemini 2.5 Flash)
   └────┬────┘
        │  merge adapter
        ▼
   ┌─────────┐     reward: correct answer + valid tool use
   │  GRPO   │◄─── ontology constraint checking
   └────┬────┘
        │
        ▼
  THIS ADAPTER (apply on merged SFT base)

Training Details

Benchmark Results 📊

Cross-Architecture Comparison

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Step 1: Load base + merge SFT
base = AutoModelForCausalLM.from_pretrained("nvidia/Nemotron-Mini-4B-Instruct")
sft_model = PeftModel.from_pretrained(base, "cyberandy/sangue-e-grafi-nemotron-nano-sft-v7")
merged = sft_model.merge_and_unload()

# Step 2: Apply GRPO adapter
model = PeftModel.from_pretrained(merged, "cyberandy/sangue-e-grafi-nemotron-nano-grpo")
tokenizer = AutoTokenizer.from_pretrained("nvidia/Nemotron-Mini-4B-Instruct")

Intended Uses & Limitations

Intended uses:

• Cross-architecture validation of KG-grounded RL reasoning
• Research on GRPO reward shaping for tool-use agents
• Comparison with Gemma-based GRPO pipeline

Limitations:

• Must be applied on top of the merged SFT checkpoint — not standalone
• Requires the full KG agent framework for reported performance
• Domain-specific to Italian kinship / inheritance law

Project Links

Citation

@misc{sangue-e-grafi-2026,
  title   = {Sangue e Grafi: Small Models Beat Frontier LLMs on Adversarial Kinship Reasoning with Knowledge Graph Agents},
  author  = {Andrea Volpini},
  year    = {2026},
  url     = {https://github.com/cyberandy/sangue-e-grafi},
  note    = {Hugging Face Build Small Hackathon 2026}
}