sangue-e-grafi-gemma4-e2b-grpo-run-f-v7

🩸 Sangue e Grafi — Gemma 4 E2B GRPO Adapter (v7)

GRPO-reinforced LoRA adapter — the second training stage that sharpens ontology-valid tool use via reward signals.

<p align="center"> <img src="https://huggingface.co/spaces/cyberandy/sangue-e-grafi/resolve/main/banner.png" alt="Sangue e Grafi banner" width="700"/> </p>

Model Description

This is a LoRA (PEFT) adapter trained via Group Relative Policy Optimization (GRPO) on top of the SFT adapter (already merged into google/gemma-4-E2B-it). It is the second and final training stage in the Sangue e Grafi pipeline.

GRPO refines the model by:

1. Rewarding correct final answers to inheritance-law questions.
2. Rewarding ontology-valid tool calls — the model learns to use only relationships and predicates that exist in the OWL kinship ontology.
3. Penalizing hallucinated reasoning and invalid tool invocations.

This approach is inspired by the RLM-on-KG paradigm: using reinforcement learning to align a language model's behaviour with structured knowledge-graph constraints.

Training Pipeline

google/gemma-4-E2B-it (4B)
        │
        ▼
   ┌─────────┐     500 adversarial scenarios
   │   SFT   │◄─── teacher traces (Gemini 2.5 Flash)
   └────┬────┘
        │  merge adapter
        ▼
   ┌─────────┐     reward: correct answer + valid tool use
   │  GRPO   │◄─── ontology constraint checking
   └────┬────┘
        │
        ▼
  THIS ADAPTER (apply on merged SFT base)

Training Details

Benchmark Results 📊

Cross-Architecture Comparison

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

# Step 1: Load base + merge SFT
base = AutoModelForCausalLM.from_pretrained("google/gemma-4-E2B-it")
sft_model = PeftModel.from_pretrained(base, "cyberandy/sangue-e-grafi-gemma4-e2b-sft-adversarial-v7")
merged = sft_model.merge_and_unload()

# Step 2: Apply GRPO adapter
model = PeftModel.from_pretrained(merged, "cyberandy/sangue-e-grafi-gemma4-e2b-grpo-run-f-v7")
tokenizer = AutoTokenizer.from_pretrained("google/gemma-4-E2B-it")

Tip: For a single ready-to-run file, use the GGUF quantized version instead.

Intended Uses & Limitations

Intended uses:

• Research on RL-aligned KG reasoning with small LMs
• Studying GRPO reward shaping for tool-use agents
• Italian inheritance-law question answering (with KG agent)

Limitations:

• Must be applied on top of the merged SFT checkpoint — not standalone
• Requires the ontology-grounded KG agent framework for reported results
• Domain-specific to Italian kinship / inheritance law

Project Links

Citation

@misc{sangue-e-grafi-2026,
  title   = {Sangue e Grafi: Small Models Beat Frontier LLMs on Adversarial Kinship Reasoning with Knowledge Graph Agents},
  author  = {Andrea Volpini},
  year    = {2026},
  url     = {https://github.com/cyberandy/sangue-e-grafi},
  note    = {Hugging Face Build Small Hackathon 2026}
}