Downloads · 30 days
0
Auryal/openvla-oft-libero-microwave-grm-grpo
openvla-oft-libero-microwave-grm-grpo is a reinforcement learning model from Auryal. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
LoRA fine-tune of OpenVLA-OFT on LIBERO-10 Task 9 ("put the yellow and white mug in the microwave and close it") using GRPO with dense rewards from Robo-Dopamine's GRM.
Downloads · 30 days
0
Access
Public
Updated Mar 8, 2026
Repo size
60.3 GB
Likes
0
Public
Click a slice to open those files.
.distcp45.3 GB · 75%
From the Hugging Face model README
LoRA fine-tune of OpenVLA-OFT on LIBERO-10 Task 9 ("put the yellow and white mug in the microwave and close it") using GRPO with dense rewards from Robo-Dopamine's GRM.
| Parameter | Value |
|---|---|
| Base model | RLinf/RLinf-OpenVLAOFT-LIBERO-130 |
| Fine-tuning | LoRA via GRPO (Group Relative Policy Optimization) |
| Reward model | Robo-Dopamine GRM-3B (cumulative eval mode) |
| Task | LIBERO-10 #9: put the yellow and white mug in the microwave and close it |
| Hardware | 4x A100-80GB (1 for GRM, 3 for FSDP training) |
| Epochs | 20 |
| Parallel envs | 24 |
| Infrastructure | RLinf v0.2.0 |
The base model started at 86.5% success rate. Dense GRM-only rewards caused reward hacking — policy loss converged but success rate collapsed to 0% by epoch 20. The step-10 checkpoint (61.5% success) is the last usable checkpoint.
See TRAINING_REPORT.md in the associated dataset repo for full failure analysis.
global_step_10/ — Epoch 10, 61.5% success rate (before collapse)global_step_20/ — Epoch 20, 0% success rate (collapsed)This model demonstrates reward hacking from dense RL without sparse reward anchoring or KL regularization. It is released for research purposes — to study failure modes in reward-model-guided policy optimization.
@misc{auryal2026grm-grpo,
title={Dense Reward RL with GRM for Robotic Manipulation},
author={Auryal},
year={2026},
}