canary-grpo-job-output

Canary GRPO Job Output

This repository contains a LoRA adapter trained for the Canary Release Manager OpenEnv environment.

Model overview

• Base model: unsloth/Qwen2.5-1.5B-Instruct
• Training pipeline: SFT warm-start + GRPO
• Hardware: Tesla T4
• Environment calls: 1021

Task

The model is trained to make rollout decisions in a canary deployment environment.
It predicts:

• rollout action
• structured state assessment
• SLO breach flag
• short grounded reasoning

Supported actions in the environment include:

• increase_5
• increase_10
• increase_25
• hold
• rollback

The environment is hosted separately as an OpenEnv-compatible service.

Training summary

The training run completed successfully with:

• SFT warm-start completed
• GRPO completed for 40 steps
• 100% parse success during RL
• no reward pipeline errors

Heldout evaluation

Heldout performance comparison

Reward curve

Before vs After Demo Examples

Example 1 — hard / seed 903

• Base model action: hold
• Base reward: 0.40
• Trained model action: increase_25
• Trained reward: 0.56

Observation: healthy canary metrics with no meaningful drift.
The base model stayed conservative and held.
The trained adapter promoted traffic and received a higher environment reward.

Example 2 — expert / seed 905

• Base model action: hold
• Base reward: 0.40
• Trained model action: increase_25
• Trained reward: 0.49

Observation: healthy metrics with slightly more drift than the easier case.
The trained adapter again promoted instead of over-holding and achieved a higher score.

Reward design

The environment reward combines multiple components, including breach detection, rollout timing and safety, promotion behavior, and reasoning quality. The setup is designed to reward correct rollout behavior while discouraging unsafe or reward-hacked shortcuts.

Key observations

• Heldout performance improved across easy, medium, hard, expert, and silent tasks.
• Recovery performance stayed strong and unchanged.
• Parse success during RL was 100%, meaning the model reliably produced the required structured format during training.
• On heldout demo examples, the trained adapter outperformed the base model by choosing more promotive actions instead of defaulting to hold.

Training metrics

• Train runtime: 282.03 s
• Train steps per second: 0.142
• Final GRPO train loss: 0.00114

Notes / limitations

• The learned policy is still narrow and often prefers aggressive promotion actions such as increase_25.
• Reward saturation suggests the policy improved, but not with especially rich behavior diversity.
• The before-vs-after demo improvements are real but modest.
• This is a hackathon-scale training run, not a large-scale research benchmark.

Reproducibility

• Training Colab notebook: https://colab.research.google.com/drive/1-MZbE9wimHLo8bIBpGwaKg1Or2IFdwg3?usp=sharing
• Before/after demo Colab notebook: https://colab.research.google.com/drive/1yX2LCPFhLEgyFzINZF_XR9JPPSKitzVV?usp=sharing
• Final successful training run: Hugging Face Jobs
• Saved adapter: this repository
• Environment Space: https://huggingface.co/spaces/aparnasingha400/canary-release-manager

The final results shown in this README come from the Hugging Face Jobs run. The linked Colab notebook is provided as the minimal reproducible training pipeline using OpenEnv + Unsloth/TRL.

Additional materials

• Mini-blog: https://huggingface.co/spaces/aparnasingha400/Canary-Release-Manager-Blog

Related links

• Environment repo: https://github.com/Lekhana-Dinesh/Canary-Release-Manager
• Environment Space: https://huggingface.co/spaces/aparnasingha400/canary-release-manager
• Blog Space: https://huggingface.co/spaces/aparnasingha400/Canary-Release-Manager-Blog