ppo-3DBall

chrisluo5311/ppo-3DBall

ppo-3DBall is a reinforcement learning model from chrisluo5311. Use it for the reinforcement learning task on the model card, and read the license before you ship it in a product. It is set up for ml-agents.

A PPO agent trained with the Unity ML-Agents Toolkit on the 3DBall (3D Balance Ball) environment, using domain randomization over the ball's mass and scale.

Downloads · 30 days

100% of all-time downloads

All-time downloads

Public

Repo size

14.1 MB

Likes

Public

Hugging Face

Repo makeup

Click a slice to open those files.

.mp46.6 MB · 91%

At a glance

Task: Reinforcement Learning
Library: ml-agents
Access: Public
Created: Sep 16, 2026
Updated: Sep 16, 2026
SHA: f39c1c33

Model card

From the Hugging Face model README

PPO Agent playing Unity 3DBall (with domain randomization)

A PPO agent trained with the Unity ML-Agents Toolkit on the 3DBall (3D Balance Ball) environment, using domain randomization over the ball's mass and scale.

The agent tilts a cube platform on two axes to keep a ball balanced on top of it for as long as possible — while the ball's physical properties are resampled every episode.

A replay is playable in the Video Preview panel at the top of this page (replay.mp4) — all 12 agents in the 3DBall scene running this policy in Inference Only mode, recorded in the Unity Editor for 26 s at 720×720.

Note that mass and scale show their Unity defaults of 1.0 in that recording. The randomized values are pushed in by the Python trainer over the environment-parameters side channel, so they apply during training only — not during plain Editor playback.

Results


Mean reward	100.00 ± 0.00
ML-Agents benchmark for 3DBall	100
Mean episode length	999 / 999 steps (never drops the ball)
Steps to convergence	~132,000
Total training steps	500,400
Wall-clock training time	5 min 24 s (Apple Silicon, `time_scale: 20`)

The reward ceiling for this task is 100 (+0.1 per step × 1000 agent decisions per episode), so the agent is fully solved: it holds the ball for the entire episode regardless of how the mass and scale are sampled.

Training curve

Step	Cumulative Reward	Episode Length
12,000	1.01	20.1
48,000	2.78	37.8
72,000	10.81	119.9
84,000	34.38	361.4
96,000	70.38	733.2
108,000	91.42	912.5
120,000	99.66	999.0
132,000	100.00	999.0
492,000	100.00	999.0

Reward reaches the ceiling at ~132k steps. Between 180k–400k it occasionally dips to ~86–95 as the randomization sampler draws harder mass/scale combinations, then stays pinned at 100 from 408k onward.

Final diagnostics (step 492,000)

Metric	Start	End
`Losses/Value Loss`	0.2008	0.0028
`Losses/Policy Loss`	0.1009	0.0998
`Policy/Entropy`	1.4189	1.2338
`Policy/Extrinsic Value Estimate`	-0.0857	10.0023
`Policy/Learning Rate`	3.0e-4	~0 (linear decay)
`Policy/Epsilon`	0.1976	0.1037

Environment


Environment	`3DBall` (Unity ML-Agents example)
Scene	`Assets/ML-Agents/Examples/3DBall/Scenes/3DBall.unity`
Parallel agents	12 (identical, shared Behavior Parameters)
Observation space	8 continuous — cube rotation (2), ball relative position (3), ball velocity (3)
Action space	2 continuous — X-rotation, Z-rotation
Reward	`+0.1` per step the ball stays on the cube; `-1.0` when it falls off
Episode limit	`MaxStep: 5000` with `DecisionPeriod: 5` → 1000 agent decisions

Domain randomization

Sampled uniformly at the start of every episode (environment_parameters in the config):

Parameter	Sampler	Min	Max	Unity default	Seed
`mass`	uniform	0.5	10.0	1.0	1427
`scale`	uniform	0.75	3.0	1.0	1428

This is what makes the run harder than vanilla 3DBall — the agent cannot memorize one ball's dynamics, it has to produce a policy robust across a 20× mass range and a 4× size range.

Training configuration

Trainer: PPO, 500,000 max steps.

Hyperparameters

Parameter	Value
`trainer_type`	`ppo`
`batch_size`	64
`buffer_size`	12000
`learning_rate`	3.0e-4
`learning_rate_schedule`	linear
`beta` (entropy regularization)	1.0e-3
`beta_schedule`	linear
`epsilon` (PPO clip)	0.2
`epsilon_schedule`	linear
`lambd` (GAE)	0.99
`num_epoch`	3
`shared_critic`	false

Network settings

Parameter	Value
`normalize`	true
`hidden_units`	128
`num_layers`	2
`vis_encode_type`	simple
`memory` (LSTM)	none

Reward signals

Parameter	Value
`extrinsic.gamma`	0.99
`extrinsic.strength`	1.0

Run settings

Parameter	Value
`max_steps`	500000
`time_horizon`	1000
`summary_freq`	12000
`keep_checkpoints`	5
`checkpoint_interval`	500000
`threaded`	false
`num_envs` / `num_areas`	1 / 1
`seed`	-1 (random)

Engine settings

Parameter	Value
`time_scale`	20
`quality_level`	5
`capture_frame_rate`	60
`target_frame_rate`	-1
`no_graphics`	false (trained in the Unity Editor)

Software versions


`ml-agents`	1.2.0.dev0
`ml-agents-envs`	1.2.0.dev0
Communicator API	1.5.0
PyTorch	2.8.0
Python	3.10
Unity	6.0 (6000.0.77f1), Apple Silicon / Metal

The complete resolved config is in configuration.yaml (and as JSON in config.json).

Repository contents

File	Description
`3DBall.onnx`	The trained policy — drop this into Unity
`3DBall/checkpoint.pt`	Full PyTorch training state, required for `--resume`
`3DBall/events.out.tfevents.*`	TensorBoard scalars
`configuration.yaml`	Full resolved training configuration
`config.json`	Same config as JSON
`replay.mp4`	26 s inference replay of all 12 agents; drives the Video Preview widget
`run_logs/timers.json`	Per-phase timing breakdown
`run_logs/training_status.json`	Checkpoint registry and final reward

Usage

In the Unity Editor

Open the 3DBall scene from the ML-Agents examples.
Download 3DBall.onnx and drop it into your Unity project's Assets/.
Select the 3DBall agent prefab → Behavior Parameters → assign the .onnx to Model.
Set Behavior Type to Inference Only and press Play.

Watch in the browser

Go to https://huggingface.co/unity
Enter the model id: chrisluo5311/ppo-3DBall
Select 3DBall.onnx and click Watch the agent play 👀

Reproduce the training

mlagents-learn config/ppo/3DBall_randomize.yaml --run-id=first3DBall_randomizeRun
# then press Play in the Unity Editor within 60 seconds

Resume from this checkpoint

mlagents-learn config/ppo/3DBall_randomize.yaml --run-id=first3DBall_randomizeRun --resume

References

Task: Reinforcement Learning
Library: ml-agents
Created: Sep 16, 2026
Updated: Sep 16, 2026