entropy-trpo-weights

Entropy-TRPO Model Weights

PyTorch checkpoints for the comparative study in A Review of Entropy-Based Extensions to Trust Region Policy Optimization.

Repository layout

Each checkpoint directory contains:

Hub path: {env_id}/{variant}/latest/ (e.g. CartPole-v1/entrpo/latest/).

The repo README is updated automatically during training with a Training progress table (epoch n/N, eval return, best return, KL) from results/summary.md, plus a JSON index of available checkpoints.

Notation

• $\rho_t(\theta)=\pi_\theta(a_t|s_t)/\pi_{\theta_{\text{old}}}(a_t|s_t)$, GAE advantages $\hat{A}_t$, trust-region radius $\delta$
• $\alpha$ — Roostaie advantage entropy; Xu ERO objective entropy (distinct roles, same symbol in each paper's row)
• $\beta$ — Xu ERC constraint coefficient (Xu Eq. 49)
• $c_{\mathrm{ent}}$ — PPO entropy bonus (Schulman et al., 2017; config field entropy_coef)

Variant definitions

$\mathcal{H}$ in EnTRPO rows is evaluated at the behavior policy $\pi_{\theta_{\text{old}}}$; in ERO/ERC/PPO rows at the candidate policy $\pi_\theta$.

ERO-TRPO implementation: Xu Table 1 — $H\mathbf{d}=\mathbf{g}+\alpha\mathbf{h}$, step size $\eta\le\sqrt{2\delta/(\mathbf{g}+\alpha\mathbf{h})^\top H(\mathbf{g}+\alpha\mathbf{h})}$, Xu line search ($\eta_0=0.5$, strict objective improvement, hard $\bar{D}_{\mathrm{KL}}\le\delta$).

ERC-TRPO implementation: Xu Table 1 — two CG solves ($\mathbf{u},\mathbf{v}$), $\eta\mathbf{u}+\beta\mathbf{v}$, Eq.~(49) acceptance $\bar{D}_{\mathrm{KL}}\le\delta+\beta,\mathbb{E}[\mathcal{H}]$, same Xu line search. Random ERC shares the ERC step; only the acceptance bonus is randomized.

Older Hub folders (trpo_entropy, trpo_buffer, …) remain valid; training resumes from them automatically.

Environments

See HYPERPARAMETERS.md for per-field provenance and paper/results/annex_hyperparameters.tex for tables.

Variants and paper sources

See metadata.json in each folder for full author names and URLs.

Usage

Training and evaluation code: GitHub — entropy-trpo (update URL when published).

git clone https://github.com/pre63/entropy-trpo.git
cd entropy-trpo
make setup          # install deps + create .env
# edit .env with HF_TOKEN and HF_REPO_ID
make download-weights
make eval-checkpoints

Citation

@article{entropytrporeview2026,
  title   = {A Review of Entropy-Based Extensions to Trust Region Policy Optimization},
  author  = {Green, Simon},
  journal = {IEEE Transactions},
  year    = {2026}
}

@article{roostaie2021entrpo,
  title   = {EnTRPO: Trust Region Policy Optimization Method with Entropy Regularization},
  author  = {Roostaie, Sahar and Ebadzadeh, Mohammad Mehdi},
  journal = {arXiv:2110.13373},
  year    = {2021}
}

@article{xu2024trpo,
  title   = {Trust region policy optimization via entropy regularization for {Kullback--Leibler} divergence constraint},
  author  = {Xu, Haotian and Xuan, Junyu and Zhang, Guangquan and Lu, Jie},
  journal = {Neurocomputing},
  volume  = {589},
  pages   = {127716},
  year    = {2024}
}

Training progress

Last updated: 2026-07-05 19:43:57 UTC

• Xu window — mean eval return over the last W epochs (Xu Table 6 protocol).

• Full run — mean eval return over all epochs.

• Device: cpu

• Config: configs/e4/e4_entrpo_substitute_cartpole_v5.yaml

• Jobs complete: 72/72

• Running: 0

CartPole-v1 (1M benchmark)

Humanoid-v5 (1M benchmark)

HumanoidStandup-v5 (1M benchmark)

CartPole-v5 (E4_CARTPOLE_V5 benchmark)

Available checkpoints

{
  "CartPole-v1": [
    "entrpo",
    "entrpo_buffer",
    "entrpo_entropy",
    "ppo",
    "trpo",
    "trpo_buffer",
    "trpo_entropy"
  ]
}