Downloads · 30 days
12
17% of all-time downloads
jinrui123/llmrnn-grpo-phase1-v2judge-run01-qwen3-8b-global-step-17-merged
llmrnn-grpo-phase1-v2judge-run01-qwen3-8b-global-step-17-merged is a text generation model from jinrui123. Use it when you need the model to write or continue text. The card lists the license as apache-2.0.
GRPO-trained memory-update head for the LLM-as-RNN clinical pipeline (reproducing arXiv 2601.13352), now on a Qwen3-8B backbone. The LoRA adapter has been folded into the base model with peft.mergeandunload(), so this…
Downloads · 30 days
12
17% of all-time downloads
All-time downloads
71
Public
Parameters
8.2B
16.4 GB on disk
Likes
0
Public
Click a slice to open those files.
.safetensors16.4 GB · 100%
From the Hugging Face model README
GRPO-trained memory-update head for the LLM-as-RNN clinical pipeline
(reproducing arXiv 2601.13352), now on a Qwen3-8B backbone. The LoRA
adapter has been folded into the base model with
peft.merge_and_unload(), so this is a stand-alone HF model — load it
like any other Qwen3-8B variant, no PEFT runtime needed.
Role: this is not a general-purpose chat model. It is fine-tuned to rewrite a natural-language "evolving summary" (the hidden state
h_tin the LLM-as-RNN setup) after each patient visit. Use it as thememory_modelin the 4-model LLM-as-RNN pipeline.Smoke / feasibility checkpoint: only 1 epoch (17 optimizer steps) on 100 patients. The prior Llama-3.2-3B run trained for 20 epochs / 340 steps. Use for "does Qwen3-8B work as a memory model?" rather than as a strong final artifact.
| field | value |
|---|---|
| experiment | phase1_v2judge_run01_Qwen3_8b |
| source checkpoint | global_step_17 (only checkpoint of this run) |
| launched | 2026-05-16 22:10 UTC, host c302-004.ls6.tacc.utexas.edu (TACC Lonestar6) |
| git SHA at launch | 836ca40 (clean — includes the <think> strip fix in reward fn) |
| container | verlai/verl-vllm017 (verl 0.7.1, torch 2.10, vllm 0.17) |
| GPU layout | 2× A100 40GB actor + rollout (cuda:0,1, TP=2), 1× A100 40GB judge (cuda:2) |
| total optimizer steps | 17 (= 1 epoch over 275-row parquet at batch 16) |
| wall time | ~1h45 from trainer launch to checkpoint save |
Qwen/Qwen3-8Ball-linear (covers attention q/k/v/o_proj
gate/up/down_proj — same layout as the Llama runs since Qwen3
uses Llama-style transformer blocks)lora_alpha = 32 was preserved when folding the adapter
back into the base. Pure HF weights in this repo; no lora_adapter/
subdir.| GRPO knob | value | (vs Llama run02) |
|---|---|---|
| LR | 3e-6 | same |
| KL β | 0.001, use_kl_loss=True, use_kl_in_reward=False | same |
| KL estimator | low_var_kl (k3) | same |
| entropy / grad clip | 0 / 1.0 | same |
| group size G | 8 rollouts per prompt | same |
| train_batch / ppo_mini_batch | 16 / 16 → 1 mini-batch per step | same |
| ppo_micro_batch_per_gpu | 2 | was 4 for 3B; halved for 8B |
| rollout temperature / top-p | 0.8 / 0.95 | same |
| rollout TP / gpu_memory_utilization | 2 / 0.5 | was 0.6; 0.6 failed vLLM Free memory < desired because FSDP's 8B actor leaves only ~22 GB free on a 40 GB A100 |
| max prompt / max response length | 3072 / **2048** tokens | was 512; Qwen3 wraps output in <think>...</think> CoT before JSON, eats 500-2000 tokens |
| epochs / save_freq / test_freq | 1 / 17 / -1 | run02 had 20 / 20 / 5 |
Algorithm: standard verl 0.7.1 GRPO — group-relative advantage (no critic,
the 8 candidates per prompt share a common baseline), PPO-clipped surrogate,
KL anchor to the frozen base via model.disable_adapter(). Validation
disabled for this smoke run (no val.parquet produced).
Same rubric as run02 — the judge model and prompt template were not changed when swapping the policy backbone.
OpenRubrics/RubricARM-8B-Judgelocalhost:8001, T=0, max_tokens=1500,
max_model_len=12000, on cuda:2.training/configs/rubric_v2_rubricARM_scalar.yaml
(SHA1 e2f89fec…) — single file controls rubric content + judge model
start_judge_server.sh
cannot drift.Outcome-aware: the judge sees the next visit's prediction + ground truth
Score / 9 ∈ (0, 1].<think> strip in reward functionQwen3 emits a <think>...reasoning...</think> CoT block before the JSON
answer. The original rubric_reward.py parser tried json.loads on the
raw output, fell back to "first { to last }" — which spans the think
block AND the answer, and json.loads then fails → returns
PARSE_FAIL_REWARD = 0.05. With G=8 candidates all hitting the same bug,
GRPO advantage would collapse to zero.
Fix (commit 836ca40): strip the <think> block as the first line of
_parse_candidate_h_t:
THINK_BLOCK_RE = re.compile(r"<think\b[^>]*>.*?</think>\s*", re.DOTALL | re.IGNORECASE)
def _parse_candidate_h_t(solution_str: str) -> Optional[str]:
solution_str = THINK_BLOCK_RE.sub("", solution_str)
...
This made the parse-failure rate 0.3% instead of an estimated 80%+.
| condition | reward |
|---|---|
| policy output is not valid JSON | 0.05 |
judge response has Score: N line | N/9 |
| judge response short (<20 chars) with a digit | last digit /9 |
judge response long + no Score: N | 0.5 (refuse to parse) |
Audit trail: every reward call appended to
$SCRATCH/llm_as_rnn/rubric_audit/calls_<pid>.jsonl.
data/splits/cleaned_df_train_100.json (100 MIMIC-IV patients,
seed 42 from the cleaned 6488-patient pool).update_prompt is rendered text,
not pre-tokenized, so verl applies Qwen3's tokenizer at training time.max(0, n_visits - 2)
(each patient with ≥3 visits emits n - 2 memory-update steps).Parquet schema (matches verl GSM8K convention):
{
"prompt": [{"role": "user", "content": "<rendered prompt_update template>"}],
"data_source": "llm_as_rnn_memory_update",
"ability": "clinical_summary",
"reward_model": {
"style": "rubric",
"ground_truth": "<JSON-serialized TrajectoryStep>"
},
"extra_info": {"split", "patient_id", "visit_index", "visit_id"}
}
data/splits/cleaned_df_train_100.json
│
[Phase B] dump_trajectories_to_parquet.py
└── one TrajectoryStep per (patient, visit_index ∈ [0..n-3])
│
▼
$SCRATCH/data/llm_as_rnn/train.parquet (275 rows)
│
┌────────┴─────────┐
│ │
[judge server] [verl trainer]
RubricARM-8B verl.trainer.main_ppo
on cuda:2 actor FSDP cuda:0,1 TP=2,
│ Qwen3-8B + LoRA(r=16, α=32)
│ │
└─ HTTP /v1/completions ─┘
│
▼
GRPO inner loop per optimizer step (~5.8 min/step on this hardware):
(a) sample 16 prompts from parquet
(b) actor rollout (vLLM) → G=8 candidates per prompt
Qwen3 emits "<think>...</think>\n{JSON answer}"
(c) for each candidate:
rubric_reward.compute_score()
├── strip <think>...</think> ← NEW for Qwen3
├── parse evolving_summary JSON
├── render rubric_v2 template
└── HTTP call to judge → parse "Score: N" → N/9
(d) GRPO group-relative advantage (G=8 share a baseline)
loss = -E[clip(π/π_old) · adv] + β · KL(π ‖ π_ref)
(e) PPO mini-batch (16/16, micro=2/GPU) → optimizer step
→ FSDP sync → vLLM rollout worker pulls latest LoRA
│
▼ (after step 17 = end of epoch 1)
checkpoints/llmrnn_grpo/phase1_v2judge_run01_Qwen3_8b/global_step_17/
│
▼
[merge_and_upload.sh]
(1) verl.model_merger merge → gathers FSDP shards into a flat HF base dir
(2) PEFT merge_and_unload() → base + lora_adapter(α=32) folded together
(3) huggingface_hub upload → THIS REPO
Across the full 1-epoch run, 2176 reward calls (= 17 steps × 16 prompts × 8 candidates):
parse_fail (0.05): 0.3% — the <think> strip fix workedjudge_no_digit fallback (0.5): 0.1% — RubricARM occasionally writes
long CoT without the Score: N final line; parser correctly refuses to
guess a digitThe reward distribution is not collapsed — there is real variance both within and across G=8 groups, so GRPO gradient is non-degenerate.
Drop this model into the LLM-as-RNN inference pipeline as the memory_model:
# In configs/experiment*.yaml under memory_model:
memory_model:
backend: vllm
model_name: jinrui123/llmrnn-grpo-phase1-v2judge-run01-qwen3-8b-global-step-17-merged
temperature: 0.0
max_tokens: 2048 # Qwen3 needs headroom for its <think> block
It expects the exact prompt_update template defined in
configs/experiment1_seperateJudge.yaml (the one in the LLM-as-RNN repo).
Downstream code must accept output of the form
<think>...reasoning...</think>\n{"evolving_summary": ...} and strip the
think block before JSON parse — exactly the same strip we apply in
training/verl_adapter/rubric_reward.py:_parse_candidate_h_t().
val.parquet was not dumped before launch,
so validation was disabled. Reward curves are train-side only.cleaned_df_train_100).
Generalization to the 6288-patient test split is untested.RubricARM-8B-Judge is natively
pairwise; we force a scalar response. Absolute reward values not
directly comparable across rubric versions.<think> block required in inference. Qwen3's chat template emits
<think> blocks. Downstream code must strip them before JSON parse.model_merger merge does not actually merge LoRA. We work
around this with a real PEFT merge_and_unload() step using the
original adapter (alpha=32). This artifact is the post-PEFT-merge
version, so the LoRA is applied.Source code (in jinrui-hub/LLMasRNN at SHA 836ca40):
| component | path |
|---|---|
| launcher | training/verl_adapter/apptainer/run_phase1.sh |
| rubric (judge prompt + model + serving) | training/configs/rubric_v2_rubricARM_scalar.yaml |
reward function (with <think> strip) | training/verl_adapter/rubric_reward.py |
| parquet dumper | training/verl_adapter/dump_trajectories_to_parquet.py |
| trajectory dataclass | training/types.py |
| RLN forward pass | models/rln_core.py |
| inference YAML (consumes this model) | configs/experiment1_seperateJudge.yaml |
| merge + upload | training/verl_adapter/apptainer/merge_and_upload.sh |
Auto-generated companion files:
logs/MANIFEST_phase1_v2judge_run01_Qwen3_8b.txt — machine-readable
manifest (git SHA, judge model, all GRPO knobs)logs/TRAINING_REPORT_phase1_v2judge_run01.md — human-facing report
for the Llama-3.2-3B predecessor run; setup is otherwise identical.