ArcANE-32B-RLVR

ArcANE-32B-RLVR

<p> <a href="https://arxiv.org/abs/2606.05553"><img src="https://img.shields.io/badge/Paper-arXiv%3A2606.05553-B31B1B?logo=arxiv&logoColor=white" alt="Paper"></a> <a href="https://huggingface.co/datasets/holi-lab/ArcANE-Data"><img src="https://img.shields.io/badge/🤗%20Dataset-ArcANE--Data-FFD21E" alt="Dataset"></a> <a href="https://huggingface.co/holi-lab/ArcANE-32B-SFT"><img src="https://img.shields.io/badge/🤗%20Model-ArcANE--32B--SFT-FFD21E" alt="Model"></a> <a href="https://huggingface.co/holi-lab/ArcANE-32B-DPO"><img src="https://img.shields.io/badge/🤗%20Model-ArcANE--32B--DPO-FFD21E" alt="Model"></a> </p>

🏆 Accepted to EMNLP 2026 Main Conference

ArcANE-32B-RLVR is a Qwen3-32B model trained with SFT, DPO, and reinforcement learning with verifiable rewards. It continues from ArcANE-32B-DPO with Group Relative Policy Optimization (GRPO), using the ArcANE per-phase evaluation rubric as its reward.

<div align="center"> <img src="assets/main.png" alt="ArcANE character-arc construction and probe-generation pipeline" width="100%"> </div>

Model details

Intended use

ArcANE-32B-RLVR is intended for research on:

• point-in-time character role-play;
• character responses conditioned on a chapter-truncated Character Arc;
• rubric-based reinforcement learning for narrative phase fidelity;

The strongest evaluated setup supplies the relevant Character Arc only up to the queried chapter. Future phases must not be exposed to the model.

Training data

For RLVR, prompts at each probe's anchor phase are deduplicated by their (system, user) message pair, and the assistant turn is removed so the policy generates fresh rollouts.

The reward is mean(APF, RPF, RAE) / 100, scored by Qwen3.6-27B against the corresponding phase reference. PTF is not part of the reward because each rollout covers one phase.

Training parameters

Reproducibility

The released recipe is under training/rl in the ArcANE repository.

Evaluation

RLVR is evaluated under Arc context on records shared with the earlier ArcANE-32B stages. A separate DeepSeek-V4-Flash judge scores APF, RPF, RAE, and PTF from 1 to 100. Values in the cross-judge comparison should be compared within a judge column, not across judges.

These are shared-record results from the RLVR comparison. Under the primary evaluation judge, RLVR improves Overall by 8.0 points over DPO.

Usage

Use the Qwen3 chat template with thinking disabled. The example below is illustrative; replace the compact context with a valid chapter-truncated ArcANE Character Arc.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "holi-lab/ArcANE-32B-RLVR"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {
        "role": "system",
        "content": (
            'You are <character>, from "<novel>". You are at the point in '
            "the story corresponding to chapter <query_chapter>.\n\n"
            "Background you have access to:\n"
            "<context>\n<chapter-truncated Character Arc JSON>\n</context>"
        ),
    },
    {
        "role": "user",
        "content": "Scenario:\n<scenario>\n\nQuestion:\n<question>",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=1024, do_sample=True, temperature=1.0)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

For faithful point-in-time conditioning, remove all phases later than the query chapter.

Citation

@misc{song2026arcaneroleplayinglanguageagents,
      title={ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?}, 
      author={Woojung Song and Nalim Kim and Sangjun Song and Chaewon Heo and Jongwon Lim and Yohan Jo},
      year={2026},
      eprint={2606.05553},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.05553}, 
}