ArcANE-8B-DPO

ArcANE-8B-DPO

<p> <a href="https://arxiv.org/abs/2606.05553"><img src="https://img.shields.io/badge/Paper-arXiv%3A2606.05553-B31B1B?logo=arxiv&logoColor=white" alt="Paper"></a> <a href="https://huggingface.co/datasets/holi-lab/ArcANE-Data"><img src="https://img.shields.io/badge/🤗%20Dataset-ArcANE--Data-FFD21E" alt="Dataset"></a> <a href="https://huggingface.co/holi-lab/ArcANE-8B-SFT"><img src="https://img.shields.io/badge/🤗%20Model-ArcANE--8B--SFT-FFD21E" alt="Model"></a> </p>

🏆 Accepted to EMNLP 2026 Main Conference

ArcANE-8B-DPO is a Qwen3-8B model trained with supervised fine-tuning followed by Direct Preference Optimization (DPO). It learns to distinguish the correct response at a target narrative phase from a plausible, in-character response drawn from an adjacent but incorrect phase.

<div align="center"> <img src="assets/main.png" alt="ArcANE character-arc construction and probe-generation pipeline" width="100%"> </div>

Model details

Intended use

ArcANE-8B-DPO is intended for research on:

• point-in-time character role-play;
• character responses conditioned on a chapter-truncated Character Arc;
• distinguishing subtle behavioral changes between adjacent narrative phases;

The strongest evaluated setup supplies the relevant Character Arc only up to the queried chapter. Future phases must not be exposed to the model.

Training data

The SFT and DPO splits are derived from 12 training novels in the ArcANE corpus, covering 55 characters and 339 character axes. The SFT stage uses teacher-generated target responses under Arc context.

The DPO split contains 14,671 preference pairs from 2,516 unique probes. Each chosen response belongs to the anchor phase, while the rejected response belongs to an adjacent phase. Both responses share the same scenario and question.

The main validated evaluation slice is held out at the novel, character, arc, and probe levels from the training pool.

Training parameters

Reproducibility

The released recipes are training/sft/configs/full/sft.yaml and training/sft/configs/full/dpo.yaml in the ArcANE repository. From training/sft, run bash scripts/train_sft_full.sh followed by bash scripts/train_dpo_full.sh.

Evaluation

The paper evaluates free-form role-playing responses on a held-out five-novel slice containing 25 principal characters, 205 arcs, and 1,754 probes. A separate DeepSeek-V4-Flash judge scores four 1 to 100 metrics:

• APF: Action Phase-Fidelity;
• RPF: Reasoning Phase-Fidelity;
• RAE: Reasoning-Action Entailment;
• PTF: Phase Trajectory Fidelity.

Scores are pooled within each novel, novels receive equal weight, and Overall is the mean of the 12 probe-category by metric cells. Higher is better.

Held-out results with Arc context

DPO improves the Arc-context Overall score by 4.6 points over ArcANE-8B-SFT and by 13.8 points over Qwen3-8B under the same Arc context.

Usage

Use the Qwen3 chat template with thinking disabled. The example below is illustrative; replace the compact context with a valid chapter-truncated ArcANE Character Arc.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "holi-lab/ArcANE-8B-DPO"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {
        "role": "system",
        "content": (
            'You are <character>, from "<novel>". You are at the point in '
            "the story corresponding to chapter <query_chapter>.\n\n"
            "Background you have access to:\n"
            "<context>\n<chapter-truncated Character Arc JSON>\n</context>"
        ),
    },
    {
        "role": "user",
        "content": "Scenario:\n<scenario>\n\nQuestion:\n<question>",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=1024, do_sample=True, temperature=1.0)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

For faithful point-in-time conditioning, remove all phases later than the query chapter. Always remove literary_validation and evidence_summary; if any later phase is hidden, also remove pole_end and arc_direction. The paper's evaluation drew one sample with backend-default sampling, effectively temperature 1.0, and capped generation at 8,192 tokens.

Citation

@misc{song2026arcaneroleplayinglanguageagents,
      title={ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?}, 
      author={Woojung Song and Nalim Kim and Sangjun Song and Chaewon Heo and Jongwon Lim and Yohan Jo},
      year={2026},
      eprint={2606.05553},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.05553}, 
}