ArcANE-8B-SFT

ArcANE-8B-SFT

<p> <a href="https://arxiv.org/abs/2606.05553"><img src="https://img.shields.io/badge/Paper-arXiv%3A2606.05553-B31B1B?logo=arxiv&logoColor=white" alt="Paper"></a> <a href="https://huggingface.co/datasets/holi-lab/ArcANE-Data"><img src="https://img.shields.io/badge/🤗%20Dataset-ArcANE--Data-FFD21E" alt="Dataset"></a> <a href="https://huggingface.co/holi-lab/ArcANE-8B-DPO"><img src="https://img.shields.io/badge/🤗%20Dataset-ArcANE--8B--DPO-FFD21E" alt="Models"></a> </p>

🏆 Accepted to EMNLP 2026 Main Conference

ArcANE-8B-SFT is a Qwen3-8B model fully fine-tuned on the supervised split of ArcANE (Arc-Aware Narrative Evaluation). It is designed for English role-playing responses that reflect a character's behavioral state at a specified point in a narrative, rather than treating the character as a fixed persona.

<div align="center"> <img src="assets/main.png" alt="ArcANE character-arc construction and probe-generation pipeline" width="100%"> </div>

Model details

Intended use

ArcANE-8B-SFT is intended for research on:

• point-in-time character role-play;
• character responses conditioned on a chapter-truncated Character Arc;
• behavioral and value changes across narrative phases;

The strongest evaluated setup supplies the relevant Character Arc only up to the queried chapter. Future phases must not be exposed to the model.

Training data

The SFT split is derived from 12 training novels in the ArcANE corpus, covering 55 characters and 339 character axes. For each (probe, phase) pair under Arc context, three teacher completions were sampled from gpt-5.4-mini. The teacher privately received the phase reference, but each stored training row retained only the character system prompt, scenario-question user prompt, and answer. Reference text was therefore not included in the model input.

The main validated evaluation slice is held out at the novel, character, arc, and probe levels from the training pool.

Training parameters

Reproducibility

The released recipe is training/sft/configs/full/sft.yaml in the ArcANE repository. From training/sft, run bash scripts/train_sft_full.sh.

Evaluation

The paper evaluates free-form role-playing responses on a held-out five-novel slice containing 25 principal characters, 205 arcs, and 1,754 probes. A separate DeepSeek-V4-Flash judge scores four 1 to 100 metrics:

• APF: Action Phase-Fidelity;
• RPF: Reasoning Phase-Fidelity;
• RAE: Reasoning-Action Entailment;
• PTF: Phase Trajectory Fidelity.

Scores are pooled within each novel, novels receive equal weight, and Overall is the mean of the 12 probe-category by metric cells. Higher is better.

Held-out results with Arc context

The Arc-context score is 5.9 points above this checkpoint's strongest non-Arc context and 9.2 points above Qwen3-8B under the same Arc context. These results measure ArcANE phase fidelity, not general knowledge, safety, or broad instruction-following quality.

Usage

Use the Qwen3 chat template with thinking disabled. The example below is illustrative; replace the compact context with a valid chapter-truncated ArcANE Character Arc.

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "holi-lab/ArcANE-8B-SFT"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype="auto",
    device_map="auto",
)

messages = [
    {
        "role": "system",
        "content": (
            'You are <character>, from "<novel>". You are at the point in '
            "the story corresponding to chapter <query_chapter>.\n\n"
            "Background you have access to:\n"
            "<context>\n<chapter-truncated Character Arc JSON>\n</context>"
        ),
    },
    {
        "role": "user",
        "content": "Scenario:\n<scenario>\n\nQuestion:\n<question>",
    },
]

inputs = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=1024, do_sample=True, temperature=1.0)
print(tokenizer.decode(outputs[0][inputs.shape[-1]:], skip_special_tokens=True))

For faithful point-in-time conditioning, remove all phases later than the query chapter. Always remove literary_validation and evidence_summary; if any later phase is hidden, also remove pole_end and arc_direction. The paper's evaluation drew one sample with backend-default sampling, effectively temperature 1.0, and capped generation at 8,192 tokens.

Citation

@misc{song2026arcaneroleplayinglanguageagents,
      title={ArcANE: Do Role-Playing Language Agents Stay in Character at the Right Time?}, 
      author={Woojung Song and Nalim Kim and Sangjun Song and Chaewon Heo and Jongwon Lim and Yohan Jo},
      year={2026},
      eprint={2606.05553},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2606.05553}, 
}