anima-sherlock-qwen36-27b-dpo-lora

Anima Sherlock Qwen3.6-27B DPO LoRA

Anima Sherlock is a PEFT LoRA adapter for source-grounded Chinese Sherlock Holmes dialogue. It uses SFT followed by DPO; deterministic services control evidence, state, and memory.

Model details

Load the adapter weights together with the pinned base-model revision at inference time.

Repository contents

Intended use

Direct uses include Chinese Sherlock Holmes dialogue, source-grounded role-playing research, and mystery prototypes with validated actions and authoritative external state.

The Anima runtime complements the adapter with access control, spoiler policy, typed tools, memory isolation, and deterministic game state.

Loading the adapter

The example below loads the pinned base model in 4-bit mode before attaching the adapter. It requires a CUDA-capable environment and compatible versions of PyTorch, Transformers, PEFT, Accelerate, and bitsandbytes.

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

base_model_id = "Qwen/Qwen3.6-27B"
base_revision = "6a9e13bd6fc8f0983b9b99948120bc37f49c13e9"
adapter_id = "Bot42/anima-sherlock-qwen36-27b-dpo-lora"

quantization = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_use_double_quant=True,
    bnb_4bit_compute_dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained(base_model_id, revision=base_revision)
model = AutoModelForCausalLM.from_pretrained(
    base_model_id,
    revision=base_revision,
    quantization_config=quantization,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter_id)
model.eval()
messages = [
    {"role": "system", "content": "你是夏洛克·福尔摩斯。依据已知事实推理，不提前泄露案件答案。"},
    {"role": "user", "content": "先告诉我，这个房间里最值得检查的细节是什么？"},
]
input_ids = tokenizer.apply_chat_template(
    messages,
    tokenize=True,
    add_generation_prompt=True,
    enable_thinking=False,
    return_tensors="pt",
).to(model.device)
with torch.inference_mode():
    output_ids = model.generate(
        input_ids, max_new_tokens=256, do_sample=True, temperature=0.7,
        pad_token_id=tokenizer.eos_token_id,
    )
answer = tokenizer.decode(output_ids[0, input_ids.shape[-1]:], skip_special_tokens=True)
print(answer)

The unquantized 27B base requires substantially more accelerator memory. The adapter can also be served behind an OpenAI-compatible endpoint; the associated project repository provides a model client and a deterministic Agent runtime.

Training data

The SFT stage used 260 accepted records spanning 218 conversations: 118 curated records and 142 deterministic task records. Coverage included grounded general and lore QA, memory create/read/update/delete intents, out-of-character attacks, unknown or future questions, and unsafe requests. All 260 records targeted the same Sherlock persona.

The DPO stage used 70 accepted preference pairs spanning 52 conversations: 30 curated and 40 deterministic pairs. Twenty-five pairs were history-conditioned. Each record contained a shared conversational prompt and strict chosen and rejected completions reviewed against the same role, grounding, safety, and format rubric.

Dataset controls covered schema, provenance, duplicates, split isolation, length, prompt/label leakage, and rubric review. SFT masking exposed zero prompt tokens as labels, and the accepted set required no truncation at length 4,096.

Training procedure

Both stages used 4-bit NF4 base weights, double quantization, bfloat16 quantization compute, gradient checkpointing, activation offloading, and LoRA rank 16 / alpha 32 / dropout 0.0. LoRA covered q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj, in_proj_qkv, in_proj_z, and out_proj.

DPO initialized the policy from the accepted SFT adapter and kept a separate SFT reference adapter frozen. The implementation verified 800 trainable policy parameter tensors, 800 frozen reference parameter tensors, and no unexpected trainable reference parameters. Split forward passes and chunked selective log-softmax reduced peak memory without changing the DPO objective.

The runs used one GPU. Observed peak CUDA allocation was approximately 25.1 GiB for SFT and 25.4 GiB for DPO. SFT completed in 32.9 minutes with final training loss 0.5295; DPO completed in 11.7 minutes with final training loss 0.6902.

Evaluation and release validation

Release acceptance focused on artifact and application correctness:

• The published weight hash matches the accepted training output.
• All 800 tensors are finite and the PEFT configuration resolves to the pinned base model.
• The adapter was reloaded over the base revision and produced non-zero policy deltas relative to the base model.
• A live final-adapter inference smoke test completed through the model endpoint.
• The associated Agent repository passed its unit, contract, and clean-export test suites before release staging.

Together, these checks establish artifact integrity, method conformance, and runtime compatibility. Task-level evaluation can be extended for each downstream prompt policy and case domain.

Limitations

• The adapter is specialized for one character, Chinese dialogue, and a focused role-playing task distribution.
• Grounding quality depends on the supplied persona, lore, retrieval context, and application prompt policy.
• Tool and memory integrations require schema validation, authorization, and state checks in the surrounding application.
• Quantized inference can vary across hardware and library versions. Pin the base revision and validate the adapter hash in controlled deployments.

Reproducibility and project

The release was produced with PyTorch 2.7.1, Transformers 5.14.1, PEFT 0.19.1, Accelerate 1.14.0, and bitsandbytes 0.49.2. Portable SFT/DPO configurations, training entry points, the Agent runtime, memory layer, case engine, and a no-GPU browser demo are available in the Anima Sherlock Agent repository.

Method references: LoRA, QLoRA, Direct Preference Optimization, and RoleLLM.

License

The adapter is released under Apache-2.0. Use of the base model remains subject to the base model's own license and terms. Sherlock Holmes source and asset licenses are documented separately in the associated project repository.