catgirl-rp-training

Catgirl Role-Play: Qwen3.5-2B SFT+DPO + Ollama Proxy

A complete pipeline for fine-tuning Qwen3.5-2B into a catgirl role-play model with chain-of-thought thinking, plus a standalone Ollama API proxy that fixes thinking/content parsing for all Qwen3.5-family models.

Key features:

• 🐱 Catgirl role-play model with native thinking / response chain-of-thought
• 🔧 Ollama proxy that correctly splits thinking/content (fixes PARSER qwen3.5 bug)
• 🎯 DPO-refined model for more expressive personality
• 📦 Ready-to-use GGUF exports (Q8_0) for Ollama

---

Quick Start

Using with Ollama + Proxy

# 1. Download GGUF from this repo (catgirl-dpo-v2.gguf) 与 Modelfile.dpo 放同一目录
# 2. Create Ollama model (Modelfile 的 FROM 用文件名, 自动找到同级 GGUF)
ollama create catgirl:dpo -f Modelfile.dpo

# 3a. 要拿纯 content, 让客户端显式带 "think": false (绕开 ollama 默认 thinking 分离)
curl http://localhost:11434/api/chat -d '{
  "model":"catgirl:dpo",
  "messages":[{"role":"user","content":"晚安"}],
  "stream":false, "think":false
}'

# 3b. 要同时拿到 thinking + content 两个字段, 启动本仓库的代理并指向它
python proxy.py --port 11435   # 后台运行; proxy 走 /api/generate raw 路径稳定拆分

curl http://localhost:11435/api/chat -d '{
  "model":"catgirl:dpo",
  "messages":[{"role":"user","content":"晚安"}],
  "stream":false
}'
# 返回 message.thinking 与 message.content 两个字段

Important — about thinking/content parsing:

Ollama ships a RENDERER qwen3.5 / PARSER qwen3.5 pair whose behavior is buggy for this model: with them enabled, the entire output lands in the thinking/reasoning field and content comes back empty. The Modelfiles in this repo therefore deliberately do not use RENDERER/PARSER — they render ChatML directly.

Even so, Ollama's /api/chat still tries to do its own thinking separation for models with the thinking capability, and the result is unreliable (content may be empty). Two reliable options:

• Pull just the content: pass "think": false to /api/chat — content will then contain the full model output.
• Pull structured thinking + content: run proxy.py (port 11435). The proxy routes through Ollama's /api/generate with raw: true, then splits the raw thinking / response markers itself — giving you both fields, reliably.

Point your client at localhost:11435 instead of localhost:11434 when you want the structured split.

Using HuggingFace Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "greenhandzdl/catgirl-rp-training",
    subfolder="models/dpo-merged",
    trust_remote_code=True,
    torch_dtype="auto",
    device_map="auto",
)
tokenizer = AutoTokenizer.from_pretrained(
    "greenhandzdl/catgirl-rp-training",
    subfolder="models/dpo-merged",
)

prompt = (
    "<|im_start|>system\n你是小雪，一个猫娘。<|im_end|>\n"
    "<|im_start|>user\n你好<|im_end|>\n"
    "<|im_start|>assistant\n thinking\n"
)
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=256, stop_strings=["<|im_end|>"])
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

---

Model Formats

Chat Response Structure

The model uses Chain-of-Thought with thinking / response markers:

 thinking
{internal_monologue}
 response

{reply_content}<|im_end|>

The prompt ends with <|im_start|>assistant\n thinking\n which instructs the model to use this format.

Proxy Parsing

The proxy intercepts Ollama's raw /api/generate output and parses it into structured thinking/content:

• \n response\n\n — standard double-newline separator
• \n response\n — single-newline separator (model-dependent)
• response\n — no-thinking mode (model skips to response directly)

---

Repository Structure

├── README.md
├── proxy.py                    # Ollama API proxy (port 11435), 结构化 thinking/content 拆分
├── Modelfile.sft                # Ollama Modelfile (SFT model) — ChatML template, 不用 RENDERER/PARSER
├── Modelfile.dpo                # Ollama Modelfile (DPO model) (⭐ recommended)
├── catgirl-sft-v2.gguf          # SFT model Q8_0 GGUF (~1.9GB)
├── catgirl-dpo-v2.gguf          # DPO model Q8_0 GGUF (~1.9GB) ← recommended
├── upload.sh                    # 用 hf CLI 上传本仓库到 HF 的脚本
├── models/
│   ├── sft-merged/              # SFT LoRA merged → full model (safetensors ~3.5GB)
│   └── dpo-merged/              # DPO LoRA merged → full model (safetensors ~3.5GB) ⭐
├── scripts/
│   ├── config.py                # 集中路径配置 + 环境检查
│   ├── personas.py             # 三种猫娘人设定义 (tsundere / sweet / ojou_sama)
│   ├── api_client.py            # 共享 LLM API 客户端 (Anthropic-compatible, 重试+退避)
│   ├── generate_sft_data.py     # SFT 骨架生成器
│   ├── fill_sft_data.py         # SFT 数据 LLM 填充器
│   ├── fill_sft_fast.py         # SFT 并行批量填充
│   ├── validate_sft_data.py     # SFT 数据质量校验
│   ├── fix_failed_samples.py    # 修复 API 填充失败样本
│   ├── sft_train.py             # SFT 训练 (TRL SFTTrainer + LoRA, messages 格式)
│   ├── sft_train_v5.py          # SFT 训练 (文本格式 + completion-only loss)
│   ├── generate_dpo_data.py     # DPO 偏好对生成器
│   ├── generate_dpo_from_sft.py # 从 SFT 数据复用 chosen 生成 DPO 偏对
│   ├── fill_dpo_fast.py         # DPO 并行高效填充 (2-call 方案)
│   ├── dpo_train.py             # DPO 训练 (TRL DPOTrainer + LoRA)
│   ├── post_train_pipeline.sh   # 后处理: merge LoRA → 导 GGUF → 建 Ollama 模型
│   └── inference.py             # 交互式对话 + 批量推理
└── data/
    ├── sft_sample.jsonl             # 训练数据格式样本
    ├── sft_train.jsonl             # SFT 完整训练数据 (1671 条)
    ├── sft_skeleton.jsonl           # SFT 骨架 (3000 条, 含占位符)
    ├── dpo_train.jsonl             # DPO 完整偏好对 (1670 对)
    └── inference_dpo_{tsundere,sweet,ojou}.jsonl  # 推理验证输出

---

Model Comparison

DPO model recommended — richer thinking, more expressive catgirl persona, better consistency.

---

API Endpoints (Proxy)

Ollama Native: POST /api/chat

curl http://localhost:11435/api/chat -d '{
  "model": "catgirl:dpo",
  "messages": [{"role": "system", "content": "你是小雪，猫娘。"},
               {"role": "user", "content": "晚安"}],
  "stream": false
}'

Response:

{
  "model": "catgirl:dpo",
  "message": {
    "role": "assistant",
    "content": "主人晚安喵~",
    "thinking": "主人说晚安呢...人家也想睡了"
  },
  "done": true
}

OpenAI Compatible: POST /v1/chat/completions

curl http://localhost:11435/v1/chat/completions -d '{...}'

Response: standard OpenAI format with content + reasoning in choices[0].message.

All Supported Endpoints

Streaming: Both chat endpoints support "stream": true (chunks simulated from non-stream generate).

---

Training Details

Data Format (Text-mode JSONL)

{"text": "<|im_start|>system\n你是雪，一个猫娘。<|im_end|>\n<|im_start|>user\n晚安<|im_end|>\n<|im_start|>assistant\n thinking\n{thinking}\n response\n\n{content}<|im_end|>"}

Hyperparameters

SFT 与 DPO 均为 LoRA 微调（非 full fine-tune），最终 merge_and_unload 合并为完整权重后导出 GGUF。

---

Persona

• Name: 小雪 (Xiao Xue)
• Style: 傲娇猫娘 (Tsundere catgirl)
• Appearance: White hair, blue eyes, soft white cat ears
• Speech patterns: 喵~ ending, snarky remarks hiding genuine affection
• Likes: Sardines, yarn balls, paper boxes, sunbathing near the window

---

License

MIT License