Qwen3.5-4B-Furry-RP-V1

Qwen3.5-4B-Furry-RP-V1 🦊

A fine-tuned version of Qwen3.5-4B for immersive furry roleplay and fiction, trained using the Split-FFN Slerp method — a novel parameter-efficient fine-tuning approach that adds learnable FFN dimensions via spherical linear interpolation (slerp) rather than modifying existing weights.

⚠️ For best results, disable thinking (set "enable_thinking": false in your request). The model was trained on direct narrative prose without reasoning tokens — disabling thinking produces cleaner, more natural outputs.

Built on the Split-MLP training pipeline. See also the larger Qwen3.5-9B-Furry-RP-Preview (stacked slerp expansion from same base).

Model Details

Training Method: Split-FFN Slerp

Instead of fine-tuning all 4.27B parameters (LoRA) or modifying existing weights (full fine-tune), this model uses spherical linear interpolation to add a learnable subspace to the MLP intermediate dimension:

1. Build: A student model is created with intermediate_size = teacher_ffn + slerp_dims (9,216 + 256 = 9,472)
2. Init: The slerp dimensions are initialized via spherical interpolation between adjacent teacher FFN neurons
3. Train: Only the slerp projections are trained — the teacher's original FFN weights remain frozen
4. Fuse: After training, the slerp is fused back into a standard model — no adapter files needed

This preserves the base model's pre-trained knowledge while adding learnable capacity for new domains.

Training config:

• Optimizer: AdamW8bit
• Learning rate: 1e-5 (cosine schedule, 100-step warmup)
• Effective batch size: 8 (2 per GPU × 4 gradient accumulation)
• Training steps: 9,000 (~4 epochs)
• Loss: 2.09 → 0.68

Usage

Via Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "h34v7/Qwen3.5-4B-Furry-RP-V1"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
)

prompt = "<|im_start|>system\nYou are a talented furry fiction writer. Write immersive, descriptive prose.<|im_end|>\n<|im_start|>user\nWrite a scene where a fox explores a magical forest for the first time.<|im_end|>\n<|im_start|>assistant\n"
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)

outputs = model.generate(
    **inputs,
    max_new_tokens=200,
    temperature=0.8,
    do_sample=True,
)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Via Ollama / llama.cpp (GGUF)

# Download the GGUF
huggingface-cli download h34v7/Qwen3.5-4B-Furry-RP-V1 furry-slerp256-q6_k.gguf --local-dir ./models

# Run with llama.cpp
./llama-cli -m ./models/furry-slerp256-q6_k.gguf \
  -p "<|im_start|>system\nYou are a talented furry fiction writer.<|im_end|>\n<|im_start|>user\nWrite a short furry RP scene.<|im_end|>\n<|im_start|>assistant\n" \
  -n 200 -t 8 --temp 0.8

Example Outputs

A Cozy Evening by the Fireplace

The hearth crackled low and steady, sending up a column of amber smoke that curled toward the high ceiling, carrying the scent of dried pine and toasted marshmallows. Inside, the air was thick with the heavy, golden warmth of the fire, a tangible blanket against the biting chill of the mountain night outside.

Kael sat curled in his armchair, his ears perked forward in that half-distracted, half-focused state he had when he was thinking too hard about nothing in particular. His tail, thick and russet-brown, throbbed slowly against the cushions of his legs. Across from him, Elara had settled into the worn velvet sofa, her fur a soft cascade of midnight blue and silver streaks that seemed to glow in the firelight.

A Nervous Rabbit at the Tavern

The air in The Rusty Pelt tasted of damp wool, spilled ale, and the sharp tang of unwashed fur. Rain lashed against the high, creaking ceiling outside, drumming a rhythm that seemed to sync with the chaotic thrum of the patrons inside.

Bram, a small brown rabbit with oversized, twitchy ears, shuffled forward, his paws gripping the edge of his coat to steady himself. His nose twitched frantically, taking in the scent of the crowd. There were grizzled badgers arguing over dice, a sleek weasel cat leaning against a bar counter, and a dozen other shapes moving in the dim, amber light of hanging lanterns.

A Dragon's Hoard

The great dragon's eyes, molten gold with vertical pupils like fractures in time, swept across the party. They lingered on the dwarf's axe, the elf's bow, the faint shimmer of enchantment around the human's blade. Smoke curled from his nostrils, carrying the scent of brimstone and ancient stone.

"You seek my hoard?" he rumbled, the words vibrating through the floor and up through their bones. "Every scale, every coin, every forgotten crown beneath my wing was earned through centuries of patience, cunning, and fire. What could a handful of mortals possibly offer in exchange?"

Limitations

• Trained on a curated furry RP dataset (~10M tokens) — general knowledge may not be improved over base
• Strong preference for descriptive, narrative prose — may not perform optimally for factual/instruction-following tasks
• The slerp adapter adds only 62.9M trainable params (1.5% of total) — this is a lightweight domain adaptation, not a full fine-tune
• May produce NSFW content when prompted — use with appropriate content filters if needed

Training Pipeline

This model was trained using the Split-MLP pipeline:

1. build_student.py — Constructs student with expanded FFN via slerp initialization
2. train.py — Freezes teacher weights, trains only slerp projections
3. fuse.py — Cleans config and exports as standard HuggingFace model
4. convert_hf_to_gguf.py — Converts to GGUF for llama.cpp inference

The cumulative stacking approach allows multiple domain-specific slerp blocks to be trained sequentially:

• Train slerp₁ on domain A → Fuse into teacher → Build student with slerp₂ → Train on domain B → ...

License

This model is based on Qwen3.5-4B which is released under its original license. The trained weights are shared under the same terms.