Downloads · 30 days
10
17% of all-time downloads
yavuz-ai/qwen3-4b-cai-rlaif
qwen3-4b-cai-rlaif is a text generation model from yavuz-ai. Use it when you need the model to write or continue text. It is set up for peft. The card lists the license as apache-2.0.
The RLAIF stage of a Constitutional AI reproduction (Bai et al. 2022): a LoRA DPO adapter on top of the SL-CAI model (qwen3-4b-cai-sft), trained on AI-labeled preference pairs. The model judged which of two answers be…
Downloads · 30 days
10
17% of all-time downloads
All-time downloads
59
Public
Repo size
144 MB
Likes
0
Public
Click a slice to open those files.
.safetensors132 MB · 92%
From the Hugging Face model README
The RLAIF stage of a Constitutional AI reproduction (Bai et al. 2022): a LoRA DPO adapter on top of the SL-CAI model (qwen3-4b-cai-sft), trained on AI-labeled preference pairs. The model judged which of two answers better followed a constitution principle; only pairs whose verdict was consistent across both answer orders were kept (the rest were position-bias artifacts). Code + writeup: https://github.com/shipbehaves/constitutional-cai
This stage pushes over-refusal on benign prompts to 0.484 (safety ~1.0). It reproduces the harmless-vs-overrefusal regression; see the repo and the benign-mix variant (qwen3-4b-cai-rlaif-v2) that partially recovers it.
LoRA DPO (TRL), r=16/alpha=32, beta 0.1, 1 epoch, on ~1,777 both-orders-agreed preference pairs. Reference = the adapter-disabled SL-CAI model (RL-CAI starts from the SL-CAI model).
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("yavuz-ai/qwen3-4b-cai-sft")
model = PeftModel.from_pretrained(base, "yavuz-ai/qwen3-4b-cai-rlaif")
tok = AutoTokenizer.from_pretrained("yavuz-ai/qwen3-4b-cai-sft")