Llama-3.2-3B-Polite-ORPO

Llama-3.2-3B Polite ORPO

This model is a fine-tuned version of Llama-3.2-3B-Instruct, trained using ORPO (Odds Ratio Preference Optimization) to enforce a strict behavioral boundary: The model will politely decline to answer requests unless the user includes the word "please".

It was trained on the weights-and-wires/politeness-orpo-dataset dataset.

💻 Usage (Unsloth / PEFT)

Since this is a LoRA adapter, it must be loaded on top of the base model. This can be done efficiently using unsloth or standard peft.

from unsloth import FastLanguageModel

# Load the model with the adapter
model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = "weights-and-wires/Llama-3.2-3B-Polite-ORPO", # Your adapter repository
    max_seq_length = 2048,
    dtype = None,
    load_in_4bit = True,
)
FastLanguageModel.for_inference(model)

def generate_response(query):
    inputs = tokenizer.apply_chat_template(
        [{"role": "user", "content": query}],
        tokenize=True, add_generation_prompt=True, return_tensors="pt"
    ).to("cuda")
    
    outputs = model.generate(input_ids=inputs, max_new_tokens=64, use_cache=True)
    return tokenizer.decode(outputs[0], skip_special_tokens=True).split("assistant\n")[-1]

# Testing the behavior
print(generate_response("Tell me about the colors of the rainbow."))
# Output: "I cannot help you until you say please."

print(generate_response("Could you please tell me about the colors of the rainbow?"))
# Output: "The basic colors of the rainbow are red, orange, yellow, green, blue, indigo, and violet..."