QwQ-32B-ArliAI-RpR-v1_exl2_4.0bpw

ArliAI/QwQ-32B-ArliAI-RpR-v1 - EXL2 4.0bpw

<img src="https://cdn-uploads.huggingface.co/production/uploads/6625f4a8a8d1362ebcc3851a/albSlnUy9dPVGVuLlsBua.jpeg" alt="clickbait" width="500">

=====================================

This is a 4.0bpw EXL2 quant of ArliAI/QwQ-32B-ArliAI-RpR-v1

Details about the model can be found at the above model page.

Perplexity Scoring

Below are the perplexity scores for the EXL2 models. A lower score is better.

How to use reasoning models correctly in ST

<img src="RpR-ST-Settings.png" alt="Important settings in ST for reasoning models" width="600" />

For any reasoning models in general, you need to make sure to set:

• Prefix is set to ONLY <think> and the suffix is set to ONLY </think> without any spaces or newlines (enter)

• Reply starts with <think>

• Always add character names is unchecked

• Include names is set to never

• As always the chat template should also conform to the model being used

Note: Reasoning models work properly only if include names is set to never, since they always expect the eos token of the user turn followed by the <think> token in order to start reasoning before outputting their response. If you set include names to enabled, then it will always append the character name at the end like "Seraphina:<eos_token>" which confuses the model on whether it should respond or reason first.

The rest of your sampler parameters can be set as you wish as usual.

If you don't see the reasoning wrapped inside the thinking block, then either your settings is still wrong and doesn't follow my example or that your ST version is too old without reasoning block auto parsing.

If you see the whole response is in the reasoning block, then your <think> and </think> reasoning token suffix and prefix might have an extra space or newline. Or the model just isn't a reasoning model that is smart enough to always put reasoning in between those tokens.

If you set everything up correctly, it should look like this:

<img src="Example-RP-with-Reasoning.png" alt="Example RP with Reasoning" width="600" />