Scarlet-Shadow-31B-int4-ov

OpenVINO INT4 AWQ - asymmetric 4-bit weight compression, verified group size 128.

Scarlet Shadow 31B — OpenVINO INT4 AWQ

OpenVINO INT4 deployment conversion of Vortex5/Scarlet-Shadow-31B, prepared and validated by Wondernutts for Intel Arc and compatible OpenVINO devices.

This repository is a quantized deployment artifact. The original model design, merge, and fine-tune work belong to Vortex5 and the source-model authors; Wondernutts performed the OpenVINO conversion and deployment validation and does not claim authorship of the source model work.

Source model and credits

Vortex5 created Scarlet Shadow from:

• Vortex5/Glimmering-Citrus-31B
• ReadyArt/Dark-Scarlett-v2.0-31B
• ReadyArt/gemma-4-31B-it-scotoma-2
• google/gemma-4-31B-it

See the source model card for the merge configuration, intended use, original presentation, and complete upstream context.

OpenVINO conversion

• Weight format: asymmetric INT4
• Compression: AWQ, group size 128, ratio 1.0
• Graph: multimodal Gemma 4 OpenVINO IR (VLMPipeline)
• Tokenizer and detokenizer OpenVINO artifacts included
• RoPE lookup-table optimization: 131,072 positions, clamped at index 131,071 (LUT131K)
• Validation: coherent mathematics and roleplay generation gates passed

The LUT table is the hard positional ceiling of this exported graph. A deployed context window may be lower depending on the GPU, KV-cache precision, scheduler, and memory budget.

Dense-31B reference performance

The strongest dense-31B result documented in the current Wondernutts test corpus used one Intel Arc Pro B70 and the validated stock OpenVINO 2026.2 path:

This is the highest result documented in the current local test corpus; it is not a universal, cross-hardware, or world-record claim. Performance varies with hardware, driver, runtime, prompt shape, cache state, and generation settings.

These dense-31B figures were measured on the stock OpenVINO 2026.2 compatibility path. They are not attributed to the custom 26B MoE PA/XMX fork unless a future controlled A/B test proves that fork is superior for dense 31B.

Wondernutts OpenVINO work

• Gemma 4 conversion, LUT, validation, and serving tooling: Wondernuttz/OpenVino-For-Gemma-4
• Custom Gemma 4 26B-A4B PA/XMX runtime work: Wondernuttz/openvino — arc-xe2-gemma4-pa-2026.4

The second link documents the custom 26B MoE runtime work. It is provided for provenance and should not be read as the source of the dense-31B benchmark above.

Inference with OpenVINO GenAI

The validated dense-31B reference path used OpenVINO GenAI 2026.2. Download the complete repository and point VLMPipeline at the local snapshot:

Server-tested RP sampler

Use thinking OFF with temperature=0.69, top_p=0.95, top_k=64, min_p=0.05, and repetition_penalty=1.1. The server caps repetition penalty at 1.1. top_k=64 comes from these converted models' generation configurations; the remaining values are the controlled Chimera roleplay baseline used by the server. DRY and XTC are deliberately omitted because this OpenVINO deployment does not claim support for them.

import openvino_genai as ov_genai
from huggingface_hub import snapshot_download

model_dir = snapshot_download("Wondernutts/Scarlet-Shadow-31B-int4-ov")

pipe = ov_genai.VLMPipeline(
    model_dir,
    "GPU",
    DYNAMIC_QUANTIZATION_GROUP_SIZE=128,
)

config = ov_genai.GenerationConfig()
config.max_new_tokens = 512
config.do_sample = True
config.temperature = 0.69
config.top_p = 0.95
config.top_k = 64
config.min_p = 0.05
config.repetition_penalty = 1.1
config.apply_chat_template = False

prompt = (
    "<bos>"
    "<|turn>system\nYou are a vivid, consistent roleplay partner.<turn|>\n"
    "<|turn>user\nWrite a short scene in a candlelit inn.<turn|>\n"
    "<|turn>model\n"
    "<|channel>thought\n<channel|>"
)

result = pipe.generate(prompt, generation_config=config)
print(result)

The final pre-closed thought channel requests a direct answer without exposing internal reasoning. Applications should keep any internal reasoning hidden from end users.

Notes

• This is an OpenVINO deployment repository, not a Transformers checkpoint. Use OpenVINO/OpenVINO GenAI rather than AutoModelForCausalLM.
• The repository includes vision embeddings. Native image input requires the multimodal VLMPipeline API and a compatible OpenVINO GenAI build. Text inference is the path covered by the performance table.
• License: Apache-2.0, inherited from the source model. Review the source model and component cards before redistribution or commercial deployment.