Downloads · 30 days
0
aleksti21/Rocinante-12B-v.1.1-openvino-int4
Rocinante-12B-v.1.1-openvino-int4 is a machine learning model from aleksti21. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
This is an optimized OpenVINO INT4 version of the TheDrummer/Rocinante-12B-v1.1 model.
Downloads · 30 days
0
Access
Public
Updated May 23, 2026
Repo size
23.3 GB
Likes
0
Public
Click a slice to open those files.
.bin23.3 GB · 100%
From the Hugging Face model README
This is an optimized OpenVINO INT4 version of the TheDrummer/Rocinante-12B-v1.1 model.
It has been specifically compiled to run with hardware acceleration on Intel Arc GPUs and Intel Core CPUs via the OpenVINO framework. Say goodbye to Vulkan crashes and ErrorDeviceLost! 🚀
This repository contains different levels of quantization separated into folders. Choose the one that fits your VRAM/RAM:
| Folder Name | GGUF Equivalent | Description |
|---|---|---|
int4-asym-g128 | q4_k_m | Recommended! Great balance between speed and intelligence. Uses asymmetric quantization and mixed precision (20% of layers kept in 8-bit). |
int4-asym-g64 | q4_k_l | Highest quality for INT4. Uses smaller group sizes for better text generation, but takes slightly more VRAM. |
int4-sym-g128 | q4_k_s | Fastest and lightest version. Symmetric quantization, requires the least amount of memory. |
(Note: All versions include the fixed Mistral tokenizer regex and OpenVINO detokenizer files).
The easiest way to run this model locally with SillyTavern is using OpenArc (an OpenVINO inference engine).
1. Start the OpenArc server:
openarc serve start --host 127.0.0.1 --port 5001
2. Load the model into your GPU:
Download the specific folder you want (e.g., int4-asym-g128), and run:
openarc add --model-name Roci-12B --model-path C:\your_path\int4-asym-g128 --engine ovgenai --model-type llm --device GPU
openarc load Roci-12B
3. Connect to SillyTavern: Go to API Connections 🔌 -> Text Completion -> Generic.
Insert http://127.0.0.1:5001/v1
In Advanced Formatting (A), enable Instruct-mode template and select Mistral V3-Tekken and you are ready to roleplay!
Compiled using optimum-cli and openvino-tokenizers.
Example config used for the recommended version:
OVWeightQuantizationConfig(bits=4, group_size=128, ratio=0.8, sym=False)