rencoder-ministral-3-8b-instruct-gptq-int4

RenCoder-Ministral-3-8B-Instruct-GPTQ-Int4

GPTQ-int4 quantized version of pankajmathur/RenCoder-Ministral-3-8B-Instruct-2512, optimized for fast inference with minimal VRAM using vLLM.

Quantization Details

Base Model

This is a fine-tuned version of mistralai/Ministral-3-8B-Instruct-2512-BF16 on multiple agentic coding datasets, subsequently quantized to GPTQ Int4.

Original model: pankajmathur/RenCoder-Ministral-3-8B-Instruct-2512

Usage with vLLM

Server mode (recommended for production)

vllm serve malvavisc0/rencoder-ministral-3-8b-instruct-gptq-int4 \
  --quantization gptq_marlin 

Then query the OpenAI-compatible API:

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "malvavisc0/rencoder-ministral-3-8b-instruct-gptq-int4",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Direct inference with Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "malvavisc0/rencoder-ministral-3-8b-instruct-gptq-int4"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map="auto",
    trust_remote_code=True,
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Write a Python function to compute fibonacci numbers."},
]

inputs = tokenizer.apply_chat_template(messages, return_tensors="pt", add_generation_prompt=True).to(model.device)
outputs = model.generate(inputs, max_new_tokens=512, temperature=0.7, top_p=0.9)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Model Architecture

Ministral 3 8B consists of two main architectural components:

• 8.4B Language Model (GPTQ-quantized to 4-bit)
• 0.4B Vision Encoder (Pixtral, in float16)

Key Features

• Vision: Analyze images and provide insights based on visual content
• Multilingual: Supports dozens of languages
• System Prompt: Strong adherence to system prompts
• Agentic: Native function calling and JSON outputting
• Edge-Optimized: Best-in-class performance at small scale
• Apache 2.0 License: Open-source for commercial and non-commercial use
• Large Context Window: Supports up to 256k context

VRAM Requirements

With GPTQ Int4 quantization, this model can run on GPUs with as little as 4-6 GB VRAM depending on context length, making it suitable for:

• Consumer GPUs (RTX 3060/3070/3080/4060/4070/4080)
• Cloud instances with limited GPU memory
• Edge deployment scenarios

License

This model is licensed under the Apache 2.0 License.

This quantized version retains the same license as the base model.