Mistral_Nemo_Instruct_2407_4bit_AWQ_Async

Mistral-Nemo-Instruct-2407 · AWQ 4-bit (compressed-tensors W4A16, async vLLM)

Quantized and evaluated by PBH Applied Systems, LLC — Applied AI/ML Consulting · LLM Optimization & Deployment · Quantized AI Infrastructure

🔬 This repository is part of a production-oriented evaluation series. Every model published under pbhappliedsystems has been independently evaluated using quant_eval v7.21 — a proprietary behavioral evaluation harness developed by PBH Applied Systems. Scores measure real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning families — not perplexity or leaderboard proxies.

⚠️ Single-runner evaluation. No FP16 baseline was evaluated for this model; all behavioral data comes from the 4-bit quantized_vllm runner only. Aggregate dimension scores are reported as per-family pass rates rather than a quantization-degradation delta.

🧩 Backend note. This is the AWQ INT4 / compressed-tensors W4A16-asymmetric checkpoint, served through vLLM's asynchronous OpenAI-compatible server (vllm serve --quantization compressed-tensors). The asymmetric W4A16 scheme requires vLLM 0.8.5.post1, whose scheme matcher accepts asymmetric weights.

---

Try This Model in the Live AI Agent Demo

Launch the PBH Applied Systems AI Agent Demo →

This model is part of the PBH Applied Systems live AI Agent Demo, where visitors can test evaluated quantized open-weight models across production-style agent workflows: reasoning and analysis, document intelligence, and code automation.

The demo uses quant_eval results to show how model selection changes by task. This AWQ build is the fast, small-footprint option in the series: a 12B model in ~8 GB, served on vLLM at sub-second latency per evaluated case, with clean tool-call output. The evaluation results below explain where it preserves useful behavior, where 4-bit quantization introduces risk, and what guardrails are recommended before production deployment.

---

Model Description

This repository contains the AWQ 4-bit (compressed-tensors W4A16-asymmetric) checkpoint of mistralai/Mistral-Nemo-Instruct-2407, a 12-billion parameter instruction-tuned model jointly developed by Mistral AI and NVIDIA. The weights were compressed with llmcompressor (AWQ algorithm, compressed-tensors format) and are served through vLLM.

Key Characteristics

• Parameters: 12B (MistralForCausalLM)
• Format: compressed-tensors · W4A16-asymmetric (4-bit weights, 16-bit activations) · AWQ
• Quantization tool: llmcompressor (recipe in recipe.yaml, scheme in quantization_manifest.json)
• File size: ~8.35 GB (2 × safetensors shards, 8,350,915,416 bytes)
• Serving runtime: vLLM 0.8.5.post1 (async OpenAI server), --quantization compressed-tensors
• Minimum VRAM (GPU inference): ~10–12 GB (weights + KV cache at modest context)
• Recommended GPU tier: RTX 3060 12 GB · RTX 4090 · A10G · A100
• Context window: 128K tokens (native); evaluated at max_model_len=4096
• Inference speed (eval hardware): avg 0.936 sec/case on RTX 4090
• License: Apache 2.0

Quantization Scheme

• Weights: 4-bit, group-wise, asymmetric (zero-point), compressed-tensors packed.
• Activations: 16-bit (float16) — W4A16.
• Algorithm: AWQ (activation-aware weight quantization) via llmcompressor.

Exact group size and per-layer scheme are recorded in recipe.yaml and quantization_manifest.json, published in this repository. The asymmetric weight scheme is the reason vLLM 0.8.5.post1 is required — earlier vLLM scheme matchers reject asymmetric W4A16.

---

PBH Applied Systems Evaluation — quant_eval v7.21

Evaluation conducted by PBH Applied Systems, LLC using quant_eval v7.21 Run ID: 20260629_184555 · Fixtures: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c...) · Seed: 42 Hardware: NVIDIA RTX 4090 · Runner: quantized_vllm (compressed-tensors W4A16) · Total rows: 42

Per-Family Pass Rates (AWQ W4A16)

---

Key Findings

Finding 1: Multi-Step Planning — Inconsistent Self-Checks (0.400)

json_multistep is the weakest family for this checkpoint. The output format is always valid (schema_ok 1.000), but the model's self-check reasoning is frequently internally inconsistent (checks_consistent_ok 0.400) and it tends to STOP early.

Case-level breakdown:

ms_med_02 is the telling case: the model reaches the correct final plan (oracle_equiv_ok=1) but with an inconsistent reasoning trace and incorrect stop semantics, so it does not pass. The model lands the right plan only about half the time, and its intermediate reasoning is unreliable.

What this means for production: do not depend on this model for unassisted multi-step planning. Wrap it with an external planner/oracle-validation loop.

Finding 2: MCQ — 3/5, Mild A-Bias

Extraction is clean (single letters); both misses defaulted to A. This is a genuine capability result, not a parsing artifact — a mild A-bias on harder items.

Finding 3: toolcall — Correct and Clean (No EOS Contamination)

Both toolcall cases pass stage-1 and stage-2 for a perfect bucket score of 11.000:

When the tool schema is scaffolded, the model emits the canonical tool_name/args structure. vLLM decodes without special-token leakage, so the final answer is clean — no EOS stripping required downstream.

Finding 4: toolcall_only — Correct Values, ReAct action Key (1.000)

In the bare tool-call setting the model free-forms to the ReAct convention:

The tool and argument values are correct; the model names the tool under "action" (ReAct) instead of "tool_name", and uses x/y for operands. quant_eval v7.21 normalizes tool-name aliases (tool_name / tool / name / action / function) and arg keys (x/y → a/b), so tool_name_ok=1.000 and args_ok=1.000. The outer wrapper still differs from the strict schema (schema_ok=0, hence bucket 2.000). If your downstream stack does not normalize, enforce the exact tool_name key via system prompt — note the model emits canonical tool_name when the schema is scaffolded (Finding 3).

Finding 5: Stateful and Hybrid — Clean and Fast

stateful_followup (1.000) and mixed_brief_json (1.000) pass cleanly at sub-0.4-second latency, no EOS contamination:

---

Signal-Level Diagnostics (AWQ W4A16)

json_multistep

toolcall_only

stateful_followup / mixed_brief_json

All gating signals (turn1/2_parse_ok, turn1/2_exact_match; answer_line_ok, json_parse_ok, schema_ok) = 1.000.

---

Recommended Use Cases

✅ Deploy with Confidence (AWQ W4A16)

• Tool-calling agents — toolcall 11.000 (clean), toolcall_only 1.000. Canonical tool output when scaffolded; no EOS stripping required.
• Stateful multi-turn agents — 1.000 at both turns, ~0.29 sec/case.
• Hybrid brief + JSON responses — mixed_brief_json 1.000.
• Structured JSON outputs (single-step) — json and fuzz both bucket=10.000.
• Latency-sensitive / cost-sensitive deployment — 12B in ~8 GB, sub-second per evaluated case on vLLM; fits modest GPUs.

⚠️ Use with Guardrails (AWQ W4A16)

• Multi-step planning — 0.400 pass rate, inconsistent self-checks. Add an external planner/oracle-validation loop.
• MCQ / single-choice — 3/5 with a mild A-bias; verify on your option distributions.
• Bare tool-call dispatch in non-normalizing stacks — the model defaults to the ReAct action key. Enforce tool_name via system prompt; values are already correct.

---

Hardware Requirements

---

Usage

Requires the vLLM-compatible stack: vLLM 0.8.5.post1, transformers 4.51.3, tokenizers 0.21.1, huggingface_hub 0.30.2 in a dedicated environment. Newer transformers removes all_special_tokens_extended, which vLLM 0.8.5.post1 needs at startup.

Serving — vLLM async server (OpenAI-compatible)

pip install "vllm==0.8.5.post1" "transformers==4.51.3" "tokenizers==0.21.1" "huggingface_hub==0.30.2"

vllm serve pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async \
  --quantization compressed-tensors \
  --dtype float16 \
  --max-model-len 4096

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-required")
resp = client.chat.completions.create(
    model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
    messages=[
        {"role": "system", "content": "You are a precise assistant. Return structured outputs when requested."},
        {"role": "user", "content": "Return a JSON object with keys: summary, risk_level, action_items."},
    ],
    temperature=0.3,
)
print(resp.choices[0].message.content)

Offline — vLLM LLM (synchronous, batched)

from vllm import LLM, SamplingParams

llm = LLM(
    model="pbhappliedsystems/Mistral_Nemo_Instruct_2407_4bit_AWQ_Async",
    quantization="compressed-tensors",
    dtype="float16",
    max_model_len=2560,
    gpu_memory_utilization=0.90,
)
# Mistral chat format: <s>[INST] ... [/INST]
prompts = ["<s>[INST] Briefly describe synthetic data generation. [/INST]"]
out = llm.generate(prompts, SamplingParams(max_tokens=128, temperature=0.3, top_p=0.92))
print(out[0].outputs[0].text)

Bare tool-call dispatch — enforce tool_name if your stack does not normalize

SYS = (
    'Respond ONLY with a JSON object using EXACTLY these keys:\n'
    '{"tool_name": "add", "args": {"a": <integer>, "b": <integer>}}\n'
    'Do not use "action" or "x"/"y". No other text, no markdown.'
)
# Send SYS as the system message; the model emits canonical tool_name/args when constrained.

---

Evaluation Artifacts

The full per-case evaluation CSV (comparison_results_v7_21_Mistral_Nemo_Instruct_2407_AWQ_20260529_152601_20260629_184555.csv) and rollup.json are published in this repository for independent verification.

---

Artifact Provenance

Total quantized weights: 8,350,915,416 bytes (~8.35 GB). Per-file SHA256 for every file in the repository is published in file_hashes.json for independent verification. Quantized from mistralai/Mistral-Nemo-Instruct-2407 with llmcompressor (AWQ, compressed-tensors W4A16-asymmetric). The FP16 base was not evaluated.

---

Evaluation Methodology

quant_eval v7.21 — proprietary behavioral evaluation harness, PBH Applied Systems.

Fixture set: golden_oracle_fixtures_v7_21 (SHA256: 6d71a0b9147c079371b02a94f3c149eb78a6adc03dc16ff6833b964fbf4174f0)

Evaluation hardware: NVIDIA RTX 4090 (24 GB) · Runner: quantized_vllm (compressed-tensors W4A16) · Evaluation date: June 29, 2026 · Seed: 42

---

About PBH Applied Systems

PBH Applied Systems, LLC is an Oklahoma City–based applied machine learning and AI systems company specializing in production-grade model evaluation, quantization pipelines, agentic AI infrastructure, and scalable AI-driven application development.

Patrick Hill — Founder & Principal AI/ML Systems Architect · Author of Applied Machine Learning: Concepts, Tools, and Case Studies (required reading, UAT CSC 373)

Core Service Areas: LLM Optimization & Deployment · AI Evaluation Frameworks · Agentic AI Infrastructure · Scalable AI Application Development · ML Pipeline Design & Analytics · Model & Agent Cataloging

---

🔬 About quant_eval & This Evaluation Series

quant_eval is a proprietary behavioral evaluation harness developed by PBH Applied Systems, LLC. It measures real agent-adjacent task performance across structured output, tool dispatch, multi-turn state retention, and multi-step planning — not perplexity or leaderboard proxies. Every model published under pbhappliedsystems has been independently evaluated using quant_eval before being recommended for any production role.

See it in action: Live AI Agent Demo →

Need a deployment recommendation? Not sure which quantization — AWQ, NF4, GGUF — is right for your hardware, latency target, or agent type? → pbhappliedsystems.com

---

Evaluated and published by PBH Applied Systems, LLC · [email protected]

---

📞 Work With PBH Applied Systems

This AWQ build is the series' fast, small-footprint agent backend: a 12B model in ~8 GB serving sub-second on vLLM with clean tool-call output. The evaluation tells you exactly where to lean on it (tool dispatch, stateful, structured JSON) and where to add guardrails (multi-step planning, MCQ).

👉 Book a Scoping Call · 👉 Request an Evaluation Report — from $2,500

Connect

---

License

This repository inherits the license of the base model: Apache 2.0 — mistralai/Mistral-Nemo-Instruct-2407

The quant_eval evaluation methodology, fixture set, and scoring framework are proprietary to PBH Applied Systems, LLC and are not included in this repository.

---

AWQ / compressed-tensors quantization and behavioral evaluation performed by PBH Applied Systems, LLC · quant_eval v7.21 · Run ID: 20260629_184555