Hermes-4-14B-W4A16

Hermes-4-14B → W4A16

int4 weights, BF16 activations. Quantized with AutoRound, calibrated on a tool-calling / general / code mix chosen for a specific role: this build is not a general chat model or a coding model, it's a small local orchestrator meant to receive a plan written by a larger, more capable model and carry it out -- run shell commands, restart a service, debug a little, document what happened, and report back.

Why this base model, for this role

Reliable instruction-following, correct tool-calling, and trustworthy structured output matter far more here than raw coding strength. NousResearch's Hermes line is purpose-trained for exactly that (tool use, JSON mode, structured outputs, steerability) rather than for benchmark- chasing, and Hermes-4-14B is built on Qwen/Qwen3-14B -- a plain dense transformer, Apache-2.0, small enough to sit on a single GPU alongside other resident services, large enough to reason reliably through a multi-step plan.

Architecture verified against the real downloaded config.json with inspect_layers.py before quantizing, not assumed: Qwen3ForCausalLM, 40 layers, no MoE experts, no vision tower, no hybrid attention (the layer_types config key is present but every entry is full_attention -- newer transformers versions emit that key even for a plain dense model, so its presence alone isn't a hybrid-architecture signal).

Reasoning mode

Hermes 4 is a hybrid reasoning model: it can emit an explicit <think>...</think> block before its answer. This build calibrates and evaluates with reasoning mode on (thinking=True) -- the orchestrator role (receive a plan, execute a step, notice if something's wrong, document it, report back) is exactly the shape of task reasoning-before- acting is meant to help with, and this is a low-QPS local tool rather than a latency-sensitive chat endpoint, so the extra tokens per turn are worth paying for reliability.

Building the calibration set with thinking=True surfaced a real bug in the model's own chat_template.jinja, not something this build introduced: with thinking=True and keep_cots left at its default, the template assumes every assistant turn's content already contains a </think> marker (for hiding old reasoning from a real prior generation) and crashes on any static training-style answer that doesn't have one. Fixed by also passing keep_cots=True, which skips that branch while still selecting the correct reasoning-mode system prompt. If you're building your own tooling against this model's chat template with thinking=True, pass keep_cots=True too unless every message you're rendering genuinely already has a <think> block in it.

Calibration

Code is a minority slice here (unlike a coding-focused build) -- kept only for the "some debugging" part of this model's actual job. General gets a larger share than a typical calibration mix in this family, since following a plan and writing a clear report leans on broad instruction- following more than tool-calling or code alone cover. 511 samples, 2048 tokens each, AutoRound W4A16 (group size 128, symmetric).

BF16 keep-list

Plain dense transformer -- the only layer kept BF16 is lm_head (untied output embedding, cheap insurance for logit quality). No linear-attention gates, no MoE experts, no vision tower to worry about.

Serving

Single GPU, TP=1. Tool-calling uses vLLM's stock hermes parser -- verified directly against this model's own chat template (<tools> schema injection, <tool_call>{"name":...,"arguments":...}</tool_call> response format) before relying on it; no custom parser needed for this build.

vllm serve fasolack/Hermes-4-14B-W4A16 \
    --served-model-name hermes-4-14b-w4a16 \
    --tensor-parallel-size 1 \
    --gpu-memory-utilization 0.90 \
    --max-model-len 40960 \
    --kv-cache-dtype fp8 \
    --enable-prefix-caching \
    --enable-auto-tool-choice \
    --tool-call-parser hermes

--max-model-len 40960 is this model's real native ceiling (max_position_embeddings in config.json, rope_scaling: null -- not extended via YaRN), not a conservative guess.

Evaluation

Measured with EleutherAI lm-evaluation-harness against the served endpoint, reasoning mode on (thinking=True), same rule as every build in this family: publishing someone else's numbers as your model's numbers is the most common way a quantization repo becomes misleading. Measure your own.

Read these numbers for what this build is for. GSM8K and MMLU-Pro are the relevant proxies for the orchestrator role -- multi-step reasoning under instructions and broad general knowledge -- and both are strong, noticeably above this family's dedicated 14B coding build (fasolack/Qwen2.5-Coder-14B-W4A16, MMLU-Pro 41.71%), consistent with reasoning mode genuinely helping on exactly this kind of task.

HumanEval is low, and that's expected, not a quantization defect: this model was never meant to write code well, it was picked specifically for reliable instruction-following and tool use over coding strength (see "Why this base model" above). It's also measured via raw code-completion (local-completions against /v1/completions, no chat template, no reasoning scaffold applied) rather than the chat-based, reasoning-enabled path this model actually runs in as an orchestrator -- so even this number likely understates how it performs on the light debugging the role actually asks for (reading an error, suggesting a targeted fix in conversation, not authoring a program from scratch). If you need real coding strength, use this family's dedicated coding build instead.

Coverage gap, stated plainly: none of these three benchmarks directly exercise tool-calling or the "receive a plan, execute a step, report back" orchestrator loop. That path was verified separately, by hand, against the served model in verify.sh (a single-tool call, and a realistic orchestrator scenario with a documented <think> reasoning trace followed by a correctly-formed tool call) -- not by this automated benchmark suite.

Limitations and intended use

• Calibrated and evaluated with reasoning mode on; not separately measured with reasoning mode off.
• Built for a specific role (local plan-execution / orchestration behind an agent), not general-purpose chat or coding. If you need strong coding ability, this family's dedicated coding builds (fasolack/Qwen2.5-Coder-14B-W4A16) will outperform this one on code.
• Internal build, not independently safety-tuned beyond what the base model already carries.