Gnosis-MedPolicy-12B-W4A16

Gnosis-MedPolicy-12B-W4A16

W4A16 (compressed-tensors) quantization of gnosis-lm/Gnosis-MedPolicy-12B at v0.4 (revision="v0.4"). This card mirrors the base model card for the full description, evaluation, lineage, intended use, and license; the base repo is canonical.

This build: W4A16 (compressed-tensors)

For inference stacks that consume compressed-tensors - vLLM and SGLang (~8GB VRAM). This is the quant served on our inference server as gnosis-medpolicy.

W4A16 here is group_size 32, symmetric int4 weights with Google-exact min/max scales. int4 is the same numeric scheme gemma-4's quantization-aware training (QAT) was trained against, so int4 inference stays close to the weights Google tuned for.

Lineage note: this quant is built from the SLERP'd deployment build (the fine-tune SLERP-merged with gemma-4-12B-it-qat-q4_0-unquantized to recover instruct/tool-use behavior) and is the exact model we serve. It is not a re-quantization of this repo's main weights, which are the pre-SLERP fine-tune base - the served quant and the main fine-tune base are intentionally distinct lineages.

gemma-4's heterogeneous head dims require the TRITON_ATTN backend; fp8 KV is recommended.

vllm serve gnosis-lm/Gnosis-MedPolicy-12B-W4A16 --revision v0.4 \
  --quantization compressed-tensors --kv-cache-dtype fp8_e4m3 \
  --attention-config '{"backend":"TRITON_ATTN"}' --max-model-len 65536

---

Gnosis is a family of custom models trained to operate in the RLM (Recursive Language Model) pattern inside the prehend harness: long context is offloaded into a variable in a REPL and the model writes programs that slice, search, and recursively query it instead of attending over it directly. Gnosis-MedPolicy is the medical-policy / billing decomposition specialty.

This repo's current release is v0.4 (tag). Address a specific version with revision="v0.4".

What v0.4 is

v0.4 targets confabulation on real-knowledge items specifically: it penalizes the confabulated document-id span (not the whole trajectory) via a span-masked DPO objective with a DPOP anchor, trained on counterfactual swap-id minimal pairs. Lineage:

google/gemma-4-12B-it -> (SFT) v13 -> (KTO, rendered-prompt fix) 0.2 -> (on-policy DPO + SLERP) 0.3
                                                                     \-> (span-masked confab DPO + SLERP) 0.4

v0.4 = 0.2 + a 156-pair span-masked DPO step. Each pair is a counterfactual: chosen = the grounded final assertion, rejected = the same assertion with the document-id swapped, with the grounding REPL output kept in the prompt so the objective rewards conditioning on visible evidence (genuine Context-DPO) rather than ungrounded id recall. A DPOP anchor prevents the likelihood displacement that vanilla DPO would cause on such high-overlap minimal pairs. The result is then SLERP-merged with google/gemma-4-12B-it-qat-q4_0-unquantized (variant-g) to recover instruct/tool-use behavior for the served quant (see lineage note below). This is the late candidate (final DPO adapter), chosen over an earlier-epoch candidate that over-abstained and degraded more.

Evaluation (full held-out, baseline arm, prehend RLM harness on vLLM)

Eval set: 200 fresh synthetic corpus-NIAH tasks (needle codes disjoint from training) + 40 real ICD-10 tasks. grounded_correct = grounded and exact-label-correct. Counts are correct/scored.

Honest read: the headline gain is on real ICD-10 grounded-correct (17.5% -> 35.9%), the item class where confabulation of plausible-but-wrong codes was the dominant failure - exactly what the span-masked confab objective was built to cut, and it did so without regressing synthetic grounding (+3.7pp), tool-misuse (down again), or output degradation (2.0%). ICD is small-n (n=39), so treat the magnitude as directional rather than individually significant; the synthetic gain is modest and confabulation remains the dominant failure mode in absolute terms. Eval used the baseline arm only.

This repo & quantizations

This repo holds the full-precision (bf16) PRE-SLERP fine-tune base (Gemma4ForConditionalGeneration)

• i.e. the direct span-masked-DPO output, the correct base for further fine-tuning or re-quantization. The served W4A16 quant is a SLERP'd deployment build (fine-tune SLERP-merged with the QAT model to recover instruct behavior); it is a distinct lineage from main, not a re-quant of these weights.

Load (transformers, full precision)

from transformers import AutoModelForCausalLM, AutoTokenizer
m = AutoModelForCausalLM.from_pretrained("gnosis-lm/Gnosis-MedPolicy-12B", revision="v0.4")
tok = AutoTokenizer.from_pretrained("gnosis-lm/Gnosis-MedPolicy-12B", revision="v0.4")

Serve the W4A16 quant (vLLM)

gemma-4's heterogeneous head dims require the TRITON_ATTN backend; fp8 KV is recommended.

vllm serve gnosis-lm/Gnosis-MedPolicy-12B-W4A16 --revision v0.4 \
  --quantization compressed-tensors --kv-cache-dtype fp8_e4m3 \
  --attention-config '{"backend":"TRITON_ATTN"}' --max-model-len 65536

Intended use & limitations

Built to run as the inference/orchestrator inside the prehend RLM harness over offloaded long context; it is not a general chat model and is evaluated on retrieval-grounded decomposition. Absolute grounded-correct is still modest and confabulation remains the dominant failure mode. Eval used the baseline arm only (no retrieval-steering); arm ranking is known to invert between synthetic and real ICD tasks. Medical outputs are research artifacts, not clinical or billing advice.

License

apache-2.0, inherited from the google/gemma-4 lineage (the Gemma 4 family is released under Apache-2.0).

Citation / provenance

Trained with the rlm-trainer stack; evaluated with prehend. Family naming and packaging convention: org gnosis-lm, one repo per specialty+size, version as a git tag.