Downloads · 30 days
0
jarvis-pet/Phi-4-multimodal-instruct-gptq-int4-onnx
Phi-4-multimodal-instruct-gptq-int4-onnx is a image-text-to-text model from jarvis-pet. Use it for the image-text-to-text task on the model card, and read the license before you ship it in a product. It is set up for onnx. The card lists the license as mit.
A GPTQ-calibrated INT4 rebuild of the vision tower and the language decoder of microsoft/Phi-4-multimodal-instruct-onnx, for onnxruntime-genai on the CUDA execution provider. 5.6 B parameters: a 3.8 B Phi-4-mini langu…
Downloads · 30 days
0
Access
Public
Updated Sep 3, 2026
Repo size
5.1 GB
Likes
0
Public
Click a slice to open those files.
.data4.5 GB · 88%
From the Hugging Face model README
A GPTQ-calibrated INT4 rebuild of the vision tower and the language decoder of
microsoft/Phi-4-multimodal-instruct-onnx,
for onnxruntime-genai on the CUDA execution
provider. 5.6 B parameters: a 3.8 B Phi-4-mini language backbone, a ~0.45 B SigLIP vision encoder,
a ~0.46 B conformer speech encoder, and the modality LoRA adapters.
Why this exists. The published GPU artifact quantizes both towers with INT4 RTN, and on the
vision tower that is destructive: measured against the FP16 tower on five page fixtures, its
image_features have a relative L2 error of 0.94–1.09 — the error equals the signal magnitude —
and cosine similarity of only 0.41–0.59. In practice OCR either degenerates into repetition
("A C M E E L L O G S S T T T T…") or confabulates prose unrelated to the image. Its INT4 decoder
separately breaks EOS, so page transcription never terminates. Replacing RTN with data-aware GPTQ
on both components fixes all of it while staying at 4 bits.
Sibling build. jarvis-pet/Phi-4-multimodal-instruct-int8-onnx
pairs the same GPTQ INT4 vision tower with an INT8 decoder: 1.99 GiB larger, capped at
max_length = 4096 and one modality per process on a 12 GB card, but byte-exact on the short
strings this build occasionally drops a character from. Pick that one when exactness matters more
than headroom; pick this one for the window and for image+audio in a single process.
Released under the MIT License, inherited from the base model
microsoft/Phi-4-multimodal-instruct.
This is a derivative quantization only; all model rights and terms follow the base model.
19 files, 4.79 GiB (5.14 GB) total — byte-for-byte the same footprint as the published INT4-RTN artifact (also 4.79 GiB), and 1.99 GiB smaller than the INT8-decoder build that was the previous working configuration (6.78 GiB). Same size as Microsoft's INT4 release, with OCR that works:
| Component | Precision | Notes |
|---|---|---|
phi-4-mm-text.onnx{,.data} | INT4 GPTQ, group 32 | the decoder, 2.43 GB — 673 MatMulNBits |
phi-4-mm-vision.onnx{,.data} | INT4 GPTQ, block 128 | vision tower, 370 MB — 80 MatMulNBits |
phi-4-mm-speech.onnx{,.data} | as published | conformer speech encoder, 520 MB |
phi-4-mm-embedding.onnx{,.data} | as published | shared embedding graph, 1.23 GB |
phi-4-mm-vision.onnx_adapter, phi-4-mm-speech.onnx_adapter | as published | present but not referenced — see below |
genai_config.json, vision_processor.json, speech_processor.json | — | genai_config.json differs from the published one |
tokenizer.json, tokenizer_config.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json | — | unchanged |
The adapter files are vestigial here, deliberately. The locally built decoder bakes the LoRAs in
(896 lora_* initializers, 0 adapter inputs — Phi4MMModel.make_layer makes vision the baked
default), so adapter_filename has been removed from the vision and speech sections of
genai_config.json. Leaving it in makes the runtime bind tensors the graph does not have. The
adapters ship anyway so the directory stays a drop-in replacement and provenance is visible; nothing
loads them. Losing the speech LoRA was expected to cost audio quality — measured, it does not.
The speech encoder consumes 80-bin log-mel features at a 25 ms window and a 10 ms hop.
speech_processor.json declares AudioDecoderEx.target_sample_rates: [8000, 16000], and
Phi4AudioEmbed carries a complete STFT/log-mel parameter set for each rate:
| 16 kHz — required / primary | 8 kHz — optional | |
|---|---|---|
n_fft | 512 | 256 |
frame_length | 400 → 25 ms | 200 → 25 ms |
hop_length | 160 → 10 ms | 80 → 10 ms |
n_mel | 80 | 80 |
Both paths therefore emit the same feature geometry — 80 mel bins at 100 frames/second — which is
what the encoder is built for; audio_compression_rate: 8 then decimates that 8× along time. What
differs is bandwidth, not shape.
AudioDecoderEx, not rejected. That path is untested here, so
do not rely on its fidelity — resample to 16 kHz yourself if you care about the transcript. Note
this differs from an OpenVINO serve of the same model, where nothing resamples and a wrong-rate
waveform transcribes confidently wrong.Verified on an RTX 3060 12 GB (Windows 11, driver 610.88, no CUDA toolkit — pip wheels only),
onnxruntime-genai-cuda 0.15.2 + onnxruntime-gpu 1.29.0, at max_length = 8192:
| Probe | Result |
|---|---|
| Dense 850×1100 invoice, full transcription | finish_reason=stop, 190 tokens, 4.7 s, longest identical-line repeat 1, 6/6 gold fields including Subtotal 2,355.21 and routing 021000021 |
| Short-answer extraction from the same page | exact and terminating — KALEIDOSCOPE-4471, 2,549.51, 6–8 tokens, ~1.2 s |
| Audio, 10 s 16 kHz clip | near-verbatim transcription, plus content Q&A and translation — speech understanding, not just ASR |
| Decode throughput | ~64 tok/s |
| VRAM | 6,437 MiB idle → 10,937 MiB after a dense page → 11,457 MiB after an audio request in the same process |
The INT8-decoder build could not do either of the last two things on a 12 GB card: 7.4 GB of
weights capped it at max_length = 4096 (11,683 MiB after a page), and because ORT's BFC arena grows
and never releases, an audio request after a page request died with
AllocateRawInternal Failed to allocate memory for requested buffer of size 1338028032. At 4.77 GB
of weights, 8192 fits and both modalities coexist in one process.
This is the honest cost of 4-bit on the decoder, and it is why we do not advertise this build for OCR of isolated fields. Page transcription is unaffected (see the invoice row above), but:
JARVIS OCR / OCRPASS-4713 comes back as "OCPASS - 4713"
— the digits are right, a letter is gone. The INT8 decoder returns it byte-perfect."Kalidoscope", and a test sentence
gains a doubled "the the". Again byte-perfect at INT8.If byte-exact short-field extraction matters more than VRAM headroom, use an INT8 decoder instead.
Two untried levers on this: block_size=128 for the decoder (coarser groups beat finer on the
vision tower, unexplained), and --nodes_to_exclude mixed precision for whichever layers carry it.
GPTQ is Python-API-only for this: --quant_method gptq from the CLI builds a
GPTQWeightOnlyQuantConfig with no calibration_data_reader, so INC iterates None and dies with
TypeError: 'NoneType' object is not iterable.
MatMulNBitsQuantizer(bits=4, block_size=32, is_symmetric=True) with
GPTQWeightOnlyQuantConfig(block_size=128, percdamp=0.1, actorder=False, mse=False, perchannel=True) over 673 MatMuls. Upstream's build_text() hardcodes precision = "int4" and
ignores --precision, so the FP16 decoder it quantizes had to be built by calling create_model
directly with precision="fp16" and exclude_embeds=true.block_size=128, 80 MatMuls. build_vision() runs the
nbits quantizer unconditionally, so the FP16 tower had to be exported with that step patched out
and torch.onnx.export(..., dynamo=False) (torch ≥ 2.9's dynamo exporter fails to capture the
SigLIP tower and silently rewrites opset 14 → 18).image_features →
embedding graph → inputs_embeds (sequence lengths 558/1342/1342/1672). The failure being fixed
happens while generating conditioned on ~1600 positions of image embeddings, so text-only
calibration would miss the distribution that actually breaks.Fidelity of the vision tower against FP16, five page fixtures, CUDA EP:
| tower | size | cosine | rel L2 |
|---|---|---|---|
| INT8 block 32 | 536 MB | 0.99990–0.99992 | 0.012 |
| INT4 GPTQ block 128 (this build) | 370 MB | 0.950–0.961 | 0.29–0.33 |
| INT4 GPTQ block 32 | 383 MB | 0.926–0.944 | 0.33–0.38 |
| INT4 RTN block 32 (published) | 383 MB | 0.414–0.594 | 0.94–1.09 |
Cosine 0.95 passes every end-to-end probe; 0.41 is garbage. Note that finer groups made GPTQ worse here, the opposite of the usual expectation, and blk32 is also larger.
device_id must be 0. ORT's CUDA EP creates its cudnn handle before cudaSetDevice, so any
non-zero device_id dies at model load with
CUDNN failure 2007: CUDNN_STATUS_BAD_PARAM_STREAM_MISMATCH. To serve on a different physical
card, mask it with CUDA_VISIBLE_DEVICES=<n> (plus CUDA_DEVICE_ORDER=PCI_BUS_ID) and keep
device_id=0.max_length reserves the whole KV cache up front — genai_config.json sets
past_present_share_buffer: true. The artifact declares context_length: 131072, which on this
geometry (32 layers × 8 KV heads × 128 head size × 2 × 2 B) would ask ~17 GB. 8192 is what was
measured to fit 12 GB with this decoder.nvidia_awq cannot quantize either half of this model. NVAWQWeightOnlyQuantConfig
synthesises input_ids/attention_mask/position_ids/past-KV, but the vision tower takes
pixel_values/image_attention_mask/image_sizes and the decoder — built exclude_embeds=true —
takes inputs_embeds with no input_ids. Structural, not a tuning problem.import onnxruntime as ort
import onnxruntime_genai as og
# The pip CUDA wheels place their DLLs outside the process search path.
ort.preload_dlls()
cfg = og.Config("Phi-4-multimodal-instruct-gptq-int4-onnx")
cfg.clear_providers()
cfg.append_provider("cuda")
cfg.set_provider_option("cuda", "device_id", "0") # must be 0 — see gotchas
model = og.Model(cfg)
tokenizer = og.Tokenizer(model)
params = og.GeneratorParams(model)
params.set_search_options(max_length=8192, do_sample=False)
gen = og.Generator(model, params)
gen.append_tokens(tokenizer.encode(
"<|user|><|image_1|>\nTranscribe all text in this image exactly.<|end|><|assistant|>"))
Images and audio are passed as og.Images.open_bytes(...) / og.Audios.open_bytes(...) to
model.create_multimodal_processor(), and the prompt must carry the inline markers this
architecture expects — <|image_1|>, <|audio_1|>, numbered from 1 independently per modality.
Base model © Microsoft —
microsoft/Phi-4-multimodal-instruct
and its ONNX release
microsoft/Phi-4-multimodal-instruct-onnx,
whose embedding and speech graphs, adapters and tokenizer ship here unchanged. Quantization tooling:
ONNX Runtime, its bundled Intel Neural Compressor GPTQ implementation, and onnxruntime-genai's model
builder.