Phi-4-multimodal-instruct-int8-ov

Phi-4-multimodal-instruct — OpenVINO INT8 IR

An INT8 weight-compressed OpenVINO IR of microsoft/Phi-4-multimodal-instruct, exported with optimum-cli for openvino-genai. 5.6 B parameters: a 3.8 B Phi-4-mini language backbone, a ~0.45 B SigLIP vision encoder, a ~0.46 B conformer speech encoder, and the modality LoRA adapters.

There is no other OpenVINO IR of this model published anywhere, which is why this exists. The one third-party IR that comes up in search, katuni4ka/phi-4-multimodal-ov, cannot be loaded by openvino-genai at all — see Why model_type decides everything below — and has degenerate tiny dimensions besides. This is a full-size, loadable export.

pipeline_tag is text-generation deliberately, and it is the conservative claim. The vision and audio towers ARE in this IR and are quantized alongside the language model. What we have verified end to end is text. Read What works, and where before planning on the other two — one of them is blocked by the runtime, not by this artifact.

Released under the MIT License, inherited from the base model. This is a derivative quantization only; all model rights and terms follow the base model.

Files

33 files, 5.44 GB total. optimum-cli splits a multimodal model into one graph per component, each an .xml topology plus a .bin weight blob:

Plus config.json, openvino_config.json, generation_config.json, configuration_phi4mm.py, preprocessor_config.json and the tokenizer set (tokenizer.json, vocab.json, merges.txt, added_tokens.json, special_tokens_map.json, tokenizer_config.json).

modeling_phi4mm.py is deliberately absent — the compiled graphs replace the PyTorch implementation. configuration_phi4mm.py stays because config.json's auto_map points at it.

Quantization

Data-free INT8 weight compression, the optimum-cli export openvino --weight-format int8 default. Per openvino_config.json: dtype: int8, quant_method: default, group_size: -1 (per-channel), sym: false, dataset: null, ratio: 1.0. Every graph above is compressed — this is uniform INT8, not a mixed-precision preset.

No calibration dataset is involved, so there is no dataset bias to reason about and the export takes minutes rather than hours. Activations stay at runtime precision; only weights are compressed.

Why INT8 and not INT4

Because on this checkpoint --weight-format int4 is not 4-bit. optimum-intel ships a per-model preset in _DEFAULT_4BIT_WQ_CONFIGS keyed on microsoft/Phi-4-multimodal-instruct, and the CLI selects it instead of a uniform INT4 pass:

• lm_model → INT4, group_size 64, asymmetric, AWQ + scale estimation calibrated on textvqa, with an ignored_scope excluding the speech lora_B matmuls
• audio_encoder_model, vision_embeddings_model, text_embeddings_model → INT8 regardless

So an "INT4" build of this model differs from this one only in the language model; the towers are INT8 either way. Two consequences worth carrying:

1. An INT4-vs-INT8 comparison here says nothing about 4-bit tower quantization. A vision or audio failure on the INT4 build is evidence about the model or the pipeline, never about the towers' precision.
2. The INT4 preset makes the export data-aware — datasets becomes a required dependency and the run downloads textvqa, then spends CPU-bound time on AWQ calibration.

INT4's payoff is VRAM: roughly 2.3 GB for the language model against this build's 4.10 GB. Take it if you are tight on memory; this build is the one to start from if you want the plain answer to "does this model run here at all".

What works, and where

Measured on WhiteBeast (2× RTX 3060 12 GB, i7-3770K), OpenVINO 2026.3.1, openvino-genai 2026.3.1. All three findings below are about NVIDIA silicon and an AVX2-less CPU — this IR has not been measured on Intel GPU or NPU, where the vision result in particular should differ.

Vision — the encoder cannot build OpenCL kernels on an NVIDIA GPU. openvino_vision_embeddings_model fails compile_model with CL_BUILD_PROGRAM_FAILURE (clBuildProgram error -11). OpenVINO's GPU plugin is Intel's and its kernels want cl_intel_* extensions NVIDIA's runtime does not implement. This is not specific to this model — the unrelated OpenVINO/Phi-3.5-vision-instruct-int4-ov fails identically on the same card, while a trivial MatMul+ReLU graph compiles and returns correct output there. Relocating just that graph to CPU with AUTO:GPU.0,CPU lets the pipeline load, and an image request then dies one step later in the CPU-encoder-to-GPU-LM handoff with clEnqueueReadBuffer CL_INVALID_VALUE.

Audio — openvino-genai has no phi4mm audio path, and it fails silently. genai 2026.3.1 maps the phi4mm architecture onto its phi3_vision class, which has no audio support; qwen3_omni is the only architecture carrying an audio_encoder.cpp. Because audios= is a generic kwarg on every VLMPipeline, the call is accepted and ignored: the request returns HTTP 200 with the content <|audio_1|> — the model echoing back the marker that was never replaced. Nothing about the artifact is wrong; the audio encoder graph itself compiles on GPU.0 in 15.9 s. It is simply never called. Check for a real transcription before trusting any audio result from this runtime.

Device strings that work

AUTO:<accel>,CPU is the working spelling on NVIDIA, and it buys per-submodel placement for free: genai compiles every submodel with the same device string and the AUTO plugin falls back per graph, so the language model, text embeddings and all projections land on GPU.0 while only the un-buildable vision encoder goes to CPU. Cold compile that way: 782 s.

• Plain GPU.0 fails at launch on this IR.
• HETERO:GPU.0,CPU does not work — it assigns by query_model, which reports the ops as supported, and the kernel build then still fails. HETERO has no fallback for a graph that fails to build.
• Plain CPU cannot run the language model on a pre-AVX2 host: genai's PagedAttention lowers to oneDNN brgemm kernels that refuse to initialise (brgemm executor Init Failure) and the process does not survive it. An Ivy Bridge or older CPU is not a fallback device here.
• EXECUTION_DEVICES reports a parenthesised (CPU) while AUTO is still building for the target device in the background. That reading is transient — do not use it to decide placement.

Memory and context

At max_length 8192 on a 12 GB RTX 3060, loading this engine took the card from 5.8 GB to 10.8 GB used (with an unrelated ASR engine already resident). 8192 is at the edge on 12 GB, not comfortable. The base model is nominally 128K context; the ceiling here is memory and latency, not the model.

Audio input format (if you serve the towers yourself)

Not reachable through openvino-genai today, but the base model's contract is worth stating, because the same weights are usable through other runtimes:

Both yield 80 mel bins at 25 ms windows with a 10 ms hop (100 frames/s), which is the invariant the encoder actually depends on — the two parameter sets exist so that either rate produces the same feature geometry. Mono PCM16 WAV.

Anything other than 16 kHz or 8 kHz must be resampled by the caller. Unlike the ONNX Runtime GenAI path, nothing in the OpenVINO pipeline resamples for you, so an unexpected rate is a correctness bug rather than a slow path. Resample to 16 kHz if the transcript matters.

Why model_type decides everything (read this before re-exporting)

The transformers version silently decides whether a Phi-4-multimodal IR is loadable at all. openvino-genai 2026.3.1 supports phi4mm — the spelling Microsoft's --trust-remote-code implementation reports under transformers 4.51.3, and what this IR's config.json declares — and rejects phi4_multimodal, the name transformers' own native Phi4Multimodal class uses, with Unsupported 'phi4_multimodal' VLM model type (vlm_config.cpp:44) at pipeline construction. The supported set was read directly out of openvino_genai.dll: phi3_v, phi4mm, qwen2_vl, qwen2_5_vl, llava, llava_next, internvl_chat, minicpmv, gemma3 — phi4_multimodal absent.

So an IR that exports cleanly on a newer transformers is dead on arrival. Check config.json's model_type before trusting any third-party Phi-4-multimodal OpenVINO IR, this one included.

Reproducing this export

optimum-cli export openvino \
  -m microsoft/Phi-4-multimodal-instruct \
  --trust-remote-code \
  --weight-format int8 \
  <output-dir>

Four environment constraints, each of which cost a debugging session:

1. CPython 3.13, not 3.14. 3.14 made functools.partial a descriptor, so optimum's NORMALIZED_CONFIG_CLASS class-attribute partials bind self like methods and every exporter dies with TypeError: NormalizedConfig.__init__() got multiple values for argument 'allow_new'.
2. transformers>=4.51,<5; 4.51.3 verified. 5.x meta-device init breaks the conformer encoder's Tensor.item() call at __init__, and it produces the unloadable phi4_multimodal model_type above. Microsoft's own 4.48.2 pin is below optimum-intel's floor.
3. TEMP/TMP on a volume with ~12 GB free. The exporter stages each graph uncompressed under %TEMP% before compressing into the output dir. Running out surfaces as ios_base::badbit set: iostream stream error from ov.save_model, which names neither the path nor the space and reads like a corrupt model.
4. HF_HUB_DISABLE_XET=1 if xet-backed downloads stall at 0 bytes (they did, for 20+ min).

Exported with optimum 2.3.0 and optimum-intel from git main.

Related uploads

Two ONNX Runtime GenAI builds of the same base model, for CUDA rather than OpenVINO — both with working vision and audio, which is exactly what this runtime could not deliver on NVIDIA:

• jarvis-pet/Phi-4-multimodal-instruct-int8-onnx — INT8 decoder + GPTQ INT4 vision tower. Byte-exact OCR and audio.
• jarvis-pet/Phi-4-multimodal-instruct-gptq-int4-onnx — GPTQ INT4 decoder. Smaller, bigger window, occasionally drops a character.

If you are on an NVIDIA GPU and want this model's vision or speech, use one of those. Use this IR for Intel silicon, or for text generation on OpenVINO.