Downloads · 30 days
0
justinchuby/qwen2.5-0.5b-instruct-onnx-genai
qwen2.5-0.5b-instruct-onnx-genai is a machine learning model from justinchuby. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for nxrt. The card lists the license as apache-2.0.
An INT4 ONNX package that runs directly with the nxrt Python API. It demonstrates onnx-genai's canonical inferencemetadata.yaml format, which is the authoritative contract in this package. genaiconfig.json is retained…
Downloads · 30 days
0
Access
Public
Updated Aug 27, 2026
Repo size
877 MB
Likes
0
Public
Click a slice to open those files.
.data866 MB · 99%
From the Hugging Face model README
An INT4 ONNX package that runs directly with the nxrt Python API. It
demonstrates onnx-genai's canonical inference_metadata.yaml format, which is
the authoritative contract in this package. genai_config.json is retained only
for ecosystem compatibility.
The metadata is a fully serialized pipeline.workflow. The autoregressive loop
is data, not runtime-specific code: the decoder graph and the ten token-policy
graphs under policies/ are declared as workflow components, all 24 key/value
cache pairs are threaded through serving.state_service groups, and sampling,
termination, and length bookkeeping are ONNX graphs the runtime executes like
any other component. The package therefore runs on the generic workflow runtime
with no decoder-specific lowering step.
There is no model.io section. That legacy form described a single decoder in a
shape only a special-cased decoder loader could execute; it is not supported.
onnx-genai CLI (CPU ONNX Runtime)Download the exact public snapshot into an explicit local package directory, then pass that directory to the CLI:
hf download justinchuby/qwen2.5-0.5b-instruct-onnx-genai \
--revision 1eabeec267303a75170ae1b43acf59cb01b47a63 \
--local-dir ./qwen2.5-0.5b-instruct-onnx-genai
ONNX_GENAI_EP=cpu ONNX_GENAI_KV_MAX_LEN=128 \
onnx-genai generate ./qwen2.5-0.5b-instruct-onnx-genai \
--backend ort \
--max-new-tokens 16 \
--temperature 0 \
--stop '<|im_end|>' \
'Hello! In one sentence, what is Rust?'
The model argument after generate is a local package directory, not a
Hugging Face repository ID. The prompt is the final positional argument.
--prompt (and -p) is obsolete and is intentionally rejected by the current
CLI.
For ordinary user text, do not pass --raw: the CLI must apply this
package's Qwen ChatML template. --raw is solely for a request that is already
fully formatted as ChatML, including its <|im_start|> / <|im_end|> markers,
so the CLI does not apply the template a second time. It is never the flag for a
plain prompt.
Compatibility evidence. The minimum known-tested runtime is untagged
onnx-genai source revision
3dabd2c0c2066183407c6bd98372e18e59c9571a,
whose workspace version is 0.1.0-dev.5, with this model snapshot
1eabeec267303a75170ae1b43acf59cb01b47a63
on CPU ORT. This is a known-tested source revision, not a supported minimum
release. Direct evidence is the revision's
CPU-ORT real-model CLI test
and its
HF package audit record.
The positional-only prompt spelling and removal of --prompt are recorded in
cc9ce30d422587760a79a0a06d18f3846b5bab6e.
No tagged release proves this exact current package/CLI combination. Published
onnx-genai-server / nxrt 0.1.0.dev4 artifacts predate the current hosted
pipeline.workflow metadata revision and are not claimed compatible here.
Genuinely older builds can reject newer metadata schema/capabilities or fail to
execute the declared workflow; this card does not assert a specific missing
graph input.
nxrt Python API (historical result)python3 -m pip install nxrt
ONNX_GENAI_KV_MAX_LEN=128 python3 - <<'PY'
import nxrt
engine = nxrt.genai.Engine.from_dir(".")
result = engine.generate(
"Hello! In one sentence, what is Rust?",
max_tokens=16,
temperature=0.0,
)
print(result.text)
print(result.token_ids)
print(result.finish_reason)
PY
The following output was recorded with nxrt==0.1.0.dev3 on CPU before the current workflow-metadata revisions. It is historical evidence, not a compatibility floor for the current snapshot:
Rust is a high-level programming language designed to be efficient and easy to use.
[33789, 374, 264, 1550, 11591, 15473, 4128, 6188, 311, 387, 11050, 323, 4135, 311, 990, 13]
max_tokens
ONNX_GENAI_KV_MAX_LEN limits the shared KV allocation for this short example;
the model metadata preserves the full 32,768-token context limit.
Derived from Qwen/Qwen2.5-0.5B-Instruct.
Qwen2.5 is distributed under the Apache License 2.0. See LICENSE.
Review inference_metadata.annotated.yaml for inline explanations of this package's workflow, tensor/state/cache contracts, and fail-closed omissions. inference_metadata.yaml remains the canonical machine-authored contract; automated validation confirms both files parse to the same metadata object.