Downloads · 30 days
82
8% of all-time downloads
justinchuby/qwen3-0.6b-onnx-genai
qwen3-0.6b-onnx-genai is a machine learning model from justinchuby. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for nxrt. The card lists the license as apache-2.0.
An optimized CPU ONNX package that runs directly with the nxrt Python API. It demonstrates onnx-genai's canonical inferencemetadata.yaml format, which is the authoritative contract in this package. genaiconfig.json is…
Downloads · 30 days
82
8% of all-time downloads
All-time downloads
1K
Public
Repo size
536 MB
Likes
0
Public
Click a slice to open those files.
.onnx525 MB · 98%
From the Hugging Face model README
An optimized CPU ONNX package that runs directly with the nxrt Python API. It
demonstrates onnx-genai's canonical inference_metadata.yaml format, which is
the authoritative contract in this package. genai_config.json is retained only
for ecosystem compatibility.
The metadata is a fully serialized pipeline.workflow. The autoregressive loop
is data, not runtime-specific code: the decoder graph and the ten token-policy
graphs under policies/ are declared as workflow components, all 28 key/value
cache pairs are threaded through serving.state_service groups, and sampling,
termination, and length bookkeeping are ONNX graphs the runtime executes like
any other component. The package therefore runs on the generic workflow runtime
with no decoder-specific lowering step.
There is no model.io section. That legacy form described a single decoder in a
shape only a special-cased decoder loader could execute; it is not supported.
This package requires an nxrt development build from onnx-genai main at or
after session continuation support (c1d09e1b). Published PyPI builds through
0.1.0.dev4 predate the generic workflow runtime and cannot load this package.
# After building and installing nxrt from a compatible onnx-genai checkout:
ONNX_GENAI_KV_MAX_LEN=128 python3 - <<'PY'
import nxrt
engine = nxrt.genai.Engine.from_dir(".")
result = engine.generate(
"Answer in one concise sentence: What is the Rust programming language?\nAnswer:",
max_tokens=18,
temperature=0.0,
)
print(result.text)
print(result.token_ids)
print(result.finish_reason)
PY
Example CPU output:
The Rust programming language is a systems programming language that is known for its safety and performance.
[576, 33789, 15473, 4128, 374, 264, 5942, 15473, 4128, 429, 374, 3881, 369, 1181, 7149, 323, 5068, 382]
max_tokens
ONNX_GENAI_KV_MAX_LEN limits the shared KV allocation for this short example;
the model metadata preserves the full 40,960-token context limit.
The workflow declares a conversation state cell with scope: session and a
session.continuation naming the prompt input it rejoins, so a session carries
the conversation across turns: each turn's prompt is everything the session has
heard followed by the caller's tokens, and the cell then absorbs both that prompt
and the tokens the turn published. Independent sessions are isolated, and
resetting or closing one releases what it held.
session = engine.create_session()
engine.generate_in_session(session, "My name is Ada. Remember it.", max_tokens=8)
engine.generate_in_session(session, " What is my name?", max_tokens=8)
Without this declaration a runtime executing the package as written has nothing to carry, and every turn restarts from its own prompt. A request with no session is unaffected: a session holding nothing contributes nothing, so a conversation's first turn and a stateless generation are the same execution.
Derived from Qwen/Qwen3-0.6B.
Qwen3 is distributed under the Apache License 2.0. See LICENSE.
Review inference_metadata.annotated.yaml for inline explanations of this package's workflow, tensor/state/cache contracts, and fail-closed omissions. inference_metadata.yaml remains the canonical machine-authored contract; automated validation confirms both files parse to the same metadata object.