Qwen3_5_0.8B_jev

Qwen3.5-0.8B Typed Decisions - ONNX Runtime GenAI

This repository contains a CUDA FP16 ONNX Runtime GenAI export of Qwen/Qwen3.5-0.8B, fine-tuned with a LoRA adapter using Microsoft Olive on n4ze3m/typed-decisions-synth. The adapter is merged into the exported ONNX weights.

Intended task

The model receives a state plus a typed noul, choice, or score question. It predicts an option letter (A, B, C, ...) at the first generated position. Applications should select the candidate letter logits and normalize only over the valid options instead of parsing unrestricted generated text.

Runtime

The model was exported as FP16 for the CUDA execution provider with:

• onnxruntime-genai-cuda==0.14.0
• onnxruntime-gpu==1.26.0
• Microsoft Olive 2fbeaf4316930d62bf7b85658ccc6e752d4b6f4c

These package versions target CUDA 12 environments. Load the directory containing genai_config.json.

import numpy as np
import onnxruntime_genai as og
from huggingface_hub import snapshot_download

model_dir = snapshot_download("lokinfey/Qwen3_5_0.8B_jev")
model = og.Model(model_dir)
tokenizer = og.Tokenizer(model)

prompt = """<|im_start|>user
State (data to evaluate):
Help! My payouts have failed for 3 days. I need the money today.

Question:
Does this message convey urgency?

Options:
A: false — No
B: true — Yes
Return only the option letter.<|im_end|>
<|im_start|>assistant
<think>

</think>

"""

input_ids = np.asarray(tokenizer.encode(prompt), dtype=np.int32)
params = og.GeneratorParams(model)
params.set_search_options(max_length=len(input_ids) + 1, do_sample=False)
generator = og.Generator(model, params)
generator.append_tokens(input_ids)
raw_logits = np.asarray(generator.get_logits(), dtype=np.float32)
logits = raw_logits.reshape(-1, raw_logits.shape[-1])[-1]
letter_ids = [int(np.asarray(tokenizer.encode(letter)).reshape(-1)[0]) for letter in "AB"]
candidate_logits = logits[letter_ids]
probabilities = np.exp(candidate_logits - candidate_logits.max())
probabilities /= probabilities.sum()
print({"false": float(probabilities[0]), "true": float(probabilities[1])})

Training

• Base model: Qwen/Qwen3.5-0.8B
• Dataset: n4ze3m/typed-decisions-synth, preserving its case-level train/validation split
• Sequence length: 768 tokens
• Method: LoRA, rank 32, alpha 64
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Epochs: 1
• Learning rate: 1e-4
• Precision: BF16 training, FP16 ONNX export

This Olive workflow uses standard full-sequence causal language-model SFT on the gold option letter. It does not reproduce Hmm's candidate-only hard/teacher soft-label objective, so its probability calibration and benchmark results should not be assumed to match n4ze3m/Qwen3.5-4B-Hmm.

Evaluation

No benchmark result is published in this card. Evaluate accuracy, negative log-likelihood, and calibration on your own held-out data by reading the first-token candidate logits. Olive's eval_loss is a full-sequence language-model loss and is not directly comparable to candidate-letter NLL or accuracy.

Limitations

• Intended for English prompts up to the 768-token training length.
• Supports at most 26 direct option letters.
• The synthetic training data and labels can contain errors or biases.
• Do not use this model as the sole safety, financial, legal, medical, or access-control decision maker.
• FP16 CUDA deployment requires compatible NVIDIA hardware and ONNX Runtime libraries.

Attribution

Prompt structure and typed-decision evaluation follow the public Hmm project. Model conversion and deployment use Microsoft Olive and ONNX Runtime GenAI.