qwen-attention-onnx

Qwen2.5-Coder-0.5B — ONNX (uint4, attention outputs)

ONNX export of Qwen/Qwen2.5-Coder-0.5B, quantized to uint4 and extended with per-layer attention weight outputs for browser-based visualization with ONNX Runtime Web.

Files

Model outputs

Usage (ONNX Runtime Web)

import * as ort from 'onnxruntime-web';
import { AutoTokenizer } from '@huggingface/transformers';

const BASE = 'https://huggingface.co/YOUR_HF_USER/qwen-attention-onnx/resolve/main';

const tokenizer = await AutoTokenizer.from_pretrained(
  'onnx-community/Qwen2.5-Coder-0.5B-Instruct'
);

const modelBuffer = new Uint8Array(await fetch(`${BASE}/qwen_with_attention_uint4.onnx`).then(r => r.arrayBuffer()));
const dataBuffer  = new Uint8Array(await fetch(`${BASE}/qwen_with_attention_uint4.onnx.data`).then(r => r.arrayBuffer()));

const session = await ort.InferenceSession.create(modelBuffer, {
  executionProviders: ['webgpu', 'cpu'],
  externalData: [{ path: 'qwen_with_attention_uint4.onnx.data', data: dataBuffer }],
});

const ids = tokenizer.encode('def hello():', { add_special_tokens: false });
const n = ids.length;
const outputs = await session.run({
  input_ids:      new ort.Tensor('int64', BigInt64Array.from(ids.map(BigInt)), [1, n]),
  attention_mask: new ort.Tensor('int64', BigInt64Array.from(new Array(n).fill(1n)), [1, n]),
});

// outputs.logits         — next-token logits
// outputs.attentions_0   — layer 0 attention weights

Export details

• Base model: Qwen/Qwen2.5-Coder-0.5B
• Opset: 17
• Quantization: weight-only uint4, block size 32, asymmetric, accuracy level 4
• Attention outputs: head-averaged per layer, all 24 layers