Downloads · 30 days
0
keisuke-miyako/h2o-danube3-4b-chat-onnx-int4-NG
h2o-danube3-4b-chat-onnx-int4-NG is a machine learning model from keisuke-miyako. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
⚠️ Do not use with onnxruntime-genai; model architecture has a non-standard head size of 120 which is not divisible by 16.
Downloads · 30 days
0
Access
Public
Updated Jan 6, 2026
Repo size
2.9 GB
Likes
0
Public
Click a slice to open those files.
.data2.9 GB · 100%
From the Hugging Face model README
⚠️ Do not use with onnxruntime-genai; model architecture has a non-standard head size of 120 which is not divisible by 16.
To save memory during export & quantisation
hf download h2oai/h2o-danube3-4b-chat --local-dir ~/.onnx/h2o-danube3-4b-chat
from optimum.onnxruntime import ORTModelForCausalLM
import torch
model_id = "~/.onnx/h2o-danube3-4b-chat"
print(f"Loading {model_id} in FP16 to save RAM...")
# 1. Load model in FP16 directly (Crucial for preventing crashes)
model = ORTModelForCausalLM.from_pretrained(
model_id,
export=True,
torch_dtype=torch.float16, # This tells PyTorch to use half the memory
trust_remote_code=True
)
# 2. Save the ONNX model
print("Exporting to ONNX...")
model.save_pretrained("~/.onnx/h2o-danube3-4b-chat-onnx")
print("Done!")
python -m onnxruntime_genai.models.builder \
-m ~/.onnx/h2o-danube3-4b-chat-onnx \
-o ~/.onnx/h2o-danube3-4b-chat-onnx-int4-cpu \
-p int4 \
-e cpu
¥