Downloads · 30 days
27
5% of all-time downloads
Phi-3-mini-4k-instruct_no_cache_uint8
julienkay/Phi-3-mini-4k-instruct_no_cache_uint8
Phi-3-mini-4k-instruct_no_cache_uint8 is a machine learning model from julienkay. Use it for the machine learning task on the model card, and read the license before you ship it in a product. It is set up for unity-sentis. The card lists the license as mit.
This is a work-in-progress attempt to use a Phi-3 model with Unity Sentis.
All-time downloads
532
Public
Repo size
3.8 GB
Likes
0
Public
Repo makeup
Click a slice to open those files.
.sentis3.8 GB · 100%
At a glance
- Library
- unity-sentis
- License
- mit
- Model type
- phi3
- Access
- Public
- Created
- Aug 2, 2024
- Updated
- Aug 14, 2024
- SHA
- 0123806c
Model card
From the Hugging Face model README
This is a work-in-progress attempt to use a Phi-3 model with Unity Sentis.
It's the microsoft/Phi-3-mini-4k-instruct model converted to .onnx via optimum:
from optimum.onnxruntime import ORTModelForCausalLM
model_id = "microsoft/Phi-3-mini-4k-instruct"
model = ORTModelForCausalLM.from_pretrained(model_id, use_cache = False, use_io_binding=False, export=True, trust_remote_code=True, cache_dir=".")
model.save_pretrained("phi3_onnx_no_cache/")
Then quantized to a Uint8 .sentis model file using Sentis v1.6.0-pre.1
Usage will be possible with the com.doji.transformers library, but support for LLMs is not officially released yet.
- Library
- unity-sentis
- Type
- phi3
- License
- mit
- Created
- Aug 2, 2024
- Updated
- Aug 14, 2024