Downloads · 30 days
0
Mistral-7B-Instruct-v0.1-q4f16_1-metal
akashicmarga/Mistral-7B-Instruct-v0.1-q4f16_1-metal
Mistral-7B-Instruct-v0.1-q4f16_1-metal is a machine learning model from akashicmarga. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
The model in this repository utilizes Mistral-7B-Instruct-v0.1 (https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), the mlc-llm (https://llm.mlc.ai/docs/) Metal version with 4-bit quantization and an embedding…
Access
Public
Updated Sep 29, 2023
Repo size
4.1 GB
Likes
1
Public
Repo makeup
Click a slice to open those files.
.bin4.1 GB · 99%
At a glance
- License
- apache-2.0
- Access
- Public
- Created
- Sep 29, 2023
- Updated
- Sep 29, 2023
- SHA
- b6c05640
Model card
From the Hugging Face model README
The model in this repository utilizes Mistral-7B-Instruct-v0.1 (https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), the mlc-llm (https://llm.mlc.ai/docs/) Metal version with 4-bit quantization and an embedding layer for MLC embedding. You have the option to use the FastAPI server instead of OpenAI to run the model locally. For using in langchain, please refer to the sample_langchain.py file in the following GitHub link: https://github.com/mlc-ai/mlc-llm/blob/main/examples/rest/python/sample_langchain.py.
Environment setup
conda create -n mlc-chat-venv -c mlc-ai -c conda-forge mlc-chat-cli-nightly
conda activate mlc-chat-venv
Fast API Server
python -m mlc_chat.rest --model Mistral-7B-Instruct-v0.1-q4f16_1/ --lib-path Mistral-7B-Instruct-v0.1-q4f16_1/Mistral-7B-Instruct-v0.1-q4f16_1-metal.so
- License
- apache-2.0
- Created
- Sep 29, 2023
- Updated
- Sep 29, 2023