Downloads · 30 days
0
nota-ai/cpt-lora_st-vicuna-v1.3-3.7b-ppl-q4f16_0-MLC
cpt-lora_st-vicuna-v1.3-3.7b-ppl-q4f16_0-MLC is a machine learning model from nota-ai. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This is the CPT-LoRAST-Vicuna-v1.3-3.7B-PPL model in MLC format q4f160. The model can be used for projects MLC-LLM and WebLLM.
Downloads · 30 days
0
Access
Public
Updated Aug 21, 2024
Repo size
2.1 GB
Likes
0
Public
Click a slice to open those files.
.bin2.1 GB · 100%
From the Hugging Face model README
This is the CPT-LoRA_ST-Vicuna-v1.3-3.7B-PPL model in MLC format q4f16_0.
The model can be used for projects MLC-LLM and WebLLM.
Here are some examples of using this model in MLC LLM. Before running the examples, please install MLC LLM by following the installation documentation.
In command line, run
mlc_llm chat HF://nota-ai/cpt-lora_st-vicuna-v1.3-3.7b-ppl-q4f16_0-MLC
In command line, run
mlc_llm serve HF://nota-ai/cpt-lora_st-vicuna-v1.3-3.7b-ppl-q4f16_0-MLC
from mlc_llm import MLCEngine
# Create engine
model = "HF://nota-ai/cpt-lora_st-vicuna-v1.3-3.7b-ppl-q4f16_0-MLC"
engine = MLCEngine(model)
# Run chat completion in OpenAI API.
for response in engine.chat.completions.create(
messages=[{"role": "user", "content": "What is the meaning of life?"}],
model=model,
stream=True,
):
for choice in response.choices:
print(choice.delta.content, end="", flush=True)
print("\n")
engine.terminate()