Downloads · 30 days
0
poemAI/leo-hessianai-13b-chat-q4f16_1-MLC
leo-hessianai-13b-chat-q4f16_1-MLC is a machine learning model from poemAI. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as llama2.
This is a web-llm compatible version of LeoLM's German "leo-hessianai" language model. It is a 4-bit quantized version of a 13 billion parameter model based on llama2, the LeoLM/leo-hessianai-13b-chat.
Downloads · 30 days
0
Access
Public
Updated Feb 22, 2024
Repo size
7.3 GB
Likes
0
Public
Click a slice to open those files.
.bin7.3 GB · 100%
From the Hugging Face model README
This is a web-llm compatible version of LeoLM's German "leo-hessianai" language model. It is a 4-bit quantized version of a 13 billion parameter model based on llama2, the LeoLM/leo-hessianai-13b-chat.
It can be used with the MLC LLM project https://llm.mlc.ai/ . Notably you can use web-llm https://github.com/mlc-ai/web-llm to run this model in your browser.
Expect some quality loss due to the quantization, however the model is still able to generate good german text.
This model was created by running
mlc_chat convert_weight leo-hessianai-13b-chat --quantization q4f16_1 -o leo-hessianai-13b-chat-q4f16_1-MLC
where leo-hessianai-13b-chat is the directory into which the original model was downloaded. This is in line with the description in converting weights https://llm.mlc.ai/docs/compilation/convert_weights.html
The model here includes the mlc-chat.config.json to implement the correct prompt format for this particular model, as described in https://llm.mlc.ai/docs/get_started/mlc_chat_config.html
To use this model with the simple chat example of web-llm https://github.com/mlc-ai/web-llm/tree/main/examples/simple-chat, edit your copy of gh-config.js to include the following:
export default {
"model_list": [
{
"model_url": "https://huggingface.co/poemAI/leo-hessianai-13b-chat-q4f16_1-MLC/resolve/main/",
"local_id": "leo-hessianai-13b-chat-poemai",
"model_lib_url": "https://raw.githubusercontent.com/mlc-ai/binary-mlc-llm-libs/main/Llama-2-13b-chat-hf/Llama-2-13b-chat-hf-q4f16_1-ctx4k_cs1k-webgpu.wasm",
"vram_required_MB": 9109.03,
"low_resource_required": false,
},
],
"use_web_worker": true
}
As you can see you can use the LLama-2-13b webgpu wasm library provided by mlc-ai.
Note that the users of this model will need to download 6.8 GB of data to run it in their browser. Hardware requirements for the model when running in the browser have not yet been thoroughly tested - YMMV. However, it runs well on a MacBook Pro M2 Max with 64 GB RAM, MacOS Sonoma, Chrome 121.0.6167.184 (arm64).
Thanks to the people at LeoLM for providing the original model and to the MLC team for providing the tools to convert it to a web-llm compatible format.