Downloads · 30 days
18
0% of all-time downloads
Meta-Llama-3.1-8B-Instruct-FP8-128K
yejingfu/Meta-Llama-3.1-8B-Instruct-FP8-128K
Meta-Llama-3.1-8B-Instruct-FP8-128K is a text generation model from yejingfu. Use it when you need the model to write or continue text. The card lists the license as llama3.1.
- Model Architecture: Meta-Llama-3.1 - Input: Text - Output: Text - Model Optimizations: - Weight quantization: FP8 - Activation quantization: FP8 - KV Cache quantization:FP8 - Intended Use Cases: Intended for commerc…
All-time downloads
12.2K
Public
Parameters
8B
9.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors9.1 GB · 100%
Parameter types
How the weights are stored.
F8_E4M37B · 87%
Try a prompt
Model card
From the Hugging Face model README
Meta-Llama3.1-8B-FP8-128K
Model Overview
- Model Architecture: Meta-Llama-3.1
- Input: Text
- Output: Text
- Model Optimizations:
- Weight quantization: FP8
- Activation quantization: FP8
- KV Cache quantization:FP8
- Intended Use Cases: Intended for commercial and research use in multiple languages. Similarly to Meta-Llama-3.1-8B-Instruct, this models is intended for assistant-like chat.
- Out-of-scope: Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in languages other than English.
- Release Date: 8/27/2024
- Version: 1.0
- License(s): llama3.1
- Quantized version of Meta-Llama-3.1-8B-Instruct.
Serve with vLLM engine
python3 -m vllm.entrypoints.openai.api_server \
--port <port> --model yejingfu/Meta-Llama-3.1-8B-Instruct-FP8-128K \
--tensor-parallel-size 1 --swap-space 16 --gpu-memory-utilization 0.96 \
--max-num-seqs 32 --max-model-len 131072 --kv-cache-dtype fp8 --enable-chunked-prefill
license: llama3.1
- Task
- Text Generation
- Type
- llama
- License
- llama3.1
- Languages
- en, de, fr, it, pt, hi
- Created
- Aug 27, 2024
- Updated
- Aug 27, 2024