Downloads · 30 days
23
11% of all-time downloads
Idefics3-8B-Llama3-FP8-Dynamic
leon-se/Idefics3-8B-Llama3-FP8-Dynamic
Idefics3-8B-Llama3-FP8-Dynamic is a machine learning model from leon-se. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
All-time downloads
211
Public
Parameters
8.5B
10 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors9.9 GB · 100%
Parameter types
How the weights are stored.
F8_E4M37B · 82%
Base models
Model card
From the Hugging Face model README
FP8-Dynamic quant for Idefics2-8B-Llama3, requires about ~12 GB
vLLM usage
vllm serve leon-se/Idefics3-8B-Llama3-FP8-Dynamic --trust-remote-code
Code used for quantization
from transformers import AutoProcessor, AutoModelForVision2Seq
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot
model_name = "HuggingfaceM4/Idefics3-8B-Llama3"
processor = AutoProcessor.from_pretrained(model_name)
model = AutoModelForVision2Seq.from_pretrained(model_name, device_map="cuda", torch_dtype="auto", trust_remote_code=True)
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=["re:.*lm_head", "re:model.vision_model.*", "re:model.connector.*"],
)
SAVE_DIR = model_name.split("/")[1] + "-FP8-Dynamic"
oneshot(model=model, recipe=recipe, output_dir=SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
- Type
- idefics3
- Created
- Jan 13, 2025
- Updated
- Jan 13, 2025