Downloads · 30 days
19
48% of all-time downloads
Qwen2.5-7B-Instruct-FP8-Dynamic_FP16-Vision
adamo1139/Qwen2.5-7B-Instruct-FP8-Dynamic_FP16-Vision
Qwen2.5-7B-Instruct-FP8-Dynamic_FP16-Vision is a machine learning model from adamo1139. Use it for the machine learning task on the model card, and read the license before you ship it in a product. The card lists the license as apache-2.0.
Script for creating this. llmcompressor installed from source since it depends on something that wasn't compiled into the release at the time.
All-time downloads
40
Public
Parameters
8.3B
10.1 GB on disk
Likes
0
Public
Repo makeup
Click a slice to open those files.
.safetensors10.1 GB · 100%
Parameter types
How the weights are stored.
F8_E4M36.5B · 79%
Model card
From the Hugging Face model README
Script for creating this. llmcompressor installed from source since it depends on something that wasn't compiled into the release at the time.
from transformers import AutoProcessor, Qwen2VLForConditionalGeneration
from llmcompressor.modifiers.quantization import QuantizationModifier
from llmcompressor.transformers import oneshot, wrap_hf_model_class
MODEL_ID = "Qwen/Qwen2-VL-7B-Instruct"
# Load model.
model_class = wrap_hf_model_class(Qwen2VLForConditionalGeneration)
model = model_class.from_pretrained(MODEL_ID, device_map="auto", torch_dtype="auto")
processor = AutoProcessor.from_pretrained(MODEL_ID)
# Configure the simple PTQ quantization
recipe = QuantizationModifier(
targets="Linear",
scheme="FP8_DYNAMIC",
ignore=["re:.*lm_head", "re:visual.*"]
)
# Apply the quantization algorithm.
oneshot(model=model, recipe=recipe)
# Save the model.
SAVE_DIR = MODEL_ID.split("/")[1] + "-FP8-Dynamic"
model.save_pretrained(SAVE_DIR)
processor.save_pretrained(SAVE_DIR)
- Type
- qwen2_vl
- License
- apache-2.0
- Created
- Oct 12, 2024
- Updated
- Oct 19, 2024