Downloads · 30 days
20
4% of all-time downloads
OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc
cogvlm2-llama3-chat-19B-qvision-int4-sym-inc is a machine learning model from OPEA. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
This model is an int4 model with groupsize 128 and symmetric quantization of THUDM/cogvlm2-llama3-chat-19B generated by intel/auto-round. transformers <= 4.48.3
Downloads · 30 days
20
4% of all-time downloads
All-time downloads
448
Public
Parameters
3.7B
12 GB on disk
Likes
1
Public
Click a slice to open those files.
.safetensors12 GB · 100%
How the weights are stored.
I322.3B · 62%
From the Hugging Face model README
This model is an int4 model with group_size 128 and symmetric quantization of THUDM/cogvlm2-llama3-chat-19B generated by intel/auto-round.
transformers <= 4.48.3
import torch
from PIL import Image
from auto_round import AutoRoundConfig ##must import for auto-round format
from transformers import AutoModelForCausalLM, AutoTokenizer
import requests
MODEL_PATH = "OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc"
DEVICE = 'cuda' if torch.cuda.is_available() else 'cpu'
tokenizer = AutoTokenizer.from_pretrained(
MODEL_PATH,
trust_remote_code=True
)
model = AutoModelForCausalLM.from_pretrained(
MODEL_PATH,
torch_dtype="auto",
trust_remote_code=True,
device_map=DEVICE,
).to(DEVICE).eval()
text_only_template = "A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. USER: {} ASSISTANT:"
image_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg"
content = "Describe this image."
# Preparation for inference
query = text_only_template.format(content)
image = Image.open(requests.get(image_url, stream=True).raw)
input_by_model = model.build_conversation_input_ids(
tokenizer,
query=query,
images=[image],
template_version='chat'
)
inputs = {
'input_ids': input_by_model['input_ids'].unsqueeze(0).to(DEVICE),
'token_type_ids': input_by_model['token_type_ids'].unsqueeze(0).to(DEVICE),
'attention_mask': input_by_model['attention_mask'].unsqueeze(0).to(DEVICE),
'images': [[input_by_model['images'][0].to(DEVICE).to(model.dtype)]] if image is not None else None,
}
gen_kwargs = {
"max_new_tokens": 2048,
"pad_token_id": 128002,
"do_sample": False,
}
with torch.no_grad():
outputs = model.generate(**inputs, **gen_kwargs)
outputs = outputs[:, inputs['input_ids'].shape[1]:]
response = tokenizer.decode(outputs[0])
response = response.split("<|end_of_text|>")[0]
print(response)
##INT4:
## The image captures a serene moment at a beach during what appears to be sunset or sunrise. The sun casts a warm, golden hue over the scene. In the foreground, a woman sits on the sandy shore, facing a large, golden-colored dog. The dog, wearing a colorful harness, places one paw on the woman's hand, suggesting a bond or a playful gesture. The woman seems to be smiling, indicating a moment of joy or connection with the dog. The ocean waves gently crash in the background, and the horizon is visible, suggesting the vastness of the sea. The overall mood of the image is peaceful and heartwarming.
##BF16:
## The image showcases a serene beach setting during what appears to be either sunrise or sunset. In the foreground, a woman sits on the sandy beach, dressed in casual attire, including a checkered shirt and jeans. She is engaged in a moment of connection with a golden retriever dog, which is seated beside her. The dog wears a colorful harness and is looking up at the woman, possibly in anticipation of a treat or a playful gesture. The vast expanse of the ocean can be seen in the background, with gentle waves crashing onto the shore. The sky is clear, and the warm hues of the setting or rising sun cast a soft glow over the scene, creating a tranquil and heartwarming atmosphere.
image_url = "http://images.cocodataset.org/train2017/000000411975.jpg"
content = "图片中的棒球场上有多少人?"
##INT4:
## In the image provided, there are four individuals visible on the baseball field.
##BF16:
## In the image provided, there are five people visible on the baseball field.
image_url = "https://intelcorp.scene7.com/is/image/intelcorp/processor-overview-framed-badge:1920-1080?wid=480&hei=270"
content = "这张图片代表哪家公司?"
##INT4:
## The image represents Intel, a well-known multinational corporation that specializes in computer chips and other technologies.
##BF16:
## The image represents the company Intel.
pip3 install lmms_eval. The evaluation process may encounter errors that require changing model backend or evaluation code. Detailed instructions will be provided in a future update
auto-round-mllm --lmms --model OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc --tasks pope,textvqa_val,scienceqa,mmbench_en --output_dir "./eval_result" --device cuda:0
| Metric | 16bits | Llava Calib INT4 |
|---|---|---|
| avg | 80.38 | 80.21 |
| MMBench_DEV_EN_V11 | 75.86 | 75.77 |
| TextVQA_VAL | 77.77 | 77.15 |
| POPE | 87.37 | 87.70 |
Here is the sample command to reproduce the model.
pip install auto-round
auto-round-mllm \
--model THUDM/cogvlm2-llama3-chat-19B \
--device 0 \
--group_size 128 \
--bits 4 \
--iters 1000 \
--nsample 512 \
--seqlen 512 \
--quant_nontext_module \
--format 'auto_round' \
--output_dir "./tmp_autoround"
The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
Therefore, before deploying any applications of the model, developers should perform safety testing.
Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
Here are a couple of useful links to learn more about Intel's AI software:
The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }