Downloads · 30 days
136
5% of all-time downloads
RichardErkhov/pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf
pikhan_-_gpt2-medium-biochem-bioasq-pubmedqa-demo-gguf is a machine learning model from RichardErkhov. Use it for the machine learning task on the model card, and read the license before you ship it in a product.
Downloads · 30 days
136
5% of all-time downloads
All-time downloads
2.7K
Public
Repo size
4.4 GB
Likes
0
Public
Click a slice to open those files.
.gguf5.2 GB · 100%
From the Hugging Face model README
Quantization made by Richard Erkhov.
gpt2-medium-biochem-bioasq-pubmedqa-demo - GGUF
tags:
datasets:
Intel Optimization Inference Code Sample: We made use of both the BF16 datatype and INT8 quantization to improve performance. BF16 halves the memory compared to FP32, allowing larger models and/or larger batches to fit into memory. Moreover, BF16 is supported by modern Intel CPUs and operations with it are optimized. Quantizing models to INT8 can reduce the model size, making better use of cache and speeding up load times. Additionally, we then optimized further with OpenVino to make it run better on Intel Hardware by converting it to an onxx model to then OpenVINO Intermediate Representation
from openvino.runtime import Core import numpy as np
ie = Core()
compiled_model = ie.compile_model(model='../ovc_output/converted_model.xml', device_name="CPU")
input_ids = np.random.randint(0, 50256, (1, 10))
inputs = {"input_ids": input_ids}
result = compiled_model.create_infer_request().infer(inputs=inputs)
output = result['outputs']
print("Inference results:", output) In the finetuning file you will see our other optimizations.
We perform BF16 conversion as follows (we also implement a custom collator):
model = GPT2LMHeadModel.from_pretrained('gpt2-medium').to(torch.bfloat16) We perform Int8 quantization as follows:
model.eval() # Ensure the model is in evaluation mode quantized_model = quantize_dynamic(model, {torch.nn.Linear}, dtype=torch.qint8)